A38x: Broken Linux kernel booting over UART

Fri Nov 5 16:16:46 CET 2021

On Friday 05 November 2021 10:35:10 Tom Rini wrote:
> On Fri, Nov 05, 2021 at 12:38:21PM +0100, Pali Rohár wrote:
> > On Monday 11 October 2021 17:49:05 Pali Rohár wrote:
> > > On Monday 11 October 2021 10:45:44 Tom Rini wrote:
> > > > On Mon, Oct 11, 2021 at 04:33:44PM +0200, Pali Rohár wrote:
> > > > > On Monday 11 October 2021 10:32:22 Tom Rini wrote:
> > > > > > On Mon, Oct 11, 2021 at 04:25:48PM +0200, Pali Rohár wrote:
> > > > > > > On Monday 11 October 2021 10:03:21 Tom Rini wrote:
> > > > > > > > On Mon, Oct 11, 2021 at 12:13:29PM +0200, Pali Rohár wrote:
> > > > > > > > 
> > > > > > > > > Hello!
> > > > > > > > > 
> > > > > > > > > Current U-Boot master has broken booting of Linux kernel over UART on
> > > > > > > > > A38x.
> > > > > > > > > 
> > > > > > > > > After transferring image over UART it just prints:
> > > > > > > > > 
> > > > > > > > > CACHE: Misaligned operation at range [01000000, 014e5d96]
> > > > > > > > > ## Total Size      = 0x004e5d96 = 5135766 Bytes
> > > > > > > > > ## Start Addr      = 0x01000000
> > > > > > > > > Kernel image @ 0x1000000 [ 0x000000 - 0x4e0e60 ]
> > > > > > > > > FDT and ATAGS support not compiled in
> > > > > > > > > 
> > > > > > > > > resetting ...
> > > > > > > > > 
> > > > > > > > > It resets board and does not boot kernel. Note that I'm trying to boot
> > > > > > > > > recent 5.15 kernel image, not something old.
> > > > > > > > > 
> > > > > > > > > I did git bisect and it found following commit which broke booting:
> > > > > > > > > 
> > > > > > > > > 9774462e34faaa64a91eb9c68b438a52d22bba6a is the first bad commit
> > > > > > > > > commit 9774462e34faaa64a91eb9c68b438a52d22bba6a
> > > > > > > > > Author: Tom Rini <trini at konsulko.com>
> > > > > > > > > Date:   Mon Aug 30 09:16:30 2021 -0400
> > > > > > > > > 
> > > > > > > > >     arm: Disable ATAGs support
> > > > > > > > > 
> > > > > > > > > Prior this commit booting working fine.
> > > > > > > > > 
> > > > > > > > > Do you have any idea what is with above commit? Or any hints?
> > > > > > > > 
> > > > > > > > Can you provide the full log of what you're doing?  And what's in that
> > > > > > > > image you're passing, exactly?  Thanks.
> > > > > > > 
> > > > > > > Here is full log:
> > > > > > > 
> > > > > > > $ make CROSS_COMPILE=arm-linux-gnueabihf- turris_omnia_defconfig
> > > > > > > ...
> > > > > > > $ make CROSS_COMPILE=arm-linux-gnueabihf- -j8
> > > > > > > ...
> > > > > > > $ ./tools/kwboot -b ./u-boot-spl.kwb -B 5200000 -t /dev/ttyUSB0
> > > > > > > Patching image boot signature to UART
> > > > > > > Injecting binary header code for changing baudrate to 5200000 Bd
> > > > > > > Injecting code for changing baudrate back
> > > > > > > Aligning image header to Xmodem block size
> > > > > > > Sending boot message. Please reboot the target...-
> > > > > > > Waiting 2s and flushing tty
> > > > > > > Sending boot image header (115072 bytes)...
> > > > > > >   0 % [......................................................................]
> > > > > > > ...
> > > > > > >  93 % [...........................................................           ]
> > > > > > > Done
> > > > > > > 
> > > > > > > U-Boot SPL 2021.10-00600-gf331497d3ad4 (Oct 11 2021 - 16:13:39 +0200)
> > > > > > > High speed PHY - Version: 2.0
> > > > > > > MiniPCIe/mSATA card detection... MiniPCIe
> > > > > > > Detected Device ID 6820
> > > > > > > board SerDes lanes topology details:
> > > > > > >  | Lane # | Speed |  Type       |
> > > > > > >  --------------------------------
> > > > > > >  |   0    |   5   | PCIe0       |
> > > > > > >  |   1    |   5   | USB3 HOST0  |
> > > > > > >  |   2    |   5   | PCIe1       |
> > > > > > >  |   3    |   5   | USB3 HOST1  |
> > > > > > >  |   4    |   5   | PCIe2       |
> > > > > > >  |   5    |   0   | SGMII2      |
> > > > > > >  --------------------------------
> > > > > > > High speed PHY - Ended Successfully
> > > > > > > mv_ddr: 14.0.0 
> > > > > > > DDR3 Training Sequence - Switching XBAR Window to FastPath Window
> > > > > > > mv_ddr: completed successfully
> > > > > > > Disabling MCU watchdog... disabled
> > > > > > > Trying to boot from BOOTROM
> > > > > > > Returning to BootROM (return address 0xffff05c4)...
> > > > > > > 
> > > > > > > Changing baudrate to 5200000 Bd
> > > > > > > 
> > > > > > > Sending boot image data (747780 bytes)...
> > > > > > >   0 % [......................................................................]
> > > > > > > ...
> > > > > > >  99 % [.................................                                     ]
> > > > > > > Done
> > > > > > > Finishing transfer
> > > > > > > 
> > > > > > > Changing baudrate back to 115200 Bd
> > > > > > > 
> > > > > > > [Type Ctrl-\ + c to quit]
> > > > > > > 
> > > > > > > 
> > > > > > > U-Boot 2021.10-00600-gf331497d3ad4 (Oct 11 2021 - 16:13:39 +0200)
> > > > > > > 
> > > > > > > SoC:   MV88F6820-A0 at 1600 MHz
> > > > > > > DRAM:  2 GiB (800 MHz, 32-bit, 2T, ECC not enabled)
> > > > > > > WDT:   Started watchdog at 20300 with servicing (60s timeout)
> > > > > > > MMC:   mv_sdh: 0
> > > > > > > Loading Environment from SPIFlash... SF: Detected s25fl164k with page size 256 Bytes, erase size 4 KiB, total 8 MiB
> > > > > > > OK
> > > > > > > Model: Turris Omnia
> > > > > > > Turris Omnia:
> > > > > > >   RAM size: 2048 MiB
> > > > > > >   Serial Number: 0000000B00007B3C
> > > > > > > Regdomain set to **
> > > > > > > Net:   eth0: ethernet at 70000, eth1: ethernet at 30000, eth2: ethernet at 34000
> > > > > > > =>
> > > > > > > 
> > > > > > > $ kermit
> > > > > > > C-Kermit>set line /dev/ttyUSB0
> > > > > > > C-Kermit>set speed 115200
> > > > > > > C-Kermit>set carrier-watch off
> > > > > > > C-Kermit>connect
> > > > > > > Connecting to /dev/ttyUSB0, speed 115200
> > > > > > >  Escape character: Ctrl-\ (ASCII 28, FS): enabled
> > > > > > > Type the escape character followed by C to get back,
> > > > > > > or followed by ? to see other options.
> > > > > > > ----------------------------------------------------
> > > > > > > 
> > > > > > > => echo $uart_boot
> > > > > > > loadb $kernel_addr_r 5200000 && part uuid mmc 0:1 partuuid && setenv bootargs earlyprintk rootwait console=ttyS0,115200 rootfstype=btrfs root=PARTUUID=${partuuid} rootflags=commit=5,subvol=@ rw cfg80211.freg=${regdomain} && bootz ${kernel_addr_r}
> > > > > > > => run uart_boot
> > > > > > > ## Switch baudrate to 5200000 bps and press ENTER ...
> > > > > > > C-Kermit>set speed 5200000
> > > > > > > ?No keywords match - 5200000 (,Transmission rate for /dev/ttyUSB0 (bits per second))
> > > > > > > 
> > > > > > > ?SET SPEED fails, speed is 5260273
> > > > > > > C-Kermit>send /tmp/kernel
> > > > > > > C-Kermit>set speed 115200
> > > > > > > /dev/ttyUSB0, 115200 bps
> > > > > > > C-Kermit>connect
> > > > > > > CACHE: Misaligned operation at range [01000000, 014e5d96]
> > > > > > > ## Total Size     = 0x004e5d96 = 5135766 Bytes
> > > > > > > ## Start Addr      = 0x01000000
> > > > > > > ## Switch baudrate to 115200 bps and press ESC ...
> > > > > > > Kernel image @ 0x1000000 [ 0x000000 - 0x4e0e60 ]
> > > > > > > FDT and ATAGS support not compiled in
> > > > > > > 
> > > > > > > resetting ...
> > > > > > 
> > > > > > OK, and is /tmp/kernel something with an appended dtb then?
> > > > > 
> > > > > Yes, single image suitable for single kermit UART transfer:
> > > > > cat arch/arm/boot/zImage arch/arm/boot/dts/armada-385-turris-omnia.dtb > /tmp/kernel
> > > > 
> > > > Well, OK, there we go.  That's what you need to figure out how to fix
> > > > booting of.  You're hitting the panic in arch/arm/lib/bootm.c and need
> > > > to make sure that (a) we do whatever is needed for appended dtb to be
> > > > found by the kernel still/again and (b) check that ourselves, before
> > > > panic or not, in that case.  Before this was probably enabling ATAGs,
> > > > but that wasn't (I think? I forget the magic behind appended dtb as it's
> > > > been a while) actually used by Linux, just by U-Boot to not panic at
> > > > that point.
> > > 
> > > Ok, thanks for hints. So seems that som unrelated code path via atags
> > > was used even for DTS booting. I also do not remember details. I will
> > > try to debug it...
> > 
> > Hello Tom!
> > 
> > I would like to know, what is required to enable same boot atags
> > behavior like before applying that commit 9774462e34fa ("arm: Disable
> > ATAGs support")?
> > 
> > I have enabled following U-Boot option (as it seems to be enough):
> > 
> >   CONFIG_SUPPORT_PASSING_ATAGS=y
> > 
> > And also I have compiled kernel with following debug options:
> > 
> >   CONFIG_DEBUG_LL=y
> >   CONFIG_DEBUG_MVEBU_UART0_ALTERNATE=y
> >   CONFIG_DEBUG_UNCOMPRESS=y
> > 
> > (to see early kernel output on mvebu UART console).
> > 
> > But I see only these lines on UART:
> > 
> >   Starting kernel ...
> > 
> >   DTB:0x014E0798 (0x00004F96)
> >   C:0x010000E0-0x014E5740->0x010AF400-0x01594A60
> >   DTB:0x0158FAB8 (0x00004F96)
> >   Uncompressing Linux... done, booting the kernel.
> > 
> > And no more output. After some time watchdog reboots board.
> > 
> > Which means that something more in U-Boot is needed to enable atags
> > booting (with possible appended DTB). Or that mentioned commit really
> > broke booting. Any idea?
> 
> Looking at the U-Boot commit again, you had previously also been doing
> CMDLINE, INITRD and MEMORY ATAGs.  Since I see this in the kernel:
>                 /*
>                  * Look for an appended DTB.  If found, we cannot use it to
>                  * validate the calculated start of physical memory, as its
>                  * memory nodes may need to be augmented by ATAGS stored at
>                  * an offset from the same start of physical memory.
>                  */
> we might well need to still pass at least the memory tag?  No one has
> tried (I suspect) a completely empty ATAGs + appended DTB until more or
> less now.

Now I found it! Issue is with CMDLINE. Without CONFIG_CMDLINE_TAG kernel
does not see any cmdline passed by bootloader, it fallbacks to default
cmdline (which was empty for my zImage) and therefore has not printed
anything on UART (because console=... was not there). Missing rootfs
then caused freeze, timeout and board reset. So it looked like U-Boot
was not able to boot kernel at all, even everything worked fine. I
realized it with CONFIG_CMDLINE="earlyprintk console=ttyS0,115200"
kernel config option.

Compiling U-Boot with CONFIG_CMDLINE_TAG=y fixed this issue as U-Boot
started passing cmdline to kernel again.

So now I have a question: Do we want to support booting zImage with
appended DTB in U-Boot when ATAGs support is now disabled by default?