Re: Booting Android using U-Boot

Wed Nov 8 01:06:16 CET 2023

Hi everyone,

Good news! I disabled compression on kernel build and the resulting boot image starts booting Linux.

This is great but it does not solve the decompression issue.
Thanks to Mattijs for pointing me in the compression direction which made me further investigate compression options.

So with compression disabled, let's move on to the next issue:
I picked the Quartz64 DTS commits, built and booted it, however there seems to still be an issue:
Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 4.19.154-xos (nobody at android-build) (Android (6877366 based on r383902b1) clang version 11.0.2 (https://android.googlesource.com/toolchain/llvm-project b397f81060ce6d701042b782172ed13bee898b79), GNU ld (binutils-2.27-bd24d23f) 2.27.0.20170315) #1 SMP PREEMPT Wed Nov 8 00:54:49 CET 2023
[    0.000000] Machine model: Pine64 RK3566 Quartz64-A Board
...omitted for brevity...
[    0.000007] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[    0.001828] Insufficient stack space to handle exception!
[    0.001835] ESR: 0x81054040 -- IABT (lower EL)
[    0.002766] FAR: 0xc08eea4d24faf096
[    0.003098] Task stack:     [0xffffff8009cc1000..0xffffff8009cc5000]
[    0.003697] IRQ stack:      [0xffffff8008000000..0xffffff8008004000]
[    0.004299] Overflow stack: [0xffffffc1ff713080..0xffffffc1ff714080]
[    0.004907] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.154-xos #1
[    0.005516] Hardware name: Pine64 RK3566 Quartz64-A Board (DT)
[    0.006073] pstate: 20400009 (nzCv daif +PAN -UAO)
[    0.006544] pc : start_kernel+0x26c/0x3d0
[    0.006931] lr : start_kernel+0x25c/0x3d0
...omitted for brevity...
[    0.098560] Kernel panic - not syncing: kernel stack overflow
[    0.099108] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.154-xos #1
[    0.099717] Hardware name: Pine64 RK3566 Quartz64-A Board (DT)
[    0.100270] Call trace:
[    0.100515]  dump_backtrace+0x0/0x158
[    0.100871]  show_stack+0x14/0x1c
[    0.101195]  dump_stack+0xb8/0xf0
[    0.101519]  panic+0x12c/0x2a4
[    0.101815]  panic+0x0/0x2a4
[    0.102095]  handle_bad_stack+0x118/0x12c
[    0.102479]  __bad_stack+0xc0/0xc4
[    0.102809]  start_kernel+0x26c/0x3d0

Full log: https://termbin.com/yzx6k

Do I need to do anything to reserve stack space for the kernel?

Thanks a lot!

Cheers,
Simao

On Tuesday, November 07, 2023 21:42 CET, Simão Gomes Viana <devel at superboring.dev> wrote:
 Hi Simon,
Hi everyone from the mailing list,

I'm following up to the email you sent today about a particular problem I'm facing trying to boot AOSP 11 (built from source) using U-Boot.
I have extended my original email to include additional information and added the mailing list to CC as requested.

In essence, it boils down to this:
## Booting Android Image at 0x00c00800 ...
Kernel load addr 0x00c01000 size 15184 KiB
Kernel command line: console=ttyS2,1500000 earlycon=uart8250,mmio32,0xfe660000 androidboot.selinux=permissive service_locator.enable=1 kpti=0 loop.max_part=7 firmware_class.path=/vendor/etc/firmware buildvariant=user
RAM disk load addr 0x01ad5000 size 751 KiB
## Flattened Device Tree blob at 00600000
   Booting using the fdt blob at 0x600000
Working FDT set to 600000
   Uncompressing Kernel Image
lz4 compressed: uncompress error -71
Must RESET board to recover

I have asked the same question on SO with more details.

The kernel is built in-line with the AOSP sources, just like LineageOS does it.
It builds the kernel using make as usual, then takes the Image.lz4 and runs mkbootimg with that.

I found this in scripts/Makefile.lib:
quiet_cmd_lz4 = LZ4     $@
cmd_lz4 = (cat $(filter-out FORCE,$^) | \
        lz4 -l -12 --favor-decSpeed stdin stdout && \
        $(call size_append, $(filter-out FORCE,$^))) > $@ || \
        (rm -f $@ ; false)

It is using the -l parameter which is required for LZ4 compression of the kernel image.

This is what is present in my BoardConfig:

# Kernel
BOARD_KERNEL_CMDLINE := \
    console=ttyS2,1500000 \
    earlycon=uart8250,mmio32,0xfe660000 \
    androidboot.selinux=permissive \
    service_locator.enable=1 \
    kpti=0 \
    loop.max_part=7 \
    firmware_class.path=/vendor/etc/firmware

BOARD_KERNEL_BASE := 0x10000000
BOARD_KERNEL_PAGESIZE := 2048
BOARD_KERNEL_IMAGE_NAME := Image.lz4
TARGET_KERNEL_CLANG_COMPILE := true
#TARGET_KERNEL_CLANG_VERSION := latest
TARGET_KERNEL_SOURCE := kernel/rockchip/rk356x
TARGET_KERNEL_CONFIG := xos_rk356x_defconfig
TARGET_PREBUILT_RESOURCE ?= $(TARGET_KERNEL_SOURCE)/resource.img
TARGET_BOOTLOADER_IS_2ND := true

This is what the file utility tells me:
out/target/product/rk3566_rgo/boot.img: Android bootimg, kernel (0x10008000), ramdisk (0x11000000), second stage (0x10f00000), page size: 2048, cmdline (console=ttyS2,1500000 earlycon=uart8250,mmio32,0xfe660000 androidboot.selinux=permissive service_locator.enable=1 kpti=0 loop.m)

unpack_bootimg shows:
boot magic: ANDROID!
kernel_size: 15547433
kernel load address: 0x10008000
ramdisk size: 768194
ramdisk load address: 0x11000000
second bootloader size: 200192
second bootloader load address: 0x10f00000
kernel tags load address: 0x10000100
page size: 2048
os version: 11.0.0
os patch level: 2021-09
boot image header version: 2
product name:
command line args: console=ttyS2,1500000 earlycon=uart8250,mmio32,0xfe660000 androidboot.selinux=permissive service_locator.enable=1 kpti=0 loop.max_part=7 firmware_class.path=/vendor/etc/firmware buildvariant=user
additional command line args:
recovery dtbo size: 0
recovery dtbo offset: 0x0000000000000000
boot header size: 1660
dtb size: 11792057
dtb address: 0x0000000011f00000

This is what the working boot.img built by the manufacturer yields:
boot.img: Android bootimg, kernel (0x10008000), ramdisk (0x11000000), second stage (0x10f00000), page size: 2048, cmdline (console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=CN androidboot.veritymode=enforcing androidboot.hardware=r)

unpack_bootimg shows:
boot magic: ANDROID!
kernel_size: 31041552
kernel load address: 0x10008000
ramdisk size: 904155
ramdisk load address: 0x11000000
second bootloader size: 200192
second bootloader load address: 0x10f00000
kernel tags load address: 0x10000100
page size: 2048
os version: 11.0.0
os patch level: 2021-04
boot image header version: 2
product name:
command line args: console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=CN androidboot.veritymode=enforcing androidboot.hardware=rk30board androidboot.console=ttyFIQ0 androidboot.verifiedbootstate=orange firmware_class.path=/vendor/etc/firmware init=/init rootwait ro loop.max_part=7 androidboot.selinux=permissive buildvariant=userdebug
additional command line args:
recovery dtbo size: 0
recovery dtbo offset: 0x0000000000000000
boot header size: 1660
dtb size: 111487
dtb address: 0x0000000011f00000

Given that the kernel image is actually smaller than the manufacturer one, there can't be an overlapping issue.
The only real difference is the dtb size which is a lot bigger on the non-functional boot image (I don't know why).
However, U-Boot is able to list all the DTBs.

The kernel source I'm using is the one provided in the BSP here (the 79G file): https://wiki.pine64.org/wiki/Quartz64_Software_Releases#Android_SDK

If I build the kernel with CONFIG_COMPRESS_GZ instead of CONFIG_COMPRESS_LZ4, I also get an error:
## Booting Android Image at 0x00c00800 ...
Kernel load addr 0x00c01000 size 13176 KiB
Kernel command line: console=ttyS2,1500000 earlycon=uart8250,mmio32,0xfe660000 androidboot.selinux=permissive service_locator.enable=1 kpti=0 loop.max_part=7 firmware_class.path=/vendor/etc/firmware buildvariant=userdebug
RAM disk load addr 0x018df000 size 890 KiB
## Flattened Device Tree blob at 00600000
   Booting using the fdt blob at 0x600000
Working FDT set to 600000
   Uncompressing Kernel Image
Error: inflate() returned -5
gzip compressed: uncompress error -1
Must RESET board to recover

file utility says this about the kernel inside the boot image:
kernel: gzip compressed data, max compression, from Unix, original size modulo 2^32 30021648

I am able to decompress it using gzip -d < kernel > kernel_decompressed
Now file utility says:
kernel_decompressed: Linux kernel ARM64 boot executable Image, little-endian, 4K pages

The manufacturer kernel is not compressed, so it just tries to boot without decompressing:

## Booting Android Image at 0x00c00800 ...
Kernel load addr 0x00c01000 size 30315 KiB
Kernel command line: console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=CN androidboot.veritymode=enforcing androidboot.hardware=rk30board androidboot.console=ttyFIQ0 androidboot.verifiedbootstate=orange firmware_class.path=/vendor/etc/firmware init=/init rootwait ro loop.max_part=7 androidboot.selinux=permissive buildvariant=userdebug
RAM disk load addr 0x0299c000 size 883 KiB
## Flattened Device Tree blob at 00600000
   Booting using the fdt blob at 0x600000
Working FDT set to 600000
   XIP Kernel Image
   Loading Ramdisk to ede39000, end edf15bdb ... OK
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
   Loading Device Tree to 00000000edd36000, end 00000000ede38fff ... OK
Working FDT set to edd36000

Starting kernel ...

But the kernel does not start, the board just resets after a few seconds.

I'm guessing there's something wrong with my bootcommand.

I'd appreciate it if you could point me in the right direction.
In case you need more information or want images of what I built, feel free to ask for it.

I have been trying to get this to work for weeks without any meaningful success and am running out of ideas.

Thanks a lot and best regards,
Simao