Re: Raspberry Pi won’t boot from compressed subvolume (BTRFS)

[Matthias Brugger]

On 22/05/2022 17:36, Nathan Henrie wrote:
> Hello u-boot team,
> 
> I’ve been experimenting for about a year with a NixOS-based Raspberry Pi
> image, with the end goal of a zstd-compressed image that uses BTRFS
> subvolumes, having the root filesystem at @ and boot at @boot (with several
> other subvolumes). The NixOS Raspberry Pi images use u-boot by default.
> I’ve started a few discussions in the NixOS community as I learn and
> iterate:
> https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and
> I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi
> . I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice
> with NixOS, and that I understand very little about u-boot and bootloaders
> in general, so I hope you can be patient as I humbly ask for someone to
> point me in the right direction.
> 
> For reference, the official NixOS code for generating u-boot images is
> here:
> https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2d3/pkgs/misc/uboot/default.nix
> 
> I’ve made a lot of progress, including an image that reliably boots from
> @boot to the @ root subvolume with the u-boot configuration:
> 
> CONFIG_CMD_BTRFS=y
> CONFIG_ZSTD=y
> CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run
> distro_bootcmd;"
> 
> with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs"
> "rootflags=subvol=@"
> 
> Unfortunately, after the initial boot and going through the initial OS
> installation steps, if I have compression enabled, it reboots into a boot
> loop with the following error messages:
> 
> Scanning mmc 0:2...
> Found /@boot/extlinux/extlinux.conf
> Retrieving file: /@boot/extlinux/extlinux.conf
> 2938 bytes read in 27 ms (105.5 KiB/s)
> 
> 1 NixOS Default
> 2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf)
> 3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf)
> 4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr)
> 5: NixOS
> Enter choice: 1: NixOS - Default
> Retrieving file:
> /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd
> 20588420 bytes read in 1910 ms (10.3 MiB/s)
> Retrieving file:
> /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image
> 50629120 bytes read in 6863 ms (7 MiB/s)
> append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p
> re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0,
> 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@
> loglevel=4
> Retrieving file:
> /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb

Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be 
called bcm2837-rpi-3-b.dtb

> 14310 bytes read in 66 ms (210.9 KiB/s)
> Moving Image from 0x80000 to 0x200000, end=32f0000
> ## Flattened Device Tree blob at 03700000
>     Booting using the fdt blob at 0x3700000
>     Using Device Tree in place at 0000000003700000 end 00000000037067e5
> fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE
> ERROR: arch-specific fdt fixup failed
>   - must RESET the board to recover.
> 

Then you try to fixup that address which has no valid DTB and fails.

Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm 
not even sure which version of U-Boot you are using and if it has any patches on 
top of the upstream version.

Regards,
Matthias

> FDT creation failed!
> resetting ...
> 
> This happens even though I have CONFIG_ZSTD=y. If I do *not* enable
> compression (in my BTRFS mount options), it reboots normally and everything
> works!
> 
> I may have made some OCR / C&P errors above; I’ve posted some pictures of
> the error messages (and similar discussion to this email, with no
> responses) here:
> https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/18462/2
> 
> I’m not currently using BTRFS RAID, just a single device. One of my other
> (x86_64) machines uses a similar setup with @, @boot, and zstd-compression
> with grub and works well, so I’m hoping this is also possible with u-boot.
> 
> Seeing that the errors mention the FDT and fixup, I tried
> CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting
> changes to the error messages.
> 
> What I’m assuming, based on the above, is that re-writing the @boot
> subvolume with compression is moving the location of the device tree files
> and that u-boot is no longer able to find the necessary files (some of
> which I think may also be in the nix store in the @nix subvolume).
> 
> Does that sound like a reasonable guess as to what is going on? If so, are
> there config options or settings that I might be able to use to help u-boot
> find the necessary files after things are re-written with compression?
> 
> Many thanks in advance for your time and attention, and thanks for your
> hard work on u-boot!