Raspberry Pi won’t boot from compressed subvolume (BTRFS)

[Nathan Henrie]

Hello u-boot team,

I’ve been experimenting for about a year with a NixOS-based Raspberry Pi
image, with the end goal of a zstd-compressed image that uses BTRFS
subvolumes, having the root filesystem at @ and boot at @boot (with several
other subvolumes). The NixOS Raspberry Pi images use u-boot by default.
I’ve started a few discussions in the NixOS community as I learn and
iterate:
https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and
I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi
. I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice
with NixOS, and that I understand very little about u-boot and bootloaders
in general, so I hope you can be patient as I humbly ask for someone to
point me in the right direction.

For reference, the official NixOS code for generating u-boot images is
here:
https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2d3/pkgs/misc/uboot/default.nix

I’ve made a lot of progress, including an image that reliably boots from
@boot to the @ root subvolume with the u-boot configuration:

CONFIG_CMD_BTRFS=y
CONFIG_ZSTD=y
CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run
distro_bootcmd;"

with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs"
"rootflags=subvol=@"

Unfortunately, after the initial boot and going through the initial OS
installation steps, if I have compression enabled, it reboots into a boot
loop with the following error messages:

Scanning mmc 0:2...
Found /@boot/extlinux/extlinux.conf
Retrieving file: /@boot/extlinux/extlinux.conf
2938 bytes read in 27 ms (105.5 KiB/s)

1 NixOS Default
2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf)
3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf)
4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr)
5: NixOS
Enter choice: 1: NixOS - Default
Retrieving file:
/@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd
20588420 bytes read in 1910 ms (10.3 MiB/s)
Retrieving file:
/@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image
50629120 bytes read in 6863 ms (7 MiB/s)
append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p
re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0,
1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@
loglevel=4
Retrieving file:
/@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb
14310 bytes read in 66 ms (210.9 KiB/s)
Moving Image from 0x80000 to 0x200000, end=32f0000
## Flattened Device Tree blob at 03700000
   Booting using the fdt blob at 0x3700000
   Using Device Tree in place at 0000000003700000 end 00000000037067e5
fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE
ERROR: arch-specific fdt fixup failed
 - must RESET the board to recover.

FDT creation failed!
resetting ...

This happens even though I have CONFIG_ZSTD=y. If I do *not* enable
compression (in my BTRFS mount options), it reboots normally and everything
works!

I may have made some OCR / C&P errors above; I’ve posted some pictures of
the error messages (and similar discussion to this email, with no
responses) here:
https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/18462/2

I’m not currently using BTRFS RAID, just a single device. One of my other
(x86_64) machines uses a similar setup with @, @boot, and zstd-compression
with grub and works well, so I’m hoping this is also possible with u-boot.

Seeing that the errors mention the FDT and fixup, I tried
CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting
changes to the error messages.

What I’m assuming, based on the above, is that re-writing the @boot
subvolume with compression is moving the location of the device tree files
and that u-boot is no longer able to find the necessary files (some of
which I think may also be in the nix store in the @nix subvolume).

Does that sound like a reasonable guess as to what is going on? If so, are
there config options or settings that I might be able to use to help u-boot
find the necessary files after things are re-written with compression?

Many thanks in advance for your time and attention, and thanks for your
hard work on u-boot!