Re: Raspberry Pi won’t boot from compressed subvolume (BTRFS)

Wed May 25 21:51:07 CEST 2022

Hi Matthias, thanks for your time and response.

> Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be
> called bcm2837-rpi-3-b.dtb

Sorry, that was an OCR error (wasn't sure how to screenshot or log to
a file in this context, so pictures -> OCR seemed like my best option,
pictures available here:
https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/18462/2?u=n8henrie).
It is indeed `bc*m*`... in the actual output.

> Then you try to fixup that address which has no valid DTB and fails.
>
> Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm
> not even sure which version of U-Boot you are using and if it has any patches on
> top of the upstream version.

Sorry, I included the link to the relevant code
(https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2d3/pkgs/misc/uboot/default.nix)
but should have included those details:

```
defaultSrc = fetchurl {
url = "ftp://ftp.denx.de/pub/u-boot/u-boot-${defaultVersion}.tar.bz2";
hash = "sha256-gbRUMifbIowD+KG/XdvIE7C7j2VVzkYGTvchpvxoBBM=";
defaultVersion = "2022.01";
```

By default it will have two patches:
- https://github.com/NixOS/nixpkgs/blob/master/pkgs/misc/uboot/0001-configs-rpi-allow-for-bigger-kernels.patch
- https://github.com/NixOS/nixpkgs/blob/master/pkgs/misc/uboot/0001-rpi-Copy-properties-from-firmware-dtb-to-the-loaded-.patch

The latter looks like it might be relevant.

I was suspicious about it being a u-boot issue because everything
works fine by default, it's only when I enable zstd compression that I
get the above errors. However I'm pretty new to NixOS and obviously
know little about u-boot, so you may well be correct!

Nate

On Wed, May 25, 2022 at 2:58 AM Matthias Brugger <matthias.bgg at gmail.com> wrote:
>
>
>
> On 22/05/2022 17:36, Nathan Henrie wrote:
> > Hello u-boot team,
> >
> > I’ve been experimenting for about a year with a NixOS-based Raspberry Pi
> > image, with the end goal of a zstd-compressed image that uses BTRFS
> > subvolumes, having the root filesystem at @ and boot at @boot (with several
> > other subvolumes). The NixOS Raspberry Pi images use u-boot by default.
> > I’ve started a few discussions in the NixOS community as I learn and
> > iterate:
> > https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and
> > I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi
> > . I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice
> > with NixOS, and that I understand very little about u-boot and bootloaders
> > in general, so I hope you can be patient as I humbly ask for someone to
> > point me in the right direction.
> >
> > For reference, the official NixOS code for generating u-boot images is
> > here:
> > https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2d3/pkgs/misc/uboot/default.nix
> >
> > I’ve made a lot of progress, including an image that reliably boots from
> > @boot to the @ root subvolume with the u-boot configuration:
> >
> > CONFIG_CMD_BTRFS=y
> > CONFIG_ZSTD=y
> > CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run
> > distro_bootcmd;"
> >
> > with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs"
> > "rootflags=subvol=@"
> >
> > Unfortunately, after the initial boot and going through the initial OS
> > installation steps, if I have compression enabled, it reboots into a boot
> > loop with the following error messages:
> >
> > Scanning mmc 0:2...
> > Found /@boot/extlinux/extlinux.conf
> > Retrieving file: /@boot/extlinux/extlinux.conf
> > 2938 bytes read in 27 ms (105.5 KiB/s)
> >
> > 1 NixOS Default
> > 2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf)
> > 3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf)
> > 4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr)
> > 5: NixOS
> > Enter choice: 1: NixOS - Default
> > Retrieving file:
> > /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd
> > 20588420 bytes read in 1910 ms (10.3 MiB/s)
> > Retrieving file:
> > /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image
> > 50629120 bytes read in 6863 ms (7 MiB/s)
> > append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p
> > re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0,
> > 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@
> > loglevel=4
> > Retrieving file:
> > /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb
>
> Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be
> called bcm2837-rpi-3-b.dtb
>
> > 14310 bytes read in 66 ms (210.9 KiB/s)
> > Moving Image from 0x80000 to 0x200000, end=32f0000
> > ## Flattened Device Tree blob at 03700000
> >     Booting using the fdt blob at 0x3700000
> >     Using Device Tree in place at 0000000003700000 end 00000000037067e5
> > fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE
> > ERROR: arch-specific fdt fixup failed
> >   - must RESET the board to recover.
> >
>
> Then you try to fixup that address which has no valid DTB and fails.
>
> Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm
> not even sure which version of U-Boot you are using and if it has any patches on
> top of the upstream version.
>
> Regards,
> Matthias
>
> > FDT creation failed!
> > resetting ...
> >
> > This happens even though I have CONFIG_ZSTD=y. If I do *not* enable
> > compression (in my BTRFS mount options), it reboots normally and everything
> > works!
> >
> > I may have made some OCR / C&P errors above; I’ve posted some pictures of
> > the error messages (and similar discussion to this email, with no
> > responses) here:
> > https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/18462/2
> >
> > I’m not currently using BTRFS RAID, just a single device. One of my other
> > (x86_64) machines uses a similar setup with @, @boot, and zstd-compression
> > with grub and works well, so I’m hoping this is also possible with u-boot.
> >
> > Seeing that the errors mention the FDT and fixup, I tried
> > CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting
> > changes to the error messages.
> >
> > What I’m assuming, based on the above, is that re-writing the @boot
> > subvolume with compression is moving the location of the device tree files
> > and that u-boot is no longer able to find the necessary files (some of
> > which I think may also be in the nix store in the @nix subvolume).
> >
> > Does that sound like a reasonable guess as to what is going on? If so, are
> > there config options or settings that I might be able to use to help u-boot
> > find the necessary files after things are re-written with compression?
> >
> > Many thanks in advance for your time and attention, and thanks for your
> > hard work on u-boot!