bugs with EFI "reserved" memory and LMB?

Francesco Dolcini francesco at dolcini.it
Mon Mar 25 17:33:35 CET 2024 

Hello all,
re-opening this old email thread ...

On Sat, Aug 12, 2023 at 08:23:32AM +0200, Heinrich Schuchardt wrote:
> On 7/27/23 14:14, Emanuele Ghidoli wrote:
> > On 27/07/2023 13:21, Heinrich Schuchardt wrote:
> > > On 27.07.23 13:08, Emanuele Ghidoli wrote:
> > > > Efi loader module have its own memory management that flags as reserved the
> > > > area between ram_top to ram_end (currently where dt reserved-memory is
> > > > falling).  uboot lmb updates reserved-memory by adding these
> > > > efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
> > > > 
> > > > 
> > > > On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in
> > > > the device tree (used for comunication with other microcontroller on the SoC,
> > > > OPTEE, ...) falling in the DDR range between 456 and 512 MB.
> > > > 
> > > > Normally U-Boot relocate itself at the end of DDR so it overlap this
> > > > reserved-memory and this of course does not work [1].
> > > > 
> > > > In this case U-Boot prints the following errors:
> > > > 
> > > >     ERROR: reserving fdt memory region failed (addr=9c800000 size=300000)
> > > >     ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000)
> > > >     ...
> > > >     ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
> > > > 
> > > > because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve
> > > > memories (lmb_flag LMB_NOMAP).
> > > > 
> > > > 
> > > > To fix this I moved the U-Boot relocation address, implementing
> > > > board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value
> > > > with no overlap (448 MB).
> > > > 
> > > > 
> > > > The memory map is:
> > > > +---------------+ 512 MB
> > > > |DT reserved-mem|
> > > > +---------------+ 456 MB
> > > > |free space     |
> > > > +---------------+ 448 MB (ram_top)
> > > > |               |
> > > > |uboot          |
> > > > |               |
> > > > +---------------+ 0 MB
> > > > 
> > > > This is working fine (the board is able to boot, no memory overlaps) ...
> > > > 
> > > > however, in this configuration U-Boot we still have some errors prints while
> > > > loading linux dtb:
> > > > 
> > > >     ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4)
> > > >     ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4)
> > > >     ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4)
> > > >     ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4)
> > > >     ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4)
> > > >     ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
> > > > 
> > > > And the reason of these error is complete different: The efi loader lmb memory
> > > > reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
> > > > 
> > > > 
> > > > 1) lib/lmb.c:lmb_reserve_common() reserve these memories:
> > > >     ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
> > > >      456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
> > > > 
> > > > 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
> > > > area. This is not coalesced/merged with the u-boot reserved area because it is
> > > > not contiguous.
> > > > 
> > > > This is the reserved-memory array at this point:
> > > >     ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE)
> > > >     ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
> > > >      456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
> > > > 
> > > > 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas.
> > > > In the last two steps efi loader add some reserved area that overlap u-boot and fdt
> > > > reserved area BUT they are also contiguous to the "efi loader reserved area".
> > > > The very last step reserve also area between ram_top (448M) to ram_end (512M)
> > > > 
> > > > We fall in this condition (where we have overlapping areas!):
> > > >     ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE)
> > > >     ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE)
> > > >      456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
> > > > 
> > > > 4) Now, while loading the linux fdt the reserved-memory areas are checked
> > > > toward efi loader reserved area, they overlap, BUT they have different
> > > > lmb_flag. So we get the ERROR print.
> > > > 
> > > > Hopefully this is clear, I undestand is not that obvious ...
> > > 
> > >     efidebug memmap
> > Verdin AM62 # efidebug memmap
> > MMC: no card present
> > No EFI system partition
> > No EFI system partition
> > Failed to persist EFI variables
> > Type             Start            End              Attributes
> > ================ ================ ================ ==========
> > CONVENTIONAL     0000000080000000-0000000098eec000 WB
> > BOOT DATA        0000000098eec000-0000000098eee000 WB
> > RUNTIME DATA     0000000098eee000-0000000098eef000 WB|RT
> > BOOT DATA        0000000098eef000-0000000098ef2000 WB
> > RUNTIME DATA     0000000098ef2000-0000000098ef3000 WB|RT
> > BOOT DATA        0000000098ef3000-0000000098ef4000 WB
> > RUNTIME DATA     0000000098ef4000-0000000098ef6000 WB|RT
> > BOOT DATA        0000000098ef6000-0000000098ef7000 WB
> > RUNTIME DATA     0000000098ef7000-0000000098f07000 WB|RT
> > BOOT DATA        0000000098f07000-0000000098f10000 WB
> > BOOT CODE        0000000098f10000-000000009bf20000 WB
> > RUNTIME CODE     000000009bf20000-000000009bf30000 WB|RT
> > BOOT CODE        000000009bf30000-000000009c000000 WB
> > BOOT DATA        000000009c000000-00000000a0000000 WB
> > > 
> > > and of
> > > 
> > >     bdinfo
> > With this "debug" patch I hope giving you a better picture:
> > 
> > diff --git a/lib/lmb.c b/lib/lmb.c
> > index b2c233edb64e..1feb7189936e 100644
> > --- a/lib/lmb.c
> > +++ b/lib/lmb.c
> > @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob)
> >          if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob)
> >                  boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
> > 
> > +       printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__);
> > +       lmb_dump_all_force(lmb);
> > +
> >          if (CONFIG_IS_ENABLED(EFI_LOADER))
> >                  efi_lmb_reserve(lmb);
> >   }
> > 
> > Verdin AM62 # bdinfo
> > boot_params = 0x0000000000000000
> > DRAM bank   = 0x0000000000000000
> > -> start    = 0x0000000080000000
> > -> size     = 0x0000000020000000
> > flashstart  = 0x0000000000000000
> > flashsize   = 0x0000000000000000
> > flashoffset = 0x0000000000000000
> > baudrate    = 115200 bps
> > relocaddr   = 0x000000009bf22000
> > reloc off   = 0x000000001b722000
> > Build       = 64-bit
> > current eth = ethernet at 8000000port@1
> > ethaddr     = 00:14:2d:e5:73:c1
> > IP addr     = <NULL>
> > fdt_blob    = 0x0000000099f10890
> > new_fdt     = 0x0000000099f10890
> > fdt_size    = 0x000000000000f4e0
> > multi_dtb_fit= 0x0000000000000000
> > lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve
> > lmb_dump_all:
> >   memory.cnt = 0x1 / max = 0x10
> >   memory[0]      [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0
> >   reserved.cnt = 0x3 / max = 0x10
> >   reserved[0]    [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0
> >   reserved[1]    [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4
> >   reserved[2]    [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4
> > lmb_dump_all:
> >   memory.cnt = 0x1 / max = 0x10
> >   memory[0]      [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0
> >   reserved.cnt = 0x4 / max = 0x10
> >   reserved[0]    [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0
> >   reserved[1]    [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0
> >   reserved[2]    [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4
> >   reserved[3]    [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4
> > devicetree  = separate
> > serial addr = 0x0000000002800000
> >   width      = 0x0000000000000000
> >   shift      = 0x0000000000000002
> >   offset     = 0x0000000000000000
> >   clock      = 0x0000000002dc6c00
> > arch_number = 0x0000000000000000
> > TLB addr    = 0x000000009bff0000
> > irq_sp      = 0x0000000099f10880
> > sp start    = 0x0000000099f10880
> > Early malloc usage: 2a88 / 8000
> > > > IMO we have two different bugs:
> > > >    - there is nothing that prevent that coalesced area may overlap other areas
> > > >      (while lmb module expect that there aren't overlapping areas)
> > > >    - efi loader (correctly) reserve between ram_top and ram_end BUT this area is
> > > >      reserved by fdt.
> > > > 
> > > > One potential solution could be to override efi_add_known_memory() and set
> > > > ram_top = ram_end, e.g.
> > > > 
> > > > -           efi_add_conventional_memory_map(ram_start, ram_end, ram_top);
> > > > +           efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
> > > > 
> > > > 
> > > > but this does not really seems like something that should ve overridden at the
> > > > board level.
> 
> I am able to reproduce the issue on a Pine A64 LTS with this tree:
> https://source.denx.de/u-boot/custodians/u-boot-efi/-/tree/top_reserved

Patrice send this patch https://lore.kernel.org/all/20240308101230.2595220-1-patrice.chotard@foss.st.com/
to fix what it looks like the same issue on STM32MP1.

Should we have a similar patch for TI K3? What do you think?

Francesco

More information about the U-Boot mailing list