QEMU NUMA and U-Boot

Wed Jul 7 05:59:00 CEST 2021

Am 7. Juli 2021 05:18:20 MESZ schrieb Heinrich Schuchardt <xypron.glpk at gmx.de>:
>Am 7. Juli 2021 03:44:35 MESZ schrieb AKASHI Takahiro
><takahiro.akashi at linaro.org>:
>>François,
>>
>>On Tue, Jul 06, 2021 at 08:10:08PM +0200, Heinrich Schuchardt wrote:
>>> On 7/6/21 6:13 PM, François Ozog wrote:
>>> > Hi Heinrich, U-Boot 2021-07rc5 does not take into account memory
>>> > description when using Qemu 5.2 NUMA configuration to adapt memory
>>map
>>> > (kernel_addr_r...):
>>> > 
>>> >         -smp 4 \
>>> >           -m 8G,slots=2,maxmem=16G \
>>> >          -object memory-backend-ram,size=4G,id=m0 \
>>> >          -object memory-backend-ram,size=4G,id=m1 \
>>> >          -numa node,cpus=0-1,nodeid=0,memdev=m0 \
>>> >          -numa node,cpus=2-3,nodeid=1,memdev=m1
>>> > 
>>> > kernel_addr_r is still 0x4040000 and thus you can't use it to
>>bootefi.
>>> > 
>>> > fdt addr 0x13ede6de0; fdt print
>>> > 
>>> > Displays fdt while I think it should not.
>>> > 
>>> > If I load the kernel at dram.start, the load works but not boot
>>> > 
>>> > U-Boot 2021.07 (Jul 06 2021 - 13:26:43 +0000)
>>> > 
>>> > 
>>> > DRAM:4 GiB
>>> > 
>>> > Flash: 64 MiB
>>> > 
>>> > Loading Environment from Flash... OK
>>> > 
>>> > In:pl011 at 9000000
>>> > 
>>> > Out: pl011 at 9000000
>>> > 
>>> > Err: pl011 at 9000000
>>> > 
>>> > Net: eth0: virtio-net#32
>>> > 
>>> > Hit any key to stop autoboot:0
>>> > 
>>> > =>
>>> > 
>>> > => bdinfo
>>> > 
>>> > boot_params = 0x0000000000000000
>>> > 
>>> > DRAM bank = 0x0000000000000000
>>> > 
>>> > -> start= 0x0000000140000000
>>> > 
>>> > -> size = 0x0000000100000000
>>> > 
>>> > flashstart= 0x0000000000000000
>>> > 
>>> > flashsize = 0x0000000004000000
>>> > 
>>> > flashoffset = 0x00000000000bc990
>>> > 
>>> > baudrate= 115200 bps
>>> > 
>>> > relocaddr = 0x000000013ff27000
>>> > 
>>> > reloc off = 0x000000013ff27000
>>> > 
>>> > Build = 64-bit
>>> > 
>>> > current eth = virtio-net#32
>>> > 
>>> > ethaddr = 52:52:52:52:52:52
>>> > 
>>> > IP addr = <NULL>
>>> > 
>>> > fdt_blob= 0x000000013ede6de0
>>> > 
>>> > new_fdt = 0x000000013ede6de0
>>> > 
>>> > fdt_size= 0x0000000000100000
>>> > 
>>> > lmb_dump_all:
>>> > 
>>> > memory.cnt= 0x1
>>> > 
>>> > memory.reg[0x0].base = 0x140000000
>>> > 
>>> > .size = 0x100000000
>>> > 
>>> > 
>>> > reserved.cnt= 0x0
>>> > 
>>> > arch_number = 0x0000000000000000
>>> > 
>>> > TLB addr= 0x000000013fff0000
>>> > 
>>> > irq_sp= 0x000000013ede6dd0
>>> > 
>>> > sp start= 0x000000013ede6dd0
>>> > 
>>> > Early malloc usage: 3a8 / 2000
>>> > 
>>> > => load virtio 0:1 0x140000000 /oskit.efi
>>> > 
>>> > 853424 bytes read in 1 ms (813.9 MiB/s)
>>> > 
>>> > => bootefi0x140000000 0x13ede6dd0
>>> > 
>>> > ERROR: Failed to register WaitForKey event
>>> > 
>>> > Setting OsIndications failed
>>> > 
>>> > Error: Cannot initialize UEFI sub-system, r = 9
>>> > 
>>> > 
>>> > I think there is a need to calculate memory map based on previous
>>> > firmware (TFA, QEMU can be considered as previous frimware)
>>information
>>> > (DT or blob_list).
>>> > 
>>> > What do you think ?
>>> > 
>>> > Cheers
>>> > 
>>> > FF
>>> > 
>>> > --
>>> > 
>>> > François-Frédéric Ozog | /Director Business Development/
>>> > T: +33.67221.6485
>>> > francois.ozog at linaro.org <mailto:francois.ozog at linaro.org>
>>| Skype: ffozog
>>> > 
>>> > 
>>> 
>>> The kernel load address is hard coded here:
>>> include/configs/qemu-arm.h:41:  "kernel_addr_r=0x40400000\0" \
>>> 
>>> bdinfo shows:
>>> DRAM start = 0x140000000
>>> DRAM size  = 0x100000000
>>> 
>>> fdt addr $fdt_addr
>>> fdt printf
>>> 
>>> shows two memory areas. One at 40000000, one at 140000000.
>>
>>(This shows that U-Boot receives a correct memory map via dtb.)
>>
>>Is this a NUMA machine, isn't it? Why should we care of which
>>memory region be used here? Please note that this is a virtual
>machine,
>>there is no practical difference between two regions.
>>
>>The root problem is that U-Boot did not recognize there were two
>>memory regions. We can fix this issue in either way:
>>
>>1)
>>diff --git a/configs/qemu_arm64_defconfig
>>b/configs/qemu_arm64_defconfig
>>index f6e586627a8e..b70ffae8bf6e 100644
>>--- a/configs/qemu_arm64_defconfig
>>+++ b/configs/qemu_arm64_defconfig
>>@@ -1,7 +1,7 @@
>> CONFIG_ARM=y
>> CONFIG_POSITION_INDEPENDENT=y
>> CONFIG_ARCH_QEMU=y
>>-CONFIG_NR_DRAM_BANKS=1
>>+CONFIG_NR_DRAM_BANKS=2
>> CONFIG_ENV_SIZE=0x40000
>> CONFIG_ENV_SECT_SIZE=0x40000
>> CONFIG_AHCI=y
>>
>>2)
>>diff --git a/lib/fdtdec.c b/lib/fdtdec.c
>>index 4b097fb588ed..4067ea2dead6 100644
>>--- a/lib/fdtdec.c
>>+++ b/lib/fdtdec.c
>>@@ -1111,7 +1111,7 @@ int fdtdec_setup_memory_banksize(void)
>>                return -EINVAL;
>>        }
>> 
>>-       for (bank = 0; bank < CONFIG_NR_DRAM_BANKS; bank++) {
>>+       for (bank = 0; ; bank++) {
>>                ret = ofnode_read_resource(mem, reg++, &res);
>>                if (ret < 0) {
>>                        reg = 0;
>>
>>   (fdtdec_setup_memory_banksize() is called in dram_init_banksize().)
>>
>>
>>(2) seems much better, but I don't know why we had to use
>>CONFIG_NR_DRAM_BANKS here.
>>
>>In this case, other occurrences of CONFIG_NR_DRAM_BANKS in this file
>>should be replaced with a variable for it. 
>>
>>> Your use case is well beyond the typical U-Boot usage. So I guess it
>>> will be up to Linaro to provide the necessary patches:
>>> 
>>> * determine the active CPU
>>> * determine the RAM assigned to the active CPU according
>>>   to the numa-node-id in the device-tree
>>> * make sure that U-Boot only uses the memory of the active CPU
>>>   internally
>>> * make sure that the UEFI memory map contains a compliant
>description
>>> * possibly, dynamically set up the environment variables
>>> 
>>> +CC Tuomas Tynkkynen (maintainer for qemu_arm64_defconfig)
>>
>>For (1), we'd better have a different config, or increase
>>the value of CONFIG_NR_DRAM_BANKS to a bigger number?
>
>Is the system configured such that each CPU can access the others CPU's
>RAM when entering U-Boot?
>
>Best regards
>
>Heinrich
>

At least the comments for this patch sound as if on a physical system cross NUMA node memory access is only available after full SMP initialization:

https://patchwork.kernel.org/project/linux-acpi/patch/20180625130552.5636-1-lorenzo.pieralisi@arm.com/

QEMU may be less restrictive.

QEMU allows the node distance to be 255 indicating that cross node access is infeasible.

Best regards

Heinrich

>>
>>-Takahiro Akashi
>>
>>
>>> Best regards
>>> 
>>> Heinrich