qemu_arm64_defconfig: PCI autoconfig fails for qemu-system-aarch64 -m 4G

[Heinrich Schuchardt]

On 1/2/20 12:28 PM, Tuomas Tynkkynen wrote:
> Hi Heinrich,
>
> On Wed, 1 Jan 2020 at 19:58, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>
>> Dear all,
>>
>> I want to run qemu_arm64_defconfig with 4G.
>>
>>      qemu-system-aarch64 -machine virt -m 4G -smp cores=2 \
>>      -bios u-boot.bin -cpu cortex-a53 -nographic -gdb tcp::1234 \
>>      -netdev user,id=eth0,tftp=tftp -device e1000,netdev=eth0 \
>>      -device virtio-rng-pci
>
> Out of curiosity, why E1000 over virtio-net?
>
>> I see an error:
>>
> ...
>> pci_hose_phys_to_bus: invalid physical address
>>
>> The error does not occur for a smaller RAM size.
>>
>> I added some debug output:
>>
>> _dm_pci_phys_to_bus:
>> phys_addr 0x000000013ffecb40,
>> res->phys_start 0x0000000010000000,
>> res->bus_start 0x10000000
>>
>> As qemu_arm64_defconfig does not define CONFIG_SYS_PCI_64BIT the
>> calculated bus address is truncated.
>
> Makes sense.
>
>> If CONFIG_SYS_PCI_64BIT is defined for qemu_arm64_defconfig and the
>> memory is less then 4 GiB an error
>> PCI: Failed autoconfig bar 10
>> occurs.
>
> I believe this happens because the handling of the ranges DT property
> handling (or it
> could be in the PCI core itself, not sure) is a bit simplistic in U-Boot.
>
> The ranges property in QEMU-generated dtb contains two PCI_REGION_MEM ranges,
> one below 32-bit boundary and one above it. But decode_regions() in U-Boot only
> supports one range of given type. When CONFIG_SYS_PCI_64BIT is unset,
> only the region below 32-bit boundary is used, otherwise the one above 32-bits
> is used. See commit 52ba907328ec069ff1cec6cbe659f1714c68ed33.
>
> Thus with CONFIG_SYS_PCI_64BIT set any PCI devices which do not support 64-bit
> mem BARs stop working. This should happen regardless of the RAM size, right?
>
>> For 3GiB memory I observed the following values:
>>
>> _dm_pci_phys_to_bus:
>> phys_addr 0x00000000fffecbc0,
>> res->phys_start 0x0000000010000000,
>> res->bus_start 0x0000000010000000
>>
>> When I define type pci_addr_t as 64bit everything works correctly.
>>
>> Why does the number of bits in pci_addr_t depend on CONFIG_SYS_PCI_64BIT
>> and not on the bitness of the system?
>
> As I understand it, some PCI host controllers may require using 64-bit PCI bus
> addresses but still be connected to a 32-bit CPU.
>
>> It is especially worrisome that CONFIG_SYS_PCI_64BIT is not documented
>> at all. I wonder if Kumar remembers why he introduced it in 2008.
>>
>> I tried to change the definition of the following types in pci.h
>>
>> typedef phys_addr_t pci_addr_t;
>> typedef phys_addr_t pci_size_t;
>>
>> This lets qemu-x86_defconfig fail with
>> Error binding driver 'cpu_qemu': -12
>>
>> Using
>>
>> typedef unsigned long pci_addr_t;
>> typedef unsigned long pci_size_t;
>>
>> does not produce the error but I am not sure if this is the right approach.
>>
>
> One solution would be to fix decode_regions() to work with multiple regions
> of the same type and then enable CONFIG_SYS_PCI_64BIT.
>
> I think another solution would be to simply limit U-Boot from being
> relocated above
> the 32-bit boundary. IIRC some other 64-bit boards do this as well. In
> fact, this
> might be the better solution because some bus mastering PCI devices cannot
> access memory above 32-bit. As far as I can tell, E1000 is not affected but
> for example EHCI is.
>

Hello Tuomas,

thank you for your analysis.

Changing the relocation address has amongst others the following side
effects:

* reducing the maximum memory for function tracing
* reducing the available memory for UEFI

Actually I was increasing RAM for QEMU due to working with function tracing.

Where in decode_regions() do you see the limitation to work with
multiple regions of the same type?

Best regards

Heinrich