qemu_arm64_defconfig: PCI autoconfig fails for qemu-system-aarch64 -m 4G

[Tuomas Tynkkynen]

Hi Heinrich,

On Wed, 1 Jan 2020 at 19:58, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> Dear all,
>
> I want to run qemu_arm64_defconfig with 4G.
>
>     qemu-system-aarch64 -machine virt -m 4G -smp cores=2 \
>     -bios u-boot.bin -cpu cortex-a53 -nographic -gdb tcp::1234 \
>     -netdev user,id=eth0,tftp=tftp -device e1000,netdev=eth0 \
>     -device virtio-rng-pci

Out of curiosity, why E1000 over virtio-net?

> I see an error:
>
...
> pci_hose_phys_to_bus: invalid physical address
>
> The error does not occur for a smaller RAM size.
>
> I added some debug output:
>
> _dm_pci_phys_to_bus:
> phys_addr 0x000000013ffecb40,
> res->phys_start 0x0000000010000000,
> res->bus_start 0x10000000
>
> As qemu_arm64_defconfig does not define CONFIG_SYS_PCI_64BIT the
> calculated bus address is truncated.

Makes sense.

> If CONFIG_SYS_PCI_64BIT is defined for qemu_arm64_defconfig and the
> memory is less then 4 GiB an error
> PCI: Failed autoconfig bar 10
> occurs.

I believe this happens because the handling of the ranges DT property
handling (or it
could be in the PCI core itself, not sure) is a bit simplistic in U-Boot.

The ranges property in QEMU-generated dtb contains two PCI_REGION_MEM ranges,
one below 32-bit boundary and one above it. But decode_regions() in U-Boot only
supports one range of given type. When CONFIG_SYS_PCI_64BIT is unset,
only the region below 32-bit boundary is used, otherwise the one above 32-bits
is used. See commit 52ba907328ec069ff1cec6cbe659f1714c68ed33.

Thus with CONFIG_SYS_PCI_64BIT set any PCI devices which do not support 64-bit
mem BARs stop working. This should happen regardless of the RAM size, right?

> For 3GiB memory I observed the following values:
>
> _dm_pci_phys_to_bus:
> phys_addr 0x00000000fffecbc0,
> res->phys_start 0x0000000010000000,
> res->bus_start 0x0000000010000000
>
> When I define type pci_addr_t as 64bit everything works correctly.
>
> Why does the number of bits in pci_addr_t depend on CONFIG_SYS_PCI_64BIT
> and not on the bitness of the system?

As I understand it, some PCI host controllers may require using 64-bit PCI bus
addresses but still be connected to a 32-bit CPU.

> It is especially worrisome that CONFIG_SYS_PCI_64BIT is not documented
> at all. I wonder if Kumar remembers why he introduced it in 2008.
>
> I tried to change the definition of the following types in pci.h
>
> typedef phys_addr_t pci_addr_t;
> typedef phys_addr_t pci_size_t;
>
> This lets qemu-x86_defconfig fail with
> Error binding driver 'cpu_qemu': -12
>
> Using
>
> typedef unsigned long pci_addr_t;
> typedef unsigned long pci_size_t;
>
> does not produce the error but I am not sure if this is the right approach.
>

One solution would be to fix decode_regions() to work with multiple regions
of the same type and then enable CONFIG_SYS_PCI_64BIT.

I think another solution would be to simply limit U-Boot from being
relocated above
the 32-bit boundary. IIRC some other 64-bit boards do this as well. In
fact, this
might be the better solution because some bus mastering PCI devices cannot
access memory above 32-bit. As far as I can tell, E1000 is not affected but
for example EHCI is.