[PATCH v1 09/10] pci: Add driver for Broadcom STB PCIe controller
Marek Szyprowski
m.szyprowski at samsung.com
Tue Apr 28 11:49:48 CEST 2020
Hi All,
On 27.04.2020 15:54, Marek Szyprowski wrote:
> On 27.04.2020 12:15, Nicolas Saenz Julienne wrote:
>> On Fri, 2020-04-24 at 18:50 +0200, Sylwester Nawrocki wrote:
>>> This patch adds basic driver for the Broadcom STB PCIe host controller.
>>> The code is based on Linux upstream driver (pcie-brcmstb.c) with MSI
>>> handling removed. The inbound access memory region is not currently
>>> parsed from dma-ranges DT property and a fixed 4GB region is used.
>>>
>>> The patch has been tested on RPI4 board, i.e. on BCM2711 SoC with VL805
>>> USB Host Controller.
>>>
>>> Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne at suse.de>
>>> Signed-off-by: Sylwester Nawrocki <s.nawrocki at samsung.com>
>>> ---
>>> Changes since RFC:
>>> - reworked to align with current Linux mainline version and u-boot
>>> driver
>>> by Nicolas Saenz Julienne
>> [...]
>>
>>> +
>>> + /*
>>> + * TODO: Use the base address and size(s) provided in the
>>> dma-ranges
>>> + * property.
>>> + */
>>> + rc_bar2_offset = 0;
>>> + rc_bar2_size = 1ULL << 32;
>> From experience this works fine, although it highly depends on how
>> DMA memory is
>> handled in u-boot.
>>
>> For example, in arm64 Linux, DMA memory was allocated from
>> ZONE_DMA32, which
>> would return memory smaller than 4GB. This was not good enough for
>> bcm2711b0 --
>> revision b0 of rpi4's SoC, so far the most common out there -- which
>> has an
>> internal wiring bug that prevents PCIe from accessing memory higher
>> than 3GB
>> (RPi4 might have up to 4GB). So we had to introduce a ZONE_DMA,
>> which covers
>> the lower GB of memory, in order to allocate suitable DMA memory for
>> PCI and
>> other DMA limited devices.
>>
>> While playing around with u-boot's xHCI I saw that all memory is
>> allocated is
>> located in the lower 1GB. But never got to look into why. I'm curious
>> to know
>> if someone knows how's that handled in u-boot. Ultimately, depending
>> on how it
>> works, we might be able to just disregard dma-ranges altogether.
>>
>> For some extra context xhci_malloc() gets its memory from memalign().
>> And it's
>> not clear to me how that function decides which memory to use.
>
> I think that memalign() allocates memory from the uboot's defined
> SDRAM (from its end). Assuming that CONFIG_SYS_SDRAM_SIZE is set to
> 128M in include/configs/rpi.h it should be always safe, but I will
> check that tomorrow to be 100% sure.
Okay, I've checked and memalign always get memory from the malloc pool,
which is set almost at the end of the first memory bank (in runtime), so
it is always below the first 1GiB. So this should be safe.
It is however not safe for explicit reads (and possible other
transactions) above 3rd GiB, see the log below:
Initialising SDRAM 'Samsung' 16Gb x2 total-size: 32 Gbit
Loading recovery.elf hnd: 0x00000000
Failed to read recovery.elf error: 3
Loading start4x.elf hnd: 0x000011de
Loading fixup4x.dat hnd: 0x00000259
MEM GPU: 76 ARM: 948 TOTAL: 1024
FIXUP src: 128 256 dst: 948 1024
Starting start4x.elf @ 0xfec00200
U-Boot 2020.04-00011-gaefcd8d-dirty (Apr 28 2020 - 10:13:31 +0200)
DRAM: 3.9 GiB
RPI 4 Model B (0xc03112)
MMC: emmc2 at 7e340000: 0, mmcnr at 7e300000: 1
Loading Environment from FAT... OK
In: serial
Out: serial
Err: serial
Net: eth0: genet at 7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (!SSC)
starting USB...
Bus Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices...
3 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
Hit any key to stop autoboot: 0
U-Boot> md 00001000
00001000: 04000000 54020000 02000000 03000000 .......T........
00001010: 08000000 65020000 61666564 00746c75 .......edefault.
00001020: 03000000 04000000 54010000 0f000000 ...........T....
00001030: 01000000 5f697064 6f697067 00000030 ....dpi_gpio0...
00001040: 03000000 70000000 73020000 00000000 .......p...s....
00001050: 01000000 02000000 03000000 04000000 ................
00001060: 05000000 06000000 07000000 08000000 ................
00001070: 09000000 0a000000 0b000000 0c000000 ................
00001080: 0d000000 0e000000 0f000000 10000000 ................
00001090: 11000000 12000000 13000000 14000000 ................
000010a0: 15000000 16000000 17000000 18000000 ................
000010b0: 19000000 1a000000 1b000000 03000000 ................
000010c0: 04000000 7d020000 06000000 03000000 .......}........
000010d0: 04000000 54010000 39000000 02000000 .......T...9....
000010e0: 01000000 636d6d65 6970675f 0032326f ....emmc_gpio22.
000010f0: 03000000 18000000 73020000 16000000 ...........s....
U-Boot> usb read 00001000 0 10
usb read: device 0 block # 0, count 16 ... 16 blocks read: OK
U-Boot> md 00001000
00001000: 1000b8fa 00bcd08e 0000b8b0 c08ed88e ................
00001010: 7c00befb b90600bf a4f30200 000621ea ...|.........!..
00001020: 07bebe00 0b750438 8110c683 7507fefe ....8.u........u
00001030: b416ebf3 bb01b002 80b27c00 8b01748a .........|...t..
00001040: 13cd024c 007c00ea 00feeb00 00000000 L.....|.........
00001050: 00000000 00000000 00000000 00000000 ................
00001060: 00000000 00000000 00000000 00000000 ................
00001070: 00000000 00000000 00000000 00000000 ................
00001080: 00000000 00000000 00000000 00000000 ................
00001090: 00000000 00000000 00000000 00000000 ................
000010a0: 00000000 00000000 00000000 00000000 ................
000010b0: 00000000 00000000 00000000 00000000 ................
000010c0: 00000000 00000000 00000000 00000000 ................
000010d0: 00000000 00000000 00000000 00000000 ................
000010e0: 00000000 00000000 00000000 00000000 ................
000010f0: 00000000 00000000 00000000 00000000 ................
U-Boot> md 40001000
40001000: ffffffff ffffffff ffffffff ffffffff ................
40001010: ffffffff ffffffff ffffffff ffffffff ................
40001020: ffdfffff ffffffff ffffffff fffffffd ................
40001030: ffffefff ffffffff ffffffff ffffffff ................
40001040: ffffffff fe7fffff ffffff7f ffffffff ................
40001050: ff7fdfff fdffffff ffffffff ffffffff ................
40001060: ffffffff 77ffbfdf ffffffff ffffdfff .......w........
40001070: fffdffdf fffff7ff ffffffff fffffdff ................
40001080: ffffffff ffffffff fffffffd ffffffff ................
40001090: fffffffb ffffffff ffffffff ffffffff ................
400010a0: ffffffff fffff7ff fffffdff fff7ffff ................
400010b0: ffffffff ffffffff fffeffff f7ffffff ................
400010c0: ffffffff ffffffff ffffffff fffffffd ................
400010d0: ffffffff ffdfffff fffdffff ffffffff ................
400010e0: 7fff7ffd ffffffff 7fffffff ffffffff ................
400010f0: ffffff7f ffbfffbf ffffdfff fffffffe ................
U-Boot> usb read 40001000 0 10
usb read: device 0 block # 0, count 16 ... 16 blocks read: OK
U-Boot> md 40001000
40001000: 1000b8fa 00bcd08e 0000b8b0 c08ed88e ................
40001010: 7c00befb b90600bf a4f30200 000621ea ...|.........!..
40001020: 07bebe00 0b750438 8110c683 7507fefe ....8.u........u
40001030: b416ebf3 bb01b002 80b27c00 8b01748a .........|...t..
40001040: 13cd024c 007c00ea 00feeb00 00000000 L.....|.........
40001050: 00000000 00000000 00000000 00000000 ................
40001060: 00000000 00000000 00000000 00000000 ................
40001070: 00000000 00000000 00000000 00000000 ................
40001080: 00000000 00000000 00000000 00000000 ................
40001090: 00000000 00000000 00000000 00000000 ................
400010a0: 00000000 00000000 00000000 00000000 ................
400010b0: 00000000 00000000 00000000 00000000 ................
400010c0: 00000000 00000000 00000000 00000000 ................
400010d0: 00000000 00000000 00000000 00000000 ................
400010e0: 00000000 00000000 00000000 00000000 ................
400010f0: 00000000 00000000 00000000 00000000 ................
U-Boot> md 80001000
80001000: 6d254fbc 252fa38e e985672f 03b6d513 .O%m../%/g......
80001010: 3e681fbe d65269ad b503e985 be48b50b ..h>.iR.......H.
80001020: 6ddacf4f 25d0a371 e97a67d9 0341d5ec O..mq..%.gz...A.
80001030: 3e971fc1 d6ad695e b5f4e96e beb7b5f4 ...>^i..n.......
80001040: dd117c69 5c71e187 42f6a0c7 1c71f2e6 i|....q\...B..q.
80001050: b6c2f1cb 63a6939a 945719b1 21cf29ad .......c..W..).!
80001060: ddeefc9e 5c8ce97e 4209a82e 1c86520d ....~..\...B.R..
80001070: b6b53164 63599365 c4a8194e a030a952 d1..e.YcN...R.0.
80001080: d9d1919f be400dbd 98d16e64 22e6f443 ...... at .dn..C.."
80001090: 567291df 12f2eae6 8c555bba bf08666e ..rV.....[U.nf..
800010a0: d82e9164 febf8dc6 d82feebf 2211f4ae d........./...."
800010b0: 568d1160 1205ea3b 8ca31b45 3ff7a681 `..V;...E......?
800010c0: 11bb547b 46761efb 87165bb6 fc41e2ce {T....vF.[....A.
800010d0: 951b2ba4 4737b38b 3f286527 073e22ee .+....7G'e(?.">.
800010e0: 1144ddaf 46891e87 87e35b49 fcbca221 ..D....FI[..!...
800010f0: 95e5ab5f 45c9b364 3fd565c8 07c12a17 _...d..E.e.?.*..
U-Boot> usb read 80001000 0 10
usb read: device 0 block # 0, count 16 ... 16 blocks read: OK
U-Boot> md 80001000
80001000: 1000b8fa 00bcd08e 0000b8b0 c08ed88e ................
80001010: 7c00befb b90600bf a4f30200 000621ea ...|.........!..
80001020: 07bebe00 0b750438 8110c683 7507fefe ....8.u........u
80001030: b416ebf3 bb01b002 80b27c00 8b01748a .........|...t..
80001040: 13cd024c 007c00ea 00feeb00 00000000 L.....|.........
80001050: 00000000 00000000 00000000 00000000 ................
80001060: 00000000 00000000 00000000 00000000 ................
80001070: 00000000 00000000 00000000 00000000 ................
80001080: 00000000 00000000 00000000 00000000 ................
80001090: 00000000 00000000 00000000 00000000 ................
800010a0: 00000000 00000000 00000000 00000000 ................
800010b0: 00000000 00000000 00000000 00000000 ................
800010c0: 00000000 00000000 00000000 00000000 ................
800010d0: 00000000 00000000 00000000 00000000 ................
800010e0: 00000000 00000000 00000000 00000000 ................
800010f0: 00000000 00000000 00000000 00000000 ................
U-Boot> md c0001000
c0001000: ff6fffff fffff7ff fffffff7 ffffffff ..o.............
c0001010: 7ffffdff ffeff7ff ff7fffff ffffdfff ................
c0001020: feffffff ffffbdff ffffffff fffff77f ................
c0001030: ffffffff ffdfffff fffffff7 efffffff ................
c0001040: fdffffff ffffdfff 7ff5fdff ffffffef ................
c0001050: ffbfffff fffffdf7 fff7dfff ffffffff ................
c0001060: ff7fffff ffffffff ffffffff ffffffff ................
c0001070: dff7ffff ffffffff ffffffff dfffffff ................
c0001080: ffffffff fbffffff ffffffff ffffff7f ................
c0001090: ffffffdf ff7ffeff ffffffff ffffdfff ................
c00010a0: ffffffff ffffffff 7ffffeff ffffdfff ................
c00010b0: ffffffff ffffffff ffffffff ffffffff ................
c00010c0: feffffff ffffffff ffffbfff 7fffdfff ................
c00010d0: ffffffff dfffffff 7fffffff ffffffff ................
c00010e0: ffff7fdf ffffffff dfbfffff ffffffff ................
c00010f0: ffffffff ffffffff ff7ff5ff ffffffff ................
U-Boot> usb read c0001000 0 10
usb read: device 0 block # 0, count 16 ... 16 blocks read: OK
U-Boot> md c0001000
c0001000: ff6fffff fffff7ff fffffff7 ffffffff ..o.............
c0001010: 7ffffdff ffeff7ff ff7fffff ffffdfff ................
c0001020: feffffff ffffbdff ffffffff fffff77f ................
c0001030: ffffffff ffdfffff fffffff7 efffffff ................
c0001040: fdffffff ffffdfff 7ff5fdff ffffffef ................
c0001050: ffbfffff fffffdf7 fff7dfff ffffffff ................
c0001060: ff7fffff ffffffff ffffffff ffffffff ................
c0001070: dff7ffff ffffffff ffffffff dfffffff ................
c0001080: ffffffff fbffffff ffffffff ffffff7f ................
c0001090: ffffffdf ff7ffeff ffffffff ffffdfff ................
c00010a0: ffffffff ffffffff 7ffffeff ffffdfff ................
c00010b0: ffffffff ffffffff ffffffff ffffffff ................
c00010c0: feffffff ffffffff ffffbfff 7fffdfff ................
c00010d0: ffffffff dfffffff 7fffffff ffffffff ................
c00010e0: ffff7fdf ffffffff dfbfffff ffffffff ................
c00010f0: ffffffff ffffffff ff7ff5ff ffffffff ................
U-Boot>
I think that there cannot be done much about it. u-boot doesn't have any
true DMA-mapping layer or a way to express the current limitations. IMHO
it is enough that it works for malloc'ed memory and everything else
should be considered as not really supported.
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
More information about the U-Boot
mailing list