Testing pci_mvebu.c with Kirkwood SoCs
Tony Dinh
mibodhi at gmail.com
Mon Nov 8 21:54:39 CET 2021
Hi Pali,
On Mon, Nov 8, 2021 at 3:11 AM Pali Rohár <pali at kernel.org> wrote:
>
> Hello!
>
> On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi at gmail.com> wrote:
> > >
> > > Hi Pali,
> > >
> > > Looks like it is working! But in a weird way. Please see my responses
> > > in between, and also in the test log below.
> > >
> > > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali at kernel.org> wrote:
> > > >
> > > > Hello!
> > > >
> > > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > I've applied your patch, and ran some tests. Looks like we got to the
> > > > > bind, but it was not probing (or the probe died silently). Later, I
> > > > > tried pci enum and got some errors during probing.
> > > >
> > > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > > > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > > > "conflicts with another window" which you see in the log.
> > > >
> > > > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > > > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > > > once.
> > > >
> > > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> > > >
> ...
> > > > >
> > > > > - These are relevant configs used in the build (not sure if I need to
> > > > > use CONFIG_DM_PCI_COMPAT?)
> > > > >
> > > > > CONFIG_DM=y
> > > > > CONFIG_CMD_PCI=y
> > > > > CONFIG_PCI=y
> > > > > CONFIG_PCI_MVEBU=y
> > > > > CONFIG_PCI_ENDPOINT=y
> > > > > CONFIG_DM_PCI_COMPAT=y
> > > > > CONFIG_USB_XHCI_HCD=y
> > > > > CONFIG_USB_XHCI_PCI=y
> > > > > CONFIG_USB_XHCI_MVEBU=y
> > > >
> > > > Following options should be enough:
> > > >
> > > > CONFIG_CMD_PCI=y
> > > > CONFIG_PCI=y
> > > > # CONFIG_DM_PCI_COMPAT is not set
> > > > CONFIG_PCI_PNP=y
> > > > CONFIG_PCI_MVEBU=y
> > >
> > > Since the PCI bus in this box is used for USB 3.0, I've included
> > > various XHCI configs to see if I can probe this USB 3.0 flash drive
> > > also (that's the ultimate test).
>
> OK!
>
> > > >
> > > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > > > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > > > should be enough.
> > > >
> > > > > BTW, the topic is no longer kwboot, should we move this to another new
> > > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> > > >
> > > > Changed :-)
> > >
> > > Thanks :)
> > >
> > > Here is the log of a successful spin up for the PCI bus and USB 3.0
> > > drive. Please see my commentary ### in between the logs.
> > >
> > > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> > > Pogoplug V4
> > >
> > > SoC: Kirkwood 88F6281_A1
> > > DRAM: 128 MiB
> > > device_probe:
> > > device_probe: exit 0
> > >
> > >
> > > ### The device_probe above seems to have nothing happening.
> > >
> > >
> > > mvebu_pcie_bind: begin
> > > mvebu_pcie_bind: got an available node
> > > Bound device to pcie at 82000000
> > > mvebu_pcie_bind: bound
> > > mvebu_pcie_bind: exit 0
> > > Bound device pcie at 82000000 to mbus at f1000000
> > > Bound device mbus at f1000000 to root_driver
> > > Bound device ehci at 50000 to ocp at f1000000
> > > Bound device ethernet-controller at 72000 to ocp at f1000000
> > > Bound device sata at 80000 to ocp at f1000000
> > > Bound device mvsdio at 90000.blk to mvsdio at 90000
> > > Bound device mvsdio at 90000 to ocp at f1000000
> > > Bound device ocp at f1000000 to root_driver
> > > NAND: 128 MiB
> > > MMC: device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > device_probe:
> > > device_probe:
> > > mvsdio at 90000: 0
> > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > environment
> > >
> > > In: serial
> > > Out: serial
> > > Err: serial
> > > Net: device_probe:
> > > device_probe:
> > >
> > > Warning: ethernet-controller at 72000 (eth0) using random MAC address -
> > > 86:b8:84:ee:a9:a5
> > > device_probe: exit 0
> > > eth0: ethernet-controller at 72000
> > > Hit any key to stop autoboot: 0
> > > Pogo_V4> usb reset
> > > resetting USB...
> > > Bus ehci at 50000: device_probe:
> > > device_probe:
> > > USB EHCI 1.00
> > > device_probe: exit 0
> > > scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> > > device_probe:
> > > device_probe:
> > > Bound device usb_hub to usb_hub
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 3 USB Device(s) found
> > > scanning usb for storage devices... 1 Storage Device(s) found
> > >
> > >
> > > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> > > 2.0 was up as shown above.
> > >
> > >
> > > Pogo_V4> pci enum
> > > device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > mvebu_pcie_probe:
> > > Cannot add window '4:e8', conflicts with another window
> > > PCIe unable to add mbus window for mem at 90000000+10000000
> > > Cannot add window '4:e0', conflicts with another window
> > > PCIe unable to add mbus window for IO at c0000000+00010000
>
> Error is still there...
>
> Have you tried to comment and disable calling
> "mvebu_mbus_add_window_by_id" function from pci_mvebu.c?
>
> > > mvebu_pcie_probe: exit 0
> > > Bound device pci_0:0.0 to pcie0.0
> > > device_probe:
> > > device_probe:
> > > Bound device xhci_pci to pci_0:0.0
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > device_probe:
> > >
> > > ### So I did the pci enum above to get the XHCI bound to PCIe.
> > >
> > > Pogo_V4> usb reset
> > > resetting USB...
> > > Bus ehci at 50000: device_probe:
> > > device_probe:
> > > USB EHCI 1.00
> > > device_probe: exit 0
> > > Bus xhci_pci: device_probe:
> > > device_probe:
> > > Register 400081f NbrPorts 4
> > > Starting the controller
> > > USB XHCI 1.00
> > > device_probe: exit 0
> > > scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> > > device_probe:
> > > device_probe:
> > > Bound device usb_hub to usb_hub
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 3 USB Device(s) found
> > > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 2 USB Device(s) found
> > > scanning usb for storage devices... 2 Storage Device(s) found
> > >
> > >
> > > ### Now both drives are up!
> > >
> > >
> > > Pogo_V4> pci 0
> > > device_probe:
> > > Scanning PCI devices on bus 0
> > > BusDevFun VendorId DeviceId Device Class Sub-Class
> > > _____________________________________________________________
> > > 00.00.00 0x11ab 0x6281 Bridge device 0x04
> > > Pogo_V4> pci 1
> > > device_probe:
> > > Scanning PCI devices on bus 1
> > > BusDevFun VendorId DeviceId Device Class Sub-Class
> > > _____________________________________________________________
> > > 01.00.00 0x1b73 0x1009 Serial bus controller 0x03
> > >
> > >
> > > ### And the pci command has meaningful output at this point as shown above.
> > >
> > >
> > > Pogo_V4> usb tree
> > > USB device tree:
> > > 1 Hub (480 Mb/s, 0mA)
> > > | u-boot EHCI Host Controller
> > > |
> > > +-2 Hub (480 Mb/s, 100mA)
> > > | GenesysLogic USB2.0 Hub
> > > |
> > > +-3 Mass Storage (480 Mb/s, 224mA)
> > > SanDisk Ultra Fit 4C531001560827107320
> > >
> > > 1 Hub (5 Gb/s, 0mA)
> > > | U-Boot XHCI Host Controller
> > > |
> > > +-2 Mass Storage (5 Gb/s, 224mA)
> > > SanDisk Cruzer Glide 3.0 4C530000130418116112
> > >
> > > Pogo_V4> usb info
> > > 1: Hub, USB Revision 2.0
> > > - u-boot EHCI Host Controller
> > > - Class: Hub
> > > - PacketSize: 64 Configurations: 1
> > > - Vendor: 0x0000 Product 0x0000 Version 1.0
> > > Configuration: 1
> > > - Interfaces: 1 Self Powered 0mA
> > > Interface: 0
> > > - Alternate Setting 0, Endpoints: 1
> > > - Class Hub
> > > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> > >
> > > 2: Hub, USB Revision 2.0
> > > - GenesysLogic USB2.0 Hub
> > > - Class: Hub
> > > - PacketSize: 64 Configurations: 1
> > > - Vendor: 0x05e3 Product 0x0610 Version 144.48
> > > Configuration: 1
> > > - Interfaces: 1 Self Powered Remote Wakeup 100mA
> > > Interface: 0
> > > - Alternate Setting 0, Endpoints: 1
> > > - Class Hub
> > > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> > > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> > >
> > > 3: Mass Storage, USB Revision 2.10
> > > - SanDisk Ultra Fit 4C531001560827107320
> > > - Class: (from Interface) Mass Storage
> > > - PacketSize: 64 Configurations: 1
> > > - Vendor: 0x0781 Product 0x5583 Version 1.0
> > > Configuration: 1
> > > - Interfaces: 1 Bus Powered 224mA
> > > Interface: 0
> > > - Alternate Setting 0, Endpoints: 2
> > > - Class Mass Storage, Transp. SCSI, Bulk only
> > > - Endpoint 1 In Bulk MaxPacket 512
> > > - Endpoint 2 Out Bulk MaxPacket 512
> > >
> > > 1: Hub, USB Revision 3.0
> > > - U-Boot XHCI Host Controller
> > > - Class: Hub
> > > - PacketSize: 512 Configurations: 1
> > > - Vendor: 0x0000 Product 0x0000 Version 1.0
> > > Configuration: 1
> > > - Interfaces: 1 Self Powered 0mA
> > > Interface: 0
> > > - Alternate Setting 0, Endpoints: 1
> > > - Class Hub
> > > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> > >
> > > 2: Mass Storage, USB Revision 3.0
> > > - SanDisk Cruzer Glide 3.0 4C530000130418116112
> > > - Class: (from Interface) Mass Storage
> > > - PacketSize: 512 Configurations: 1
> > > - Vendor: 0x0781 Product 0x5597 Version 1.0
> > > Configuration: 1
> > > - Interfaces: 1 Bus Powered 224mA
> > > Interface: 0
> > > - Alternate Setting 0, Endpoints: 2
> > > - Class Mass Storage, Transp. SCSI, Bulk only
> > > - Endpoint 1 In Bulk MaxPacket 1024
> > > - Endpoint 2 Out Bulk MaxPacket 1024
> > >
> > > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> > > active as shown above.
> > >
> > > ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
> > >
> > >
> > > Pogo_V4> ext2ls usb 1:1 /
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > <DIR> 4096 .
> > > <DIR> 4096 ..
> > > 522666 uboot.2021.07-tld-1.pogo_v4.bin
> > > 524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > >
> > > Great works Pali!
> > >
> > > I think some more investigation is needed. Why did we need to do "pci
> > > enum", and then "usb reset", in that order, to get the PCI bus and the
> > > XHCI controller probed? there must be something missing in the process
> > > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > > uclass?
> >
> > I think I can see the order of enumeration. PCI must be enumerated
> > first, and then XHCI being the controller on this host bus would come
> > alive? I think we can live with 'pci enum' and 'usb reset' to get the
> > USB 3.0 drives enumerated. However, it seems just a little bit
> > unintuitive.
>
> 'pci enum' should be called internally by U-Boot during loading. So only
> 'usb start' would be needed.
>
> But from your boot log it looks like that PCI enumaration was not done
> and so calling 'pci enum' manually is needed.
>
> I will look into U-Boot code why it happens...
>
> Anyway, based on your test, PCIe must work correctly :)
Indeed, it's working perfectly :) I've also commented out the 2 calls
to mvebu_mbus_add_window_by_id(), and no longer see the conflicts
error.
- if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
- (phys_addr_t)pcie->mem.start,
- PCIE_MEM_SIZE)) {
- if (mvebu_mbus_add_window_by_id(pcie->io_target, pcie->io_attr,
- (phys_addr_t)pcie->io.start,
- MBUS_PCI_IO_SIZE)) {
> Could you send config space dump of PCIe Root Port? Following U-Boot
> command prints it on terminal: 'pci display.b 0.0.0 0 200'
Sure, here is the log with the dump.
U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 22:52:05 -0800)
Pogoplug V4
SoC: Kirkwood 88F6281_A1
DRAM: 128 MiB
NAND: 128 MiB
MMC: mvsdio at 90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment
In: serial
Out: serial
Err: serial
Net:
Warning: ethernet-controller at 72000 (eth0) using random MAC address -
56:f0:8e:db:76:2f
eth0: ethernet-controller at 72000
Hit any key to stop autoboot: 0
Pogo_V4> pci enum
Pogo_V4> pci 0
Scanning PCI devices on bus 0
BusDevFun VendorId DeviceId Device Class Sub-Class
_____________________________________________________________
00.00.00 0x11ab 0x6281 Bridge device 0x04
Pogo_V4> pci 1
Scanning PCI devices on bus 1
BusDevFun VendorId DeviceId Device Class Sub-Class
_____________________________________________________________
01.00.00 0x1b73 0x1009 Serial bus controller 0x03
Pogo_V4> pci display.b 0.0.0 0 200
00000000: ab 11 81 62 07 00 10 00 03 00 04 06 00 00 01 00
00000010: 00 00 00 00 00 00 00 00 00 01 01 00 01 f1 00 00
00000020: 00 90 00 90 01 10 01 00 00 00 00 00 00 00 00 00
00000030: 00 c0 ff bf 40 00 00 00 00 00 00 00 00 01 00 00
00000040: 01 50 03 06 00 00 00 00 00 00 00 00 00 00 00 00
00000050: 05 60 80 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060: 10 00 41 00 80 80 00 00 00 20 00 00 11 ac 07 00
00000070: 08 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000100: 01 00 01 00 00 00 00 00 00 00 00 00 10 00 06 00
00000110: 00 00 00 00 00 20 00 00 00 00 00 00 01 00 00 4a
00000120: 04 00 00 01 00 00 08 01 0c 03 30 02 00 00 00 00
00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Pogo_V4>
Thanks,
Tony
More information about the U-Boot
mailing list