Testing pci_mvebu.c with Kirkwood SoCs

Mon Nov 8 12:11:14 CET 2021

Hello!

On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
> Hi Pali,
> 
> On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi at gmail.com> wrote:
> >
> > Hi Pali,
> >
> > Looks like it is working! But in a weird way. Please see my responses
> > in between, and also in the test log below.
> >
> > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali at kernel.org> wrote:
> > >
> > > Hello!
> > >
> > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > I've applied your patch, and ran some tests. Looks like we got to the
> > > > bind, but it was not probing (or the probe died silently). Later, I
> > > > tried pci enum and got some errors during probing.
> > >
> > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > > "conflicts with another window" which you see in the log.
> > >
> > > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > > once.
> > >
> > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> > >
...
> > > >
> > > > - These are relevant configs used in the build (not sure if I need to
> > > > use CONFIG_DM_PCI_COMPAT?)
> > > >
> > > > CONFIG_DM=y
> > > > CONFIG_CMD_PCI=y
> > > > CONFIG_PCI=y
> > > > CONFIG_PCI_MVEBU=y
> > > > CONFIG_PCI_ENDPOINT=y
> > > > CONFIG_DM_PCI_COMPAT=y
> > > > CONFIG_USB_XHCI_HCD=y
> > > > CONFIG_USB_XHCI_PCI=y
> > > > CONFIG_USB_XHCI_MVEBU=y
> > >
> > > Following options should be enough:
> > >
> > > CONFIG_CMD_PCI=y
> > > CONFIG_PCI=y
> > > # CONFIG_DM_PCI_COMPAT is not set
> > > CONFIG_PCI_PNP=y
> > > CONFIG_PCI_MVEBU=y
> >
> > Since the PCI bus in this box is used for USB 3.0, I've included
> > various XHCI configs to see if I can probe this USB 3.0 flash drive
> > also (that's the ultimate test).

OK!

> > >
> > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > > should be enough.
> > >
> > > > BTW, the topic is no longer kwboot, should we move this to another new
> > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> > >
> > > Changed :-)
> >
> > Thanks :)
> >
> > Here is the log of a successful spin up for the PCI bus and USB 3.0
> > drive.  Please see my commentary ### in between the logs.
> >
> > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> > Pogoplug V4
> >
> > SoC:   Kirkwood 88F6281_A1
> > DRAM:  128 MiB
> > device_probe:
> > device_probe: exit 0
> >
> >
> > ### The device_probe above seems to have nothing happening.
> >
> >
> > mvebu_pcie_bind: begin
> > mvebu_pcie_bind: got an available node
> > Bound device  to pcie at 82000000
> > mvebu_pcie_bind: bound
> > mvebu_pcie_bind: exit 0
> > Bound device pcie at 82000000 to mbus at f1000000
> > Bound device mbus at f1000000 to root_driver
> > Bound device ehci at 50000 to ocp at f1000000
> > Bound device ethernet-controller at 72000 to ocp at f1000000
> > Bound device sata at 80000 to ocp at f1000000
> > Bound device mvsdio at 90000.blk to mvsdio at 90000
> > Bound device mvsdio at 90000 to ocp at f1000000
> > Bound device ocp at f1000000 to root_driver
> > NAND:  128 MiB
> > MMC:   device_probe:
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > device_probe: exit 0
> > device_probe:
> > device_probe:
> > mvsdio at 90000: 0
> > Loading Environment from NAND... *** Warning - bad CRC, using default
> > environment
> >
> > In:    serial
> > Out:   serial
> > Err:   serial
> > Net:   device_probe:
> > device_probe:
> >
> > Warning: ethernet-controller at 72000 (eth0) using random MAC address -
> > 86:b8:84:ee:a9:a5
> > device_probe: exit 0
> > eth0: ethernet-controller at 72000
> > Hit any key to stop autoboot:  0
> > Pogo_V4> usb reset
> > resetting USB...
> > Bus ehci at 50000: device_probe:
> > device_probe:
> > USB EHCI 1.00
> > device_probe: exit 0
> > scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> > device_probe:
> > device_probe:
> > Bound device usb_hub to usb_hub
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 3 USB Device(s) found
> >        scanning usb for storage devices... 1 Storage Device(s) found
> >
> >
> > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> > 2.0 was up as shown above.
> >
> >
> > Pogo_V4> pci enum
> > device_probe:
> > device_probe:
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > device_probe: exit 0
> > mvebu_pcie_probe:
> > Cannot add window '4:e8', conflicts with another window
> > PCIe unable to add mbus window for mem at 90000000+10000000
> > Cannot add window '4:e0', conflicts with another window
> > PCIe unable to add mbus window for IO at c0000000+00010000

Error is still there...

Have you tried to comment and disable calling
"mvebu_mbus_add_window_by_id" function from pci_mvebu.c?

> > mvebu_pcie_probe: exit 0
> > Bound device pci_0:0.0 to pcie0.0
> > device_probe:
> > device_probe:
> > Bound device xhci_pci to pci_0:0.0
> > device_probe: exit 0
> > device_probe: exit 0
> > device_probe:
> >
> > ### So I did the pci enum above to get the XHCI bound to PCIe.
> >
> > Pogo_V4> usb reset
> > resetting USB...
> > Bus ehci at 50000: device_probe:
> > device_probe:
> > USB EHCI 1.00
> > device_probe: exit 0
> > Bus xhci_pci: device_probe:
> > device_probe:
> > Register 400081f NbrPorts 4
> > Starting the controller
> > USB XHCI 1.00
> > device_probe: exit 0
> > scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> > device_probe:
> > device_probe:
> > Bound device usb_hub to usb_hub
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 3 USB Device(s) found
> > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 2 USB Device(s) found
> >        scanning usb for storage devices... 2 Storage Device(s) found
> >
> >
> > ### Now both drives are up!
> >
> >
> > Pogo_V4> pci 0
> > device_probe:
> > Scanning PCI devices on bus 0
> > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > _____________________________________________________________
> > 00.00.00   0x11ab     0x6281     Bridge device           0x04
> > Pogo_V4> pci 1
> > device_probe:
> > Scanning PCI devices on bus 1
> > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > _____________________________________________________________
> > 01.00.00   0x1b73     0x1009     Serial bus controller   0x03
> >
> >
> > ### And the pci command has meaningful output at this point as shown above.
> >
> >
> > Pogo_V4> usb tree
> > USB device tree:
> >   1  Hub (480 Mb/s, 0mA)
> >   |  u-boot EHCI Host Controller
> >   |
> >   +-2  Hub (480 Mb/s, 100mA)
> >     |  GenesysLogic USB2.0 Hub
> >     |
> >     +-3  Mass Storage (480 Mb/s, 224mA)
> >          SanDisk Ultra Fit 4C531001560827107320
> >
> >   1  Hub (5 Gb/s, 0mA)
> >   |  U-Boot XHCI Host Controller
> >   |
> >   +-2  Mass Storage (5 Gb/s, 224mA)
> >        SanDisk Cruzer Glide 3.0 4C530000130418116112
> >
> > Pogo_V4> usb info
> > 1: Hub,  USB Revision 2.0
> >  - u-boot EHCI Host Controller
> >  - Class: Hub
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered 0mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> >
> > 2: Hub,  USB Revision 2.0
> >  - GenesysLogic USB2.0 Hub
> >  - Class: Hub
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x05e3  Product 0x0610 Version 144.48
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered Remote Wakeup 100mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> >
> > 3: Mass Storage,  USB Revision 2.10
> >  - SanDisk Ultra Fit 4C531001560827107320
> >  - Class: (from Interface) Mass Storage
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x0781  Product 0x5583 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Bus Powered 224mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 2
> >      - Class Mass Storage, Transp. SCSI, Bulk only
> >      - Endpoint 1 In Bulk MaxPacket 512
> >      - Endpoint 2 Out Bulk MaxPacket 512
> >
> > 1: Hub,  USB Revision 3.0
> >  - U-Boot XHCI Host Controller
> >  - Class: Hub
> >  - PacketSize: 512  Configurations: 1
> >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered 0mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> >
> > 2: Mass Storage,  USB Revision 3.0
> >  - SanDisk Cruzer Glide 3.0 4C530000130418116112
> >  - Class: (from Interface) Mass Storage
> >  - PacketSize: 512  Configurations: 1
> >  - Vendor: 0x0781  Product 0x5597 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Bus Powered 224mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 2
> >      - Class Mass Storage, Transp. SCSI, Bulk only
> >      - Endpoint 1 In Bulk MaxPacket 1024
> >      - Endpoint 2 Out Bulk MaxPacket 1024
> >
> > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> > active as shown above.
> >
> > ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
> >
> >
> > Pogo_V4> ext2ls usb 1:1 /
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > <DIR>       4096 .
> > <DIR>       4096 ..
> >           522666 uboot.2021.07-tld-1.pogo_v4.bin
> >           524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> >
> > Great works Pali!
> >
> > I think some more investigation is needed. Why did we need to do "pci
> > enum", and then "usb reset", in that order, to get the PCI bus and the
> > XHCI controller probed? there must be something missing in the process
> > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > uclass?
> 
> I think I can see the order of enumeration. PCI must be enumerated
> first, and then XHCI being the controller on this host bus would come
> alive? I think we can live with 'pci enum' and 'usb reset' to get the
> USB 3.0 drives enumerated. However, it seems just a little bit
> unintuitive.

'pci enum' should be called internally by U-Boot during loading. So only
'usb start' would be needed.

But from your boot log it looks like that PCI enumaration was not done
and so calling 'pci enum' manually is needed.

I will look into U-Boot code why it happens...

Anyway, based on your test, PCIe must work correctly :)

Could you send config space dump of PCIe Root Port? Following U-Boot
command prints it on terminal: 'pci display.b 0.0.0 0 200'