Testing pci_mvebu.c with Kirkwood SoCs
Tony Dinh
mibodhi at gmail.com
Mon Nov 8 03:08:56 CET 2021
Hi Pali,
On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi at gmail.com> wrote:
>
> Hi Pali,
>
> Looks like it is working! But in a weird way. Please see my responses
> in between, and also in the test log below.
>
> On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali at kernel.org> wrote:
> >
> > Hello!
> >
> > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > I've applied your patch, and ran some tests. Looks like we got to the
> > > bind, but it was not probing (or the probe died silently). Later, I
> > > tried pci enum and got some errors during probing.
> >
> > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > "conflicts with another window" which you see in the log.
> >
> > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > once.
> >
> > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> >
> > > Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c
> > > and drivers/core/device.c to see the progress).
> > >
> > > - U-Boot boot log:
> > >
> > > U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 -0700)
> > > Pogoplug V4
> > >
> > > SoC: Kirkwood 88F6281_A1
> > > DRAM: 128 MiB
> > > mvebu_pcie_bind: begin
> > > mvebu_pcie_bind: got an available node
> > > Bound device to pcie at 82000000
> > > mvebu_pcie_bind: bound
> > > mvebu_pcie_bind: exit 0
> > > Bound device pcie at 82000000 to mbus at f1000000
> > > Bound device mbus at f1000000 to root_driver
> > > Bound device ehci at 50000 to ocp at f1000000
> > > Bound device ethernet-controller at 72000 to ocp at f1000000
> > > Bound device sata at 80000 to ocp at f1000000
> > > Bound device mvsdio at 90000.blk to mvsdio at 90000
> > > Bound device mvsdio at 90000 to ocp at f1000000
> > > Bound device ocp at f1000000 to root_driver
> > > NAND: 128 MiB
> > > MMC: mvsdio at 90000: 0
> > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > environment
> > >
> > > In: serial
> > > Out: serial
> > > Err: serial
> > > Net:
> > > Warning: ethernet-controller at 72000 (eth0) using random MAC address -
> > > 66:43:1f:50:f0:e7
> > > eth0: ethernet-controller at 72000
> > > Hit any key to stop autoboot: 0
> > >
> > > Pogo_V4> pci
> > > No such bus
> > > Pogo_V4> pci 0
> > > No such bus
> > > Pogo_V4> pci 1
> > > No such bus
> > >
> > > Pogo_V4> pci enum
> > > mvebu_pcie_probe:
> > > Cannot add window '4:e8', conflicts with another window
> > > PCIe unable to add mbus window for mem at 90000000+10000000
> > > Cannot add window '4:e0', conflicts with another window
> > > PCIe unable to add mbus window for IO at c0000000+00010000
> > > mvebu_pcie_probe: exit 0
> > > Bound device pci_0:0.0 to pcie0.0
> > > Bound device xhci_pci to pci_0:0.0
> > >
> > > Pogo_V4> pci regions
> > > # Bus start Phys start Size Flags
> > > 0 0x0000000090000000 0x0000000090000000 0x0000000010000000 mem
> > > 1 0x0000000000000000 0x0000000000000000 0x0000000008000000 mem sysmem
> > > 2 0x00000000c0000000 0x00000000c0000000 0x0000000000010000 io io
> > >
> > > - Linux dmesg (grep for pci and xhci)
> > >
> > > [ 13.163011][ T1] mvebu-pcie mbus at f1000000:pcie at 82000000: host
> > > bridge /mbus at f1000000/pcie at 82000000 ranges:
> > > [ 13.163147][ T1] mvebu-pcie mbus at f1000000:pcie at 82000000:
> > > MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
> > > [ 13.163217][ T1] mvebu-pcie mbus at f1000000:pcie at 82000000:
> > > MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > > [ 13.163268][ T1] mvebu-pcie mbus at f1000000:pcie at 82000000:
> > > IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > > [ 13.163754][ T1] mvebu-pcie mbus at f1000000:pcie at 82000000: PCI
> > > host bridge to bus 0000:00
> > > [ 13.163791][ T1] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [ 13.163829][ T1] pci_bus 0000:00: root bus resource [mem
> > > 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
> > > [ 13.163862][ T1] pci_bus 0000:00: root bus resource [mem
> > > 0xe0000000-0xefffffff]
> > > [ 13.163892][ T1] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff]
> > > [ 13.164075][ T1] pci 0000:00:01.0: [11ab:6281] type 01 class 0x060400
> > > [ 13.164133][ T1] pci 0000:00:01.0: reg 0x38: [mem
> > > 0x00000000-0x000007ff pref]
> > > [ 13.165893][ T1] PCI: bus0: Fast back to back transfers disabled
> > > [ 13.165949][ T1] pci 0000:00:01.0: bridge configuration invalid
> > > ([bus 00-00]), reconfiguring
> > > [ 13.166276][ T1] pci 0000:01:00.0: [1b73:1009] type 00 class 0x0c0330
> > > [ 13.166334][ T1] pci 0000:01:00.0: reg 0x10: [mem
> > > 0x00000000-0x0000ffff 64bit]
> > > [ 13.166381][ T1] pci 0000:01:00.0: reg 0x18: [mem
> > > 0x00000000-0x00000fff 64bit]
> > > [ 13.166424][ T1] pci 0000:01:00.0: reg 0x20: [mem
> > > 0x00000000-0x00000fff 64bit]
> > > [ 13.166561][ T1] pci 0000:01:00.0: supports D1
> > > [ 13.166588][ T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
> > > [ 13.166685][ T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe
> > > bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable
> > > of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
> > > [ 13.197158][ T1] PCI: bus1: Fast back to back transfers disabled
> > > [ 13.197208][ T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is
> > > updated to 01
> > > [ 13.197538][ T1] pci 0000:00:01.0: BAR 14: assigned [mem
> > > 0xe0000000-0xe00fffff]
> > > [ 13.197581][ T1] pci 0000:00:01.0: BAR 6: assigned [mem
> > > 0xe0100000-0xe01007ff pref]
> > > [ 13.197624][ T1] pci 0000:01:00.0: BAR 0: assigned [mem
> > > 0xe0000000-0xe000ffff 64bit]
> > > [ 13.197670][ T1] pci 0000:01:00.0: BAR 2: assigned [mem
> > > 0xe0010000-0xe0010fff 64bit]
> > > [ 13.197715][ T1] pci 0000:01:00.0: BAR 4: assigned [mem
> > > 0xe0011000-0xe0011fff 64bit]
> > > [ 13.197759][ T1] pci 0000:00:01.0: PCI bridge to [bus 01]
> > > [ 13.197792][ T1] pci 0000:00:01.0: bridge window [mem
> > > 0xe0000000-0xe00fffff]
> > > [ 13.197985][ T1] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> > > [ 13.198135][ T1] pci 0000:01:00.0: enabling device (0140 -> 0142)
> > >
> > > [ 15.619756][ T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > > [ 15.625694][ T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > > assigned bus number 2
> > > [ 15.635643][ T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1
> > > hci version 0x100 quirks 0x0000000000080010
> > > [ 15.646477][ T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > > [ 15.652432][ T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > > assigned bus number 3
> > > [ 15.660560][ T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
> > >
> > > Note that this board PCI and USB 3.0 have been working in the Linux
> > > kernel for quite many years.
> > >
> > > - These are relevant configs used in the build (not sure if I need to
> > > use CONFIG_DM_PCI_COMPAT?)
> > >
> > > CONFIG_DM=y
> > > CONFIG_CMD_PCI=y
> > > CONFIG_PCI=y
> > > CONFIG_PCI_MVEBU=y
> > > CONFIG_PCI_ENDPOINT=y
> > > CONFIG_DM_PCI_COMPAT=y
> > > CONFIG_USB_XHCI_HCD=y
> > > CONFIG_USB_XHCI_PCI=y
> > > CONFIG_USB_XHCI_MVEBU=y
> >
> > Following options should be enough:
> >
> > CONFIG_CMD_PCI=y
> > CONFIG_PCI=y
> > # CONFIG_DM_PCI_COMPAT is not set
> > CONFIG_PCI_PNP=y
> > CONFIG_PCI_MVEBU=y
>
> Since the PCI bus in this box is used for USB 3.0, I've included
> various XHCI configs to see if I can probe this USB 3.0 flash drive
> also (that's the ultimate test).
>
> >
> > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > should be enough.
> >
> > > BTW, the topic is no longer kwboot, should we move this to another new
> > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> >
> > Changed :-)
>
> Thanks :)
>
> Here is the log of a successful spin up for the PCI bus and USB 3.0
> drive. Please see my commentary ### in between the logs.
>
> U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> Pogoplug V4
>
> SoC: Kirkwood 88F6281_A1
> DRAM: 128 MiB
> device_probe:
> device_probe: exit 0
>
>
> ### The device_probe above seems to have nothing happening.
>
>
> mvebu_pcie_bind: begin
> mvebu_pcie_bind: got an available node
> Bound device to pcie at 82000000
> mvebu_pcie_bind: bound
> mvebu_pcie_bind: exit 0
> Bound device pcie at 82000000 to mbus at f1000000
> Bound device mbus at f1000000 to root_driver
> Bound device ehci at 50000 to ocp at f1000000
> Bound device ethernet-controller at 72000 to ocp at f1000000
> Bound device sata at 80000 to ocp at f1000000
> Bound device mvsdio at 90000.blk to mvsdio at 90000
> Bound device mvsdio at 90000 to ocp at f1000000
> Bound device ocp at f1000000 to root_driver
> NAND: 128 MiB
> MMC: device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
> device_probe:
> mvsdio at 90000: 0
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
>
> In: serial
> Out: serial
> Err: serial
> Net: device_probe:
> device_probe:
>
> Warning: ethernet-controller at 72000 (eth0) using random MAC address -
> 86:b8:84:ee:a9:a5
> device_probe: exit 0
> eth0: ethernet-controller at 72000
> Hit any key to stop autoboot: 0
> Pogo_V4> usb reset
> resetting USB...
> Bus ehci at 50000: device_probe:
> device_probe:
> USB EHCI 1.00
> device_probe: exit 0
> scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> device_probe:
> device_probe:
> Bound device usb_hub to usb_hub
> device_probe:
> device_probe:
> device_probe: exit 0
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 3 USB Device(s) found
> scanning usb for storage devices... 1 Storage Device(s) found
>
>
> ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> 2.0 was up as shown above.
>
>
> Pogo_V4> pci enum
> device_probe:
> device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> mvebu_pcie_probe:
> Cannot add window '4:e8', conflicts with another window
> PCIe unable to add mbus window for mem at 90000000+10000000
> Cannot add window '4:e0', conflicts with another window
> PCIe unable to add mbus window for IO at c0000000+00010000
> mvebu_pcie_probe: exit 0
> Bound device pci_0:0.0 to pcie0.0
> device_probe:
> device_probe:
> Bound device xhci_pci to pci_0:0.0
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
>
> ### So I did the pci enum above to get the XHCI bound to PCIe.
>
> Pogo_V4> usb reset
> resetting USB...
> Bus ehci at 50000: device_probe:
> device_probe:
> USB EHCI 1.00
> device_probe: exit 0
> Bus xhci_pci: device_probe:
> device_probe:
> Register 400081f NbrPorts 4
> Starting the controller
> USB XHCI 1.00
> device_probe: exit 0
> scanning bus ehci at 50000 for devices... Bound device usb_hub to ehci at 50000
> device_probe:
> device_probe:
> Bound device usb_hub to usb_hub
> device_probe:
> device_probe:
> device_probe: exit 0
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 3 USB Device(s) found
> scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> device_probe:
> device_probe:
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 2 USB Device(s) found
> scanning usb for storage devices... 2 Storage Device(s) found
>
>
> ### Now both drives are up!
>
>
> Pogo_V4> pci 0
> device_probe:
> Scanning PCI devices on bus 0
> BusDevFun VendorId DeviceId Device Class Sub-Class
> _____________________________________________________________
> 00.00.00 0x11ab 0x6281 Bridge device 0x04
> Pogo_V4> pci 1
> device_probe:
> Scanning PCI devices on bus 1
> BusDevFun VendorId DeviceId Device Class Sub-Class
> _____________________________________________________________
> 01.00.00 0x1b73 0x1009 Serial bus controller 0x03
>
>
> ### And the pci command has meaningful output at this point as shown above.
>
>
> Pogo_V4> usb tree
> USB device tree:
> 1 Hub (480 Mb/s, 0mA)
> | u-boot EHCI Host Controller
> |
> +-2 Hub (480 Mb/s, 100mA)
> | GenesysLogic USB2.0 Hub
> |
> +-3 Mass Storage (480 Mb/s, 224mA)
> SanDisk Ultra Fit 4C531001560827107320
>
> 1 Hub (5 Gb/s, 0mA)
> | U-Boot XHCI Host Controller
> |
> +-2 Mass Storage (5 Gb/s, 224mA)
> SanDisk Cruzer Glide 3.0 4C530000130418116112
>
> Pogo_V4> usb info
> 1: Hub, USB Revision 2.0
> - u-boot EHCI Host Controller
> - Class: Hub
> - PacketSize: 64 Configurations: 1
> - Vendor: 0x0000 Product 0x0000 Version 1.0
> Configuration: 1
> - Interfaces: 1 Self Powered 0mA
> Interface: 0
> - Alternate Setting 0, Endpoints: 1
> - Class Hub
> - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
>
> 2: Hub, USB Revision 2.0
> - GenesysLogic USB2.0 Hub
> - Class: Hub
> - PacketSize: 64 Configurations: 1
> - Vendor: 0x05e3 Product 0x0610 Version 144.48
> Configuration: 1
> - Interfaces: 1 Self Powered Remote Wakeup 100mA
> Interface: 0
> - Alternate Setting 0, Endpoints: 1
> - Class Hub
> - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
>
> 3: Mass Storage, USB Revision 2.10
> - SanDisk Ultra Fit 4C531001560827107320
> - Class: (from Interface) Mass Storage
> - PacketSize: 64 Configurations: 1
> - Vendor: 0x0781 Product 0x5583 Version 1.0
> Configuration: 1
> - Interfaces: 1 Bus Powered 224mA
> Interface: 0
> - Alternate Setting 0, Endpoints: 2
> - Class Mass Storage, Transp. SCSI, Bulk only
> - Endpoint 1 In Bulk MaxPacket 512
> - Endpoint 2 Out Bulk MaxPacket 512
>
> 1: Hub, USB Revision 3.0
> - U-Boot XHCI Host Controller
> - Class: Hub
> - PacketSize: 512 Configurations: 1
> - Vendor: 0x0000 Product 0x0000 Version 1.0
> Configuration: 1
> - Interfaces: 1 Self Powered 0mA
> Interface: 0
> - Alternate Setting 0, Endpoints: 1
> - Class Hub
> - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
>
> 2: Mass Storage, USB Revision 3.0
> - SanDisk Cruzer Glide 3.0 4C530000130418116112
> - Class: (from Interface) Mass Storage
> - PacketSize: 512 Configurations: 1
> - Vendor: 0x0781 Product 0x5597 Version 1.0
> Configuration: 1
> - Interfaces: 1 Bus Powered 224mA
> Interface: 0
> - Alternate Setting 0, Endpoints: 2
> - Class Mass Storage, Transp. SCSI, Bulk only
> - Endpoint 1 In Bulk MaxPacket 1024
> - Endpoint 2 Out Bulk MaxPacket 1024
>
> ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> active as shown above.
>
> ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
>
>
> Pogo_V4> ext2ls usb 1:1 /
> device_probe:
> device_probe:
> device_probe: exit 0
> <DIR> 4096 .
> <DIR> 4096 ..
> 522666 uboot.2021.07-tld-1.pogo_v4.bin
> 524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
>
> Great works Pali!
>
> I think some more investigation is needed. Why did we need to do "pci
> enum", and then "usb reset", in that order, to get the PCI bus and the
> XHCI controller probed? there must be something missing in the process
> somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> uclass?
I think I can see the order of enumeration. PCI must be enumerated
first, and then XHCI being the controller on this host bus would come
alive? I think we can live with 'pci enum' and 'usb reset' to get the
USB 3.0 drives enumerated. However, it seems just a little bit
unintuitive.
Thanks,
Tony
More information about the U-Boot
mailing list