pcie + nvme + cyclone V

[Brian McKee]

Hello all.

I'm trying to get an NVMe device to work with a PCIe hard core in an Altera
(intel) Cyclone V SOC from u-boot.

It is working well in linux.

I have been hacking on u-boot for about a week and have drawn some
conclusions that may or may not be accurate.

First, let me give you some background.

In linux, the device tree description for the pcie module looks like this:

bridges: bridge at c0000000 {
compatible = "altr,bridge-20.1", "simple-bus";
reg = <0xc0000000 0x20000000>,
      <0xff200000 0x00200000>;
reg-names = "axi_h2f", "axi_h2f_lw";
clocks = <&pcie &pll_65 &pcie &pll_65>;
clock-names = "h2f_axi_clock", "h2f_lw_axi_clock", "f2h_sdram0_clock",
"f2h_sdram1_clock";
#address-cells = <2>;
#size-cells = <1>;
ranges = <0x00000000 0x00200080 0xc0200080 0x00000010>,
<0x00000000 0x00200000 0xc0200000 0x00000080>,
<0x00000000 0x00000000 0xc0000000 0x00100000>,
<0x00000000 0x00100000 0xc0100000 0x00004000>,
<0x00000001 0x00000000 0xff200000 0x00100000>;

pcie: pcie at 000100000 { // The @000100000 signifies nothing!
compatible = "altr,pcie-root-port-20.1", "altr,pcie-root-port-1.0";
reg = <0x00000000 0x00000000 0x00100000>,
      <0x00000000 0x00100000 0x00004000>;
reg-names = "Txs", "Cra";
interrupt-parent = <&intc>;
interrupts = <0 40 4>; // 40 is the interrupt, add 32 to relate to the GIC
document
interrupt-controller;  // 4 is IRQ_TYPE_LEVEL_HIGH, 1 is
IRQ_TYPE_EDGE_RISING
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
#clock-cells = <0>;
clocks = <&clk_100>;
device_type = "pci";
bus-range = <0x00 0xff>;
ranges = <0x82000000 0x0 0x00000000 0x0 0x00000000 0x0 0x00100000>;
msi-parent = <&msi>;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie 1>,
<0 0 0 2 &pcie 2>,
<0 0 0 3 &pcie 3>,
<0 0 0 4 &pcie 4>;
status = "disabled"; // Disabled by default
}; //end pcie at 000100000

msi: msi at 000200080 {
compatible = "altr,msi-1.0";
reg = <0x00000000 0x00200080 0x00000010>,
      <0x00000000 0x00200000 0x00000080>;
reg-names = "csr", "vector_slave";
interrupt-parent = <&intc>;
interrupts = <0 41 4>;
clocks = <&pll_65>;
msi-controller = <1>;
num-vectors = <32>;
status = "disabled"; // Disabled by default
}; //end msi at 000200080

}; //end bridge at c0000000 (bridges)

Sorry, the indenting was lost.

That portion of socfpga.dtsi works in linux. BUS 1 BAR 0 of the PCIe
controller is programmed correctly, offset 0x10 gets an address of
0x00000000 and CPU space is mapped to 0xc0000000.

However, from what I can tell, u-boot can't handle the hierarchical nature
of "simple-bus". It doesn't know how to or the two addresses to get a
resultant CPU memory base address. That's not a big problem, because I can
flatten the tree to look like this:

pcie: pcie at 000100000 {
compatible = "altr,pcie-root-port-1.0";
reg = <0x00000000 0x00100000>,
     <0xC0100000 0x00004000>;
reg-names = "Txs", "Cra";
interrupt-parent = <&intc>;
interrupts = <0 40 4>; // 40 is the interrupt, add 32 to relate to the GIC
document
interrupt-controller;  // 4 is IRQ_TYPE_LEVEL_HIGH, 1 is
IRQ_TYPE_EDGE_RISING
#address-cells = <2>;
#size-cells = <1>;
#interrupt-cells = <1>;
#clock-cells = <0>;
clocks = <&clk_100>;
device_type = "pci";
bus-range  = <0x00 0xff>;
ranges = <0x82000000 0x00000000 0xc0000000 0x00100000>;
//       <32BITNOPF> <PCI ADDR> <HOSTADDR> <MEM  SIZE>
msi-parent = <&msi>;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie 1>,
<0 0 0 2 &pcie 2>,
<0 0 0 3 &pcie 3>,
<0 0 0 4 &pcie 4>;
}; //end pcie at 000100000

msi: msi at C0200080 {
compatible = "altr,msi-1.0";
reg = <0x0 0xC0200080 0x00000010>,
     <0x0 0xC0200000 0x00000080>;
reg-names = "csr", "vector_slave";
interrupt-parent = <&intc>;
interrupts = <0 41 4>;
clocks = <&pll_65>;
msi-controller = <1>;
num-vectors = <32>;
}; //end msi at 000200080

However, this does not work either.

The PCI driver becomes confused by the ranges description: I think because
it does not understand that the pci device perspective can be different
from the CPU perspective. It can't find the memory described in the ranges
variable and I get debug output (plus printfs that I added) that looks like
this:

pciauto_setup_device: bars_num = 6
cfg_dword_read: bus=1, byte_en=0x3, offset=0x0004, data=0x00100000
cfg_dword_write: bus=1, byte_en=0xf, offset=0x0010, data=0xffffffff
cfg_dword_read: bus=1, byte_en=0xf, offset=0x0010, data=0xffffc004
bar_response = 0xffffc004
PCI address space is 64 bit.
cfg_dword_write: bus=1, byte_en=0xf, offset=0x0014, data=0xffffffff
cfg_dword_read: bus=1, byte_en=0xf, offset=0x0014, data=0xffffffff
PCI Autoconfig: BAR 0, Mem, size=0x4000
res->bus_lower = 0x0, size = 0x4000, res->size = 0x100000
addr = 0x0, res->bus_start = 0xc0000000
No room in resource, avail start=0 / size=100000, need=4000
PCI: Failed autoconfig bar 0
bar_value = 0xffffffff

I have hacked pci_auto to get the correct values programmed into BUS 1 BAR
0, however, the nvme driver chokes when it goes and looks for the memory
indicated by BUS 1 BAR 0 and can't find it because the NVMe driver (pci
driver, actually?) needs to be aware of the different mapping of the PCI
address space and CPU address space for the PCIe controller. It looks at
location 0x00000000 in the ranges variable and only sees 0xc0000000.

Frankly, I'm hoping I'm doing it wrong and you guys can come back and say,
"you're doing it all wrong, do this instead..."

But in the event that my hunches are correct, I'm hoping you guys could
give me directions on what code needs to change to support this sort of
configuration. With a little direction, I can figure out how to change it
and submit the code back to you guys for approval.

Any assistance will be greatly appreciated.

Thanks.