[Question] Reloading pcie + nvme drivers

[Simon Glass]

Hi Brian,

On Fri, 7 May 2021 at 11:03, Brian McKee <raydude at gmail.com> wrote:
>
> Hi Gents.
>
> Background: working with the socfpga fork of u-boot-2020.10, I've added a driver for the Altera V series FPGA PCIe controller to u-boot (borrowed completely from intel's 10 series FPGA driver and the linux kernel driver). I got it working through a series of hacks to the pcie and nvme drivers. I haven't been able to get 2021.04 Denx version to work, I think because the PCIe -> memory interface is not enabled by 'bridge enable'. I could probably hack that, but for now I'm sticking with 2020.10 socfpga. Based on my experiments though, it looks like the hacks I deployed to 2020.10 are not necessary on 2021.04. I'm waiting for socfpga to port to 2021.04.
>
> I have a problem though. To keep cost down, I have a 2MB QSPI flash device on the board. And to keep software developers happy, the almost 8 MiB FPGA image should be stored on the SSD so field upgrades are less likely to brick the hardware.
>
> I have created a stripped down FPGA that only contains the PCIe controller and the HPS module. lzma compressed it is less than 300KB. It fits quite nicely into the free space of the QSPI NOR Flash memory.
>
> I have u-boot installed on the QSPI, booting and I've setup commands to grab the stripped down FPGA image, decompress it, and load it into the FPGA. Then initialize PCI and NVME scan and it all works!
>
> However, when I reload the FPGA from the NVME, pcie is okay, but the nvme crashes. I'm not sure why. It might be getting reset by the FPGA re-load. All of it's programming disappears and it drops off the bus.
>
> I have proven that if I perform a u-boot 'reset' after the full FPGA is loaded, do a 'bridge enable', 'pci' and 'nvme scan' the nvme comes back to life using the full fpga image.
>
> I have been trying to figure out how to trick u-boot into re-initializing the nvme controller, but I haven't figured out a hack to do so yet.
>
> Ideally, I'd like to be able to unload the nvme driver (and possibly the pcie driver) and then reload them but there doesn't appear to be a way to do that in u-boot at this time.
>
> Do any of you guys have any suggestions for what I can try to reinitialize the nvme driver?

You could check if the remove() method of the nvme is implemented. If
not you could implement it. Does the NVME depend on the FPGA? Are
there any reset lines controlled by the FPGA?

I tried NVMe on an Intel device (a Chromebook) a while back and it
just hung. I haven't fiddled with it since.

>
> Thanks much for your support.
>
> Brian

Regards,
Simon