[Question] Reloading pcie + nvme drivers

[Brian McKee]

Hi Gents.

Background: working with the socfpga fork of u-boot-2020.10, I've added a
driver for the Altera V series FPGA PCIe controller to u-boot (borrowed
completely from intel's 10 series FPGA driver and the linux kernel driver).
I got it working through a series of hacks to the pcie and nvme drivers. I
haven't been able to get 2021.04 Denx version to work, I think because the
PCIe -> memory interface is not enabled by 'bridge enable'. I could
probably hack that, but for now I'm sticking with 2020.10 socfpga. Based on
my experiments though, it looks like the hacks I deployed to 2020.10 are
not necessary on 2021.04. I'm waiting for socfpga to port to 2021.04.

I have a problem though. To keep cost down, I have a 2MB QSPI flash device
on the board. And to keep software developers happy, the almost 8 MiB FPGA
image should be stored on the SSD so field upgrades are less likely to
brick the hardware.

I have created a stripped down FPGA that only contains the PCIe controller
and the HPS module. lzma compressed it is less than 300KB. It fits quite
nicely into the free space of the QSPI NOR Flash memory.

I have u-boot installed on the QSPI, booting and I've setup commands to
grab the stripped down FPGA image, decompress it, and load it into the
FPGA. Then initialize PCI and NVME scan and it all works!

However, when I reload the FPGA from the NVME, pcie is okay, but the nvme
crashes. I'm not sure why. It might be getting reset by the FPGA re-load.
All of it's programming disappears and it drops off the bus.

I have proven that if I perform a u-boot 'reset' after the full FPGA is
loaded, do a 'bridge enable', 'pci' and 'nvme scan' the nvme comes back to
life using the full fpga image.

I have been trying to figure out how to trick u-boot into re-initializing
the nvme controller, but I haven't figured out a hack to do so yet.

Ideally, I'd like to be able to unload the nvme driver (and possibly the
pcie driver) and then reload them but there doesn't appear to be a way to
do that in u-boot at this time.

Do any of you guys have any suggestions for what I can try to reinitialize
the nvme driver?

Thanks much for your support.

Brian