[Question] Reloading pcie + nvme drivers

[Brian McKee]

Thanks Simon,

The NVMe driver does work, at least for me. There are some issues with
cache and Device Tree mapping, but I think those are handled in 2021.04.

I attempted to add a remove function to the nvme code. I patterned it after
a couple of sata drivers remove commands. It looks like this:

static int nvme_remove(struct udevice *udev)
{
        struct nvme_dev *dev = dev_get_priv(udev);
        free(dev->prp_pool);
        nvme_free_queues(dev, 0);
return 0;
}

However, I don't understand how it is called. I looked into the cmd
interface but I got lost in the linker_lists.h code. I'm a Hardware
Engineer and my lack of C experience is showing.

It seems to me that since the nvme is a block device, there should be a
function associated with the block commands for stopping or unbinding the
blk device and that should call nvme.remove automatically. I dug around in
the code looking for the linkage but was not able to find it.

Can you or one of the other experts give me a pointer to a bit of C code
that shows the linkage and an example of how to unbind the block device?

I also thought about putting a back door command into the probe function,
but I have to find the existing udevice in order to call the probe
function. Also I hesitate because that seems like the wrong way to go.

Any assistance would be greatly appreciated.

On another note: I have created a driver for PCIe on Cyclone V. It should
work for all V series Altera (intel) FPGAs. I kept the driver separate from
the 10 series PCIe driver published by intel because I feel that they may
want to modify that code in the future for a new series of FPGAs and I
don't want to interfere with that process.

Can I submit it as a patch to this list, or is there a different way to
submit patches for review?

Thanks much for your support,

Brian

On Fri, May 7, 2021 at 12:39 PM Simon Glass <sjg at chromium.org> wrote:

> Hi Brian,
>
> On Fri, 7 May 2021 at 11:03, Brian McKee <raydude at gmail.com> wrote:
> >
> > Hi Gents.
> >
> > Background: working with the socfpga fork of u-boot-2020.10, I've added
> a driver for the Altera V series FPGA PCIe controller to u-boot (borrowed
> completely from intel's 10 series FPGA driver and the linux kernel driver).
> I got it working through a series of hacks to the pcie and nvme drivers. I
> haven't been able to get 2021.04 Denx version to work, I think because the
> PCIe -> memory interface is not enabled by 'bridge enable'. I could
> probably hack that, but for now I'm sticking with 2020.10 socfpga. Based on
> my experiments though, it looks like the hacks I deployed to 2020.10 are
> not necessary on 2021.04. I'm waiting for socfpga to port to 2021.04.
> >
> > I have a problem though. To keep cost down, I have a 2MB QSPI flash
> device on the board. And to keep software developers happy, the almost 8
> MiB FPGA image should be stored on the SSD so field upgrades are less
> likely to brick the hardware.
> >
> > I have created a stripped down FPGA that only contains the PCIe
> controller and the HPS module. lzma compressed it is less than 300KB. It
> fits quite nicely into the free space of the QSPI NOR Flash memory.
> >
> > I have u-boot installed on the QSPI, booting and I've setup commands to
> grab the stripped down FPGA image, decompress it, and load it into the
> FPGA. Then initialize PCI and NVME scan and it all works!
> >
> > However, when I reload the FPGA from the NVME, pcie is okay, but the
> nvme crashes. I'm not sure why. It might be getting reset by the FPGA
> re-load. All of it's programming disappears and it drops off the bus.
> >
> > I have proven that if I perform a u-boot 'reset' after the full FPGA is
> loaded, do a 'bridge enable', 'pci' and 'nvme scan' the nvme comes back to
> life using the full fpga image.
> >
> > I have been trying to figure out how to trick u-boot into
> re-initializing the nvme controller, but I haven't figured out a hack to do
> so yet.
> >
> > Ideally, I'd like to be able to unload the nvme driver (and possibly the
> pcie driver) and then reload them but there doesn't appear to be a way to
> do that in u-boot at this time.
> >
> > Do any of you guys have any suggestions for what I can try to
> reinitialize the nvme driver?
>
> You could check if the remove() method of the nvme is implemented. If
> not you could implement it. Does the NVME depend on the FPGA? Are
> there any reset lines controlled by the FPGA?
>
> I tried NVMe on an Intel device (a Chromebook) a while back and it
> just hung. I haven't fiddled with it since.
>
> >
> > Thanks much for your support.
> >
> > Brian
>
> Regards,
> Simon
>


-- 
-- Consciousness moves everything.