[Question] Reloading pcie + nvme drivers

[Simon Glass]

Hi Brian,

On Sat, 8 May 2021 at 09:34, Brian McKee <raydude at gmail.com> wrote:
>
> Thanks Simon,
>
> The NVMe driver does work, at least for me. There are some issues with cache and Device Tree mapping, but I think those are handled in 2021.04.
>

[please try to avoid posting your reply at the top]

> I attempted to add a remove function to the nvme code. I patterned it after a couple of sata drivers remove commands. It looks like this:
>
> static int nvme_remove(struct udevice *udev)
> {
>         struct nvme_dev *dev = dev_get_priv(udev);
>         free(dev->prp_pool);
>         nvme_free_queues(dev, 0);
> return 0;
> }
>
> However, I don't understand how it is called. I looked into the cmd interface but I got lost in the linker_lists.h code. I'm a Hardware Engineer and my lack of C experience is showing.

Just be kind to the next software engineer who asks to borrow your
soldering iron.

>
> It seems to me that since the nvme is a block device, there should be a function associated with the block commands for stopping or unbinding the blk device and that should call nvme.remove automatically. I dug around in the code looking for the linkage but was not able to find it.
>
> Can you or one of the other experts give me a pointer to a bit of C code that shows the linkage and an example of how to unbind the block device?

I think you need to call device_remove(dev, 0) on your device

If you have a removal flag set for your device, such as
DM_FLAG_OS_PREPARE, you can call device_remove(dm_root(),
DM_REMOVE_OS_PREPARE) to remove all devices with that flag.

>
> I also thought about putting a back door command into the probe function, but I have to find the existing udevice in order to call the probe function. Also I hesitate because that seems like the wrong way to go.

uclass_first_device(UCLASS_NVME, &dev)

should return the first NVME device.


>
> Any assistance would be greatly appreciated.
>
> On another note: I have created a driver for PCIe on Cyclone V. It should work for all V series Altera (intel) FPGAs. I kept the driver separate from the 10 series PCIe driver published by intel because I feel that they may want to modify that code in the future for a new series of FPGAs and I don't want to interfere with that process.
>
> Can I submit it as a patch to this list, or is there a different way to submit patches for review?

You can always send patches to the list. If there are dependencies you
can explain them in the patch.
>
> Thanks much for your support,
>
> Brian
>

Regards,
Simon

> On Fri, May 7, 2021 at 12:39 PM Simon Glass <sjg at chromium.org> wrote:
>>
>> Hi Brian,
>>
>> On Fri, 7 May 2021 at 11:03, Brian McKee <raydude at gmail.com> wrote:
>> >
>> > Hi Gents.
>> >
>> > Background: working with the socfpga fork of u-boot-2020.10, I've added a driver for the Altera V series FPGA PCIe controller to u-boot (borrowed completely from intel's 10 series FPGA driver and the linux kernel driver). I got it working through a series of hacks to the pcie and nvme drivers. I haven't been able to get 2021.04 Denx version to work, I think because the PCIe -> memory interface is not enabled by 'bridge enable'. I could probably hack that, but for now I'm sticking with 2020.10 socfpga. Based on my experiments though, it looks like the hacks I deployed to 2020.10 are not necessary on 2021.04. I'm waiting for socfpga to port to 2021.04.
>> >
>> > I have a problem though. To keep cost down, I have a 2MB QSPI flash device on the board. And to keep software developers happy, the almost 8 MiB FPGA image should be stored on the SSD so field upgrades are less likely to brick the hardware.
>> >
>> > I have created a stripped down FPGA that only contains the PCIe controller and the HPS module. lzma compressed it is less than 300KB. It fits quite nicely into the free space of the QSPI NOR Flash memory.
>> >
>> > I have u-boot installed on the QSPI, booting and I've setup commands to grab the stripped down FPGA image, decompress it, and load it into the FPGA. Then initialize PCI and NVME scan and it all works!
>> >
>> > However, when I reload the FPGA from the NVME, pcie is okay, but the nvme crashes. I'm not sure why. It might be getting reset by the FPGA re-load. All of it's programming disappears and it drops off the bus.
>> >
>> > I have proven that if I perform a u-boot 'reset' after the full FPGA is loaded, do a 'bridge enable', 'pci' and 'nvme scan' the nvme comes back to life using the full fpga image.
>> >
>> > I have been trying to figure out how to trick u-boot into re-initializing the nvme controller, but I haven't figured out a hack to do so yet.
>> >
>> > Ideally, I'd like to be able to unload the nvme driver (and possibly the pcie driver) and then reload them but there doesn't appear to be a way to do that in u-boot at this time.
>> >
>> > Do any of you guys have any suggestions for what I can try to reinitialize the nvme driver?
>>
>> You could check if the remove() method of the nvme is implemented. If
>> not you could implement it. Does the NVME depend on the FPGA? Are
>> there any reset lines controlled by the FPGA?
>>
>> I tried NVMe on an Intel device (a Chromebook) a while back and it
>> just hung. I haven't fiddled with it since.
>>
>> >
>> > Thanks much for your support.
>> >
>> > Brian
>>
>> Regards,
>> Simon
>
>
>
> --
> -- Consciousness moves everything.