[RFC] dev_phys_to_bus() and PCI

[Mark Kettenis]

> From: Nicolas Saenz Julienne <nsaenzjulienne at suse.de>
> Date: Mon, 15 Mar 2021 11:26:18 +0100
> 
> Hi Mark!

Hi Nicolas,

> On Sat, 2021-03-13 at 10:24 +0100, Mark Kettenis wrote:
> [...]
> > Fortunately Nicolas Saenz Julienne recently introduced
> > dev_phys_to_bus() and dev_bus_to_phys() interfaces to do this.  Those
> > interfaces make use of a dma-ranges property in the device tree which
> > doesn't work so well for PCI devices though.
> 
> Why doesn't it work with PCI devices? Raspberry Pi 4 has a PCIe bus
> that needs DMA translations. This is handled by parsing its
> 'dma-ranges' property. Here's how rpi4's devicetree looks like[1]:
> 
> 	pcie0: pcie at 7d500000 {
> 		compatible = "brcm,bcm2711-pcie";
> 		ranges = <0x02000000 0x0 0xf8000000 0x6 0x00000000 0x0 0x04000000>;
> 		/*
> 		 * The wrapper around the PCIe block has a bug preventing it
> 		 * from accessing beyond the first 3GB of memory.
> 		 */
> 		dma-ranges = <0x02000000 0x0 0x00000000 0x0 0x00000000 0x0 0xc0000000>;
> 	};

Does that work even though there are no device tree nodes for the PCI
devices themselves?

> 
> > However, the PCI code in U-Boot already has a way to describe DMA address
> > translation through its regions support.
> 
> regions contain a representation of devicetree's 'ranges' property, which
> doesn't describe the DMA address translations, but CPU's view of PCI memory.

It describes both.  The DMA address translations are described by
regions that have the PCI_REGION_SYS_MEMORY flag set.  Some of the PCI
host bridge drivers set up these regions, and the generic DM_PCI code
by default adds such regions for the system RAM banks.  And there is a
fdt_pci_dma_ranges() function to populate a "dma-ranges" property
based on these regions.

The benefit of the PCI regions code is that it can describe more
complex setups where a single translation offset isn't enough.  And
the code will print a warbing if you try to do DMA to memory that isn't "visable" for the PCI device.

In the particular example of the M1 code I don't really want to add a
dma-ranges property to the device tree since the IOMMU setup is only
used by u-boot and not really usable by Linux or another OS as it only
covers a (relatively) small part of system memory.
 
> > diff --git a/include/phys2bus.h b/include/phys2bus.h
> > index 866b8b51a8..13d23ef4bb 100644
> > --- a/include/phys2bus.h
> > +++ b/include/phys2bus.h
> > @@ -23,14 +23,21 @@ static inline unsigned long bus_to_phys(unsigned long bus)
> >  
> > 
> >  #if CONFIG_IS_ENABLED(DM)
> >  #include <dm/device.h>
> > +#include <pci.h>
> >  
> > 
> >  static inline dma_addr_t dev_phys_to_bus(struct udevice *dev, phys_addr_t phys)
> >  {
> > +	if (device_is_on_pci_bus(dev))
> > +		return dm_pci_phys_to_bus(dev, phys, PCI_REGION_SYS_MEMORY);
> > +
> 
> Note that this would break USB support on RPi4.

It should work if the PCI host bridge driver sets up the regions
properly based on the dma-ranges property.  Looks like the current
driver doesn't do this, so I'll have to fix that.

Cheers,

Mark