IMX8MM 4GiB boundary issue

[Mark Kettenis]

> Date: Fri, 25 Feb 2022 14:50:59 +0100
> From: Marek Vasut <marex at denx.de>
> 
> On 2/25/22 12:37, Mark Kettenis wrote:
> >> From: Fabio Estevam <festevam at gmail.com>
> >> Date: Fri, 25 Feb 2022 08:12:58 -0300
> >>
> >> Hi Tim,
> >>
> >> On Thu, Feb 24, 2022 at 6:46 PM Tim Harvey <tharvey at gateworks.com> wrote:
> >>
> >>> Fabio,
> >>>
> >>> No, that commit is 'not' in v2021.07. Please test with master and you
> >>> should see that go away.
> >>
> >> Yes, you are right.
> >>
> >>> Regardless, Marek's suggestion is the right fix if you can manage
> >>> that... we really don't want to limit 4GB boards to 3GB. I was hoping
> >>> NXP would step up and address the peripheral drivers for this.
> >>
> >> Agreed, thanks!
> > 
> > But isn't the problem here that (some of) the hardware peripherals
> > simply can't address memory above the 4GB boundary?
> > 
> > OS kernels can work around such limitations by using an IOMMU (if
> > provided by the hardware) or by using bounce buffers (swiotlb in Linux
> > speak).
> 
> Right, see bounce_buffer in U-Boot.
> 
> > The traditional way to deal with this in u-boot is to make
> > sure that u-boot only uses memory below the 4GB boundary by
> > implementing board_get_usable_ram_top() and making sure that all the
> > addresses in the u-boot environment are in "low" memory.
> 
> The board_get_usable_ram_top() purpose was something else entirely at 
> the beginning, it only started being misused to work around driver 
> issues instead of fixing them later and that is utterly wrong.
> 
> > For EFI
> > support there is the CONFIG_EFI_LOADER_BOUNCE_BUFFER option, which
> > should be set to "y" in this case.
> 
> There is generic bounce buffer for drivers, see common/bouncebuf.c .

That implementation only seems to exist to handle misaligned buffers.
As far as I can tell it doesn't make any attempt to make sure it
allocates memory in a specific address range.  Although I suppose that
using memalign() means it allocates from the heap and boards have some
control over where the heap lives.  But doesn't that rely on
board_get_usable_ram_top()?

I'm following this discussion since I'm trying to work out the best
way to add PCIe support for the Apple M1 "boards".  There the issue
isn't so much that the hardware peripherals can't address memory above
the 4GB boundary (there is no memory below the 4GB boundary!).  But
the IOMMU only has a 4GB iova window which means that I cannot have
the IOMMU map all physical memory 1:1.  So I either have to make sure
that U-Boot (including the efi_loader subsystem) only uses memory in a
particular 4GB range.  Or I have to add an interface to have drivers
explictly map memory through the IOMMU (and have them unmap when
they're done).  Such an interface would look somewhat similar to the
bounce buffer interface.