IMX8MM 4GiB boundary issue

[Marek Vasut]

On 2/26/22 14:30, Mark Kettenis wrote:
>> Date: Fri, 25 Feb 2022 14:50:59 +0100
>> From: Marek Vasut <marex at denx.de>
>>
>> On 2/25/22 12:37, Mark Kettenis wrote:
>>>> From: Fabio Estevam <festevam at gmail.com>
>>>> Date: Fri, 25 Feb 2022 08:12:58 -0300
>>>>
>>>> Hi Tim,
>>>>
>>>> On Thu, Feb 24, 2022 at 6:46 PM Tim Harvey <tharvey at gateworks.com> wrote:
>>>>
>>>>> Fabio,
>>>>>
>>>>> No, that commit is 'not' in v2021.07. Please test with master and you
>>>>> should see that go away.
>>>>
>>>> Yes, you are right.
>>>>
>>>>> Regardless, Marek's suggestion is the right fix if you can manage
>>>>> that... we really don't want to limit 4GB boards to 3GB. I was hoping
>>>>> NXP would step up and address the peripheral drivers for this.
>>>>
>>>> Agreed, thanks!
>>>
>>> But isn't the problem here that (some of) the hardware peripherals
>>> simply can't address memory above the 4GB boundary?
>>>
>>> OS kernels can work around such limitations by using an IOMMU (if
>>> provided by the hardware) or by using bounce buffers (swiotlb in Linux
>>> speak).
>>
>> Right, see bounce_buffer in U-Boot.
>>
>>> The traditional way to deal with this in u-boot is to make
>>> sure that u-boot only uses memory below the 4GB boundary by
>>> implementing board_get_usable_ram_top() and making sure that all the
>>> addresses in the u-boot environment are in "low" memory.
>>
>> The board_get_usable_ram_top() purpose was something else entirely at
>> the beginning, it only started being misused to work around driver
>> issues instead of fixing them later and that is utterly wrong.
>>
>>> For EFI
>>> support there is the CONFIG_EFI_LOADER_BOUNCE_BUFFER option, which
>>> should be set to "y" in this case.
>>
>> There is generic bounce buffer for drivers, see common/bouncebuf.c .
> 
> That implementation only seems to exist to handle misaligned buffers.
> As far as I can tell it doesn't make any attempt to make sure it
> allocates memory in a specific address range.  Although I suppose that
> using memalign() means it allocates from the heap and boards have some
> control over where the heap lives.  But doesn't that rely on
> board_get_usable_ram_top()?

Possibly, I suspect someone will have to take a deeper look into this 
and maybe implement some better bounce buffer.

> I'm following this discussion since I'm trying to work out the best
> way to add PCIe support for the Apple M1 "boards".  There the issue
> isn't so much that the hardware peripherals can't address memory above
> the 4GB boundary (there is no memory below the 4GB boundary!).  But
> the IOMMU only has a 4GB iova window which means that I cannot have
> the IOMMU map all physical memory 1:1.  So I either have to make sure
> that U-Boot (including the efi_loader subsystem) only uses memory in a
> particular 4GB range.  Or I have to add an interface to have drivers
> explictly map memory through the IOMMU (and have them unmap when
> they're done).  Such an interface would look somewhat similar to the
> bounce buffer interface.

Maybe now is the right time to implement such interface ?
Isn't that what linux uses swiotlb for ?