32-bit DMA limit for devices (and drivers)

[Bin Meng]

Hi Andre,

On Sun, May 2, 2021 at 8:22 AM Andre Przywara <andre.przywara at arm.com> wrote:
>
> On Sat, 1 May 2021 14:23:32 +0200 (CEST)
> Mark Kettenis <mark.kettenis at xs4all.nl> wrote:
>
> Hi,
>
> > > From: Bin Meng <bmeng.cn at gmail.com>
> > > Date: Sat, 1 May 2021 19:45:02 +0800
> > >
> > > On Fri, Apr 30, 2021 at 7:22 PM Andre Przywara <andre.przywara at arm.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > We now see the first Allwinner devices [1] having DRAM located above
> > > > 4GB in address space (4GB DRAM starting at 1GB). After one fix[2]
> > > > this works somewhat fine, but the sun8i-emac network device is still
> > > > limited to 32-bit DMA addresses. With U-Boot relocating itself (plus
> > > > stack and heap) to the end of DRAM, it now runs completely beyond 4GB
> > > > on those machines, so not giving pure 32-bit addresses for buffers
> > > > anymore.
> > > > In Linux we handle this easily by just keeping the default DMA
> > > > mask at 32 bits, and letting the DMA framework deal with the nasty
> > > > details.
> > > >
> > > > I was wondering how this should be handled in U-Boot? The straight
> > > > forward solution would be:
> > > > - Let the driver allocate the RX and TX buffers separately, placing them
> > > >   below 4GB in the address space (using lmb_reserve(), I guess?)
> > > > - Use those RX buffers and hand the addresses back to the upper layers.
> > > > - We already copy TX packets, so this would also be covered, in this
> > > >   situation. Other drivers might need to introduce copying.
> > > >
> > > > This sounds like a common problem, so I was wondering if there is a
> > > > more generic solution to this? Maybe there are already platforms or
> > > > devices affected? Or should the whole heap and stack be moved below 4GB
> > > > (if this is easily possible)?
> > >
> > > My understanding is that the relocated address of U-Boot should be
> > > below 4GB then there is no problem for the 32-bit DMA. I thought this
> > > is a rule to be followed by every board, but this is not the case on
> > > your board?
>
> Bin, interesting, where is this coming from? Was this originally for
> 32-bit CPUs with some address extension (PAE/LPAE)? I think on *sane*
> 64-bit systems there would be no need for this restriction, except maybe
> for this 32-bit DMA limitation (which is more of a device problem).

Please have a look at x86 and riscv target codes
board_get_usable_ram_top() which limits the relocated address below
4G. I remember U-Boot shell does not support parsing 64-bit digits
too.

>
> > Yes, that was my impression as well.  And I think that would work fine
> > on this board as there is plenty of DRAM below 4GB.  And this can be
> > achieved by implementing the board_get_usable_ram_top() function.
>
> Ah, I think this is the thing I missed and was looking for:
> So we *can* restrict everything *U-Boot* to 32 bits and save us a lot of
> hassle.
>
> Thanks for that hint!
>
> > As I indicated in my reply, some care is needed in the EFI subsystem,
> > but there already is a solution for that.  There is
> > CONFIG_EFI_LOADER_BOUNCE_BUFFER, but that might not actually be needed
> > in this case.  By default the EFI subsystem will mark all conventional
> > memory above "ram_top" as EFI_BOOT_SERVICES_DATA.  So EFI applications
> > uch as OS loaders will not allocate that memory until they've called
> > ExitBootServices() at which point U-Boot will be completely out of the
> > picture.
>
> Oh nice, this looks like what I need. So EFI apps would never use this
> memory for I/O buffers.
>
> So I gave this a try and this solves my problem quite neatly: Linux
> sees the full DRAM, but U-Boot never touches anything beyond 4GB.
> Briefly tested Linux with both EFI and booti.
> Will include the board_get_usable_ram_top() implementation in the v2 of
> my 4GB enablement patch.
>
> Thanks again!

Regards,
Bin