Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 11:18:18 Tom Rini wrote:
> On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> > On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > > during startup.
> > > > > > > > > > > > 
> > > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > > 
> > > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > > size and crashes.
> > > > > > > > > > > > 
> > > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > > 
> > > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > > to u64.
> > > > > > > > > > > > 
> > > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > > set.
> > > > > > > > > > > > 
> > > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > > 
> > > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > > 
> > > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > > 
> > > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > > 
> > > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > > 
> > > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > > 
> > > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > > 
> > > > > > > > Ok, I will try to explain it again.
> > > > > > > > 
> > > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > > 
> > > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > > 
> > > > > > > > It is more clear what is the problem?
> > > > > > > > 
> > > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > > stored into 32-bit unsigned type.
> > > > > > > 
> > > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > > 
> > > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > > 
> > > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > > 
> > > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > > 
> > > > Yes!
> > > 
> > > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > > support what you're trying to do.  If anyone at NXP knows something
> > > perhaps they'll chime in.
> > 
> > Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.
> 
> I disagree.  It's working as intended with other 32bit platforms, unless

It really cannot work. See my previous emails with exact description. It
is mathematical limitation that you cannot store 4GB size of RAM into
U-Boot's gd structure. Maximal number which can be stored into ulong
type on 32-bit platforms is 2^32-1 which is less than 4GB.

So if U-Boot detects 4GB DDR module it crashes prior it tries to "limit"
usage just to e.g. 2GB.

> it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
> unfortunately).  If you can't configure the memory controller to know
> that 4GB exists, but limit yourself to running within 2GB, that's a
> SoC-specific problem.

Have you really read what I wrote? It is not SoC-specific problem. Full
size of RAM (not mapped memory by U-Boot which may be less than full
size of RAM) is stored in gd->ram_size variable. And this variable is
cast to ulong type which cannot store 4GB. This is common U-Boot code,
not something SoC specific.

Plus CONFIG_MAX_MEM_MAPPED is ignored in some cases which I also
described in previous email. And code for ignoring is again **not** SoC
specific, but generic in common U-Boot code.

> If you want to make the big changes so that we
> can have a larger than 2GB memory space for U-Boot, period, that's a big
> general change with challenges as you've already seen.
> 
> -- 
> Tom