Broken support for 4GB DDR on 32-bit platforms

[Pali Rohár]

On Wednesday 18 May 2022 11:35:18 Tom Rini wrote:
> On Wed, May 18, 2022 at 05:27:09PM +0200, Pali Rohár wrote:
> > On Wednesday 18 May 2022 11:18:18 Tom Rini wrote:
> > > On Wed, May 18, 2022 at 03:19:19PM +0200, Pali Rohár wrote:
> > > > On Wednesday 18 May 2022 08:19:36 Tom Rini wrote:
> > > > > On Wed, May 18, 2022 at 02:17:39PM +0200, Pali Rohár wrote:
> > > > > > On Wednesday 18 May 2022 08:16:55 Tom Rini wrote:
> > > > > > > On Wed, May 18, 2022 at 12:58:38PM +0200, Pali Rohár wrote:
> > > > > > > > On Tuesday 17 May 2022 12:38:43 Tom Rini wrote:
> > > > > > > > > On Tue, May 17, 2022 at 06:00:16PM +0200, Pali Rohár wrote:
> > > > > > > > > > On Tuesday 17 May 2022 11:52:14 Tom Rini wrote:
> > > > > > > > > > > On Mon, May 16, 2022 at 11:56:51PM +0200, Pali Rohár wrote:
> > > > > > > > > > > > On Monday 16 May 2022 08:31:43 Tom Rini wrote:
> > > > > > > > > > > > > On Sat, May 14, 2022 at 01:00:06AM +0200, Pali Rohár wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hello! I tried to enable support for 2GB+ of DDR memory (with 4GB DDR3)
> > > > > > > > > > > > > > on powerpc P2020 board in 32-bit addressing mode and U-Boot crashed
> > > > > > > > > > > > > > during startup.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I figured out that issue is not powerpc specific, but rather generic to
> > > > > > > > > > > > > > all 32-bit platforms. U-Boot stores memory size into phys_size_t type
> > > > > > > > > > > > > > (gd->ram_size) and last mapped memory address increased by one byte into
> > > > > > > > > > > > > > phys_addr_t type (gd->ram_top).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Despite size 4GB fits into 32-bit addressing mode, it does not fit into
> > > > > > > > > > > > > > above two variables, and it overflows to zero. U-Boot then see zero RAM
> > > > > > > > > > > > > > size and crashes.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I tried to workaround this issue by changing both phys_size_t and
> > > > > > > > > > > > > > phys_addr_t types to 64-bit. But it did not helped because U-Boot on
> > > > > > > > > > > > > > many places cast gd->ram_size or gd->ram_top to ulong type, which is
> > > > > > > > > > > > > > 32-bit on 32-bit platforms.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Next I changed ulong parameters of board_get_usable_ram_top() function
> > > > > > > > > > > > > > to u64.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > This still was not enough because config value CONFIG_MAX_MEM_MAPPED is
> > > > > > > > > > > > > > ignored on one important place -- in function get_effective_memsize().
> > > > > > > > > > > > > > This config value takes effect only when also CONFIG_VERY_BIG_RAM is
> > > > > > > > > > > > > > set.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Finally With this change I was able to start U-Boot with more than 2GB
> > > > > > > > > > > > > > of DDR memory inserted in SODIMM slot on P2020.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > How to fix issues with gd->ram_size and gd->ram_top? That +1 byte is
> > > > > > > > > > > > > > really stupid limitation. Changing phys_size_t and phys_addr_t types
> > > > > > > > > > > > > > unconditionally to 64-bit? Or something else?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > And what is the purpose of CONFIG_VERY_BIG_RAM config option? Why is
> > > > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED check skipped in get_effective_memsize() function,
> > > > > > > > > > > > > > but is not skipped on many more places?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > So, there's two parts to this, as I recall it all.  First, even on 64bit
> > > > > > > > > > > > > platforms we contain ourselves to 32bit address space and even then
> > > > > > > > > > > > > something within the "old" 2GB window.  We then set a CONFIG option to
> > > > > > > > > > > > > not mess with the memory node in DT which has the real value.  Second,
> > > > > > > > > > > > > for 32bit platforms which can support 4GB memory, or more, some further
> > > > > > > > > > > > > games need to be played, typically I believe around initializing the
> > > > > > > > > > > > > memory controller (I'm more confident of that for dra7xx_evm, which I
> > > > > > > > > > > > > don't have the big memory version of, just a small memory one) so that
> > > > > > > > > > > > > Linux can do whatever needs doing to enable "36bit" typically address
> > > > > > > > > > > > > support.  Looking at the other P*36BIT* configs might give you some more
> > > > > > > > > > > > > clues about what to do on your platform, or at least who might still be
> > > > > > > > > > > > > able to explain and test things on the PowerPC side.
> > > > > > > > > > > > 
> > > > > > > > > > > > I know about 36-bit addressing on e500v2 but I'm not going to enable it
> > > > > > > > > > > > due to performance reasons (see Freescale AN4064 [1]). So I want to
> > > > > > > > > > > > stick with 32-bit addressing for 2GB+ memory usage (around 3GB; it is
> > > > > > > > > > > > 4GB minus memory used by peripherals; which is still more than 2GB).
> > > > > > > > > > > > 
> > > > > > > > > > > > And due to 32-bit type for phys_size_t, phys_addr_t and casting these
> > > > > > > > > > > > types to ulong is an issue. Plus issue with CONFIG_VERY_BIG_RAM and
> > > > > > > > > > > > CONFIG_MAX_MEM_MAPPED as I already wrote.
> > > > > > > > > > > 
> > > > > > > > > > > I'm not seeing the problem, sorry.  You run U-Boot in the normal 2GB
> > > > > > > > > > 
> > > > > > > > > > Ok, I will try to explain it again.
> > > > > > > > > > 
> > > > > > > > > > I want to run U-Boot in normal 2GB area. U-Boot normally store detected
> > > > > > > > > > RAM size into the gd->ram_size structure. And here is the issue.
> > > > > > > > > > 
> > > > > > > > > > 32-bit unsigned C type cannot represent maximal 32-bit addressable size.
> > > > > > > > > > U-Boot tries to store 4GB value = 0x100000000 into gd->ram_size which
> > > > > > > > > > overflows to 0x00000000. And U-Boot then crashes because it expects that
> > > > > > > > > > can store some data at address "gd->ram_size - few_kb" as it expects
> > > > > > > > > > that RAM is in the area [0, gd->ram_size).
> > > > > > > > > > 
> > > > > > > > > > It is more clear what is the problem?
> > > > > > > > > > 
> > > > > > > > > > If I theoretically find 3.9GB RAM module (but such probably does not
> > > > > > > > > > exist) then U-Boot should work in normal 2GB area as number 3.9GB can be
> > > > > > > > > > stored into 32-bit unsigned type.
> > > > > > > > > 
> > > > > > > > > Yes, you need to disable CONFIG_ARCH_FIXUP_FDT_MEMORY
> > > > > > > > 
> > > > > > > > CONFIG_ARCH_FIXUP_FDT_MEMORY affects booting OS.
> > > > > > > > 
> > > > > > > > The issue is that **u-boot** is crashing during its initialization.
> > > > > > > 
> > > > > > > When configuring the controller for 4GB, but limiting U-Boot to 2GB?
> > > > > > 
> > > > > > Yes!
> > > > > 
> > > > > Interesting.  Maybe you just have to do the 36BIT games on that SoC, to
> > > > > support what you're trying to do.  If anyone at NXP knows something
> > > > > perhaps they'll chime in.
> > > > 
> > > > Why? It is issue in U-Boot and affects all 32-bit platforms not only powerpc/NXP.
> > > 
> > > I disagree.  It's working as intended with other 32bit platforms, unless
> > 
> > It really cannot work. See my previous emails with exact description. It
> > is mathematical limitation that you cannot store 4GB size of RAM into
> > U-Boot's gd structure. Maximal number which can be stored into ulong
> > type on 32-bit platforms is 2^32-1 which is less than 4GB.
> > 
> > So if U-Boot detects 4GB DDR module it crashes prior it tries to "limit"
> > usage just to e.g. 2GB.
> 
> Yes, this is a semi-intentional limitation of our design history.  We're
> still in the very old days of a 2GB:2GB address space split.

This is just default value in option CONFIG_MAX_MEM_MAPPED or in
function get_effective_memsize().

> > > it's stopped working on dra7xx_evm with >2GB memory (which I don't have,
> > > unfortunately).  If you can't configure the memory controller to know
> > > that 4GB exists, but limit yourself to running within 2GB, that's a
> > > SoC-specific problem.
> > 
> > Have you really read what I wrote? It is not SoC-specific problem. Full
> > size of RAM (not mapped memory by U-Boot which may be less than full
> > size of RAM) is stored in gd->ram_size variable. And this variable is
> > cast to ulong type which cannot store 4GB. This is common U-Boot code,
> > not something SoC specific.
> > 
> > Plus CONFIG_MAX_MEM_MAPPED is ignored in some cases which I also
> > described in previous email. And code for ignoring is again **not** SoC
> > specific, but generic in common U-Boot code.
> 
> I guess no, I'm not seeing everything you've wrote, and you're not
> seeing everything I wrote either.  If you have a 32bit platform and more
> than 2GB of memory, you don't tell U-Boot that you have more than 2GB of
> memory because it's not going to work right.

And this is what I'm writing since beginning. It does not work because
of issues in U-Boot code with ulong type which I described in first
email.

> You lie, one way or
> another, and have U-Boot work inside a 2GB window, and do whatever you
> need so that the OS will see the correct amount and do the right thing.
> All of the options you've noted I believe are related to the kludges
> that have been required thus far to either enable 36bit support on some
> PowerPC platforms (which you've explained you don't want to do) or the
> 36bit support for some ARM platforms (of which I'm asking around to see
> if someone can re-test on current mainline).  I don't _think_ we have an
> option today that would mirror the 1GB:3GB (peripheral:memory) split
> that was an option at some point for Linux, as how to start handling
> bigmem systems.

This option is there and I'm writing about it in every email. Option
CONFIG_MAX_MEM_MAPPED. Also there is function get_effective_memsize()
with default implementation which use CONFIG_MAX_MEM_MAPPED, but not
always. And this function defines that split.

> What I keep reading what you want to do is to support 4GB of memory
> without doing some kludges.  I have no idea how much work would be
> required to enable that, at this point.
> 
> -- 
> Tom

Please look again at my first email in this thread. I have already wrote
what is wrong there.

The issue is really only in ulong type and in config option
CONFIG_VERY_BIG_RAM which I have not understand how and why is used and
why default get_effective_memsize() ignores CONFIG_MAX_MEM_MAPPED.