ARM A53 and initial MMU mapping for EL0/1/2/3 ?

Fri Feb 11 02:26:19 CET 2022

On Fri, 11 Feb 2022 00:22:25 +0000
Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:

> On Thu, 2022-02-10 at 22:43 +0000, Andre Przywara wrote:
> > On Thu, 10 Feb 2022 21:58:30 +0000
> > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > 
> > Hi,
> >   
> > > On Thu, 2022-02-10 at 10:22 +0000, Andre Przywara wrote:  
> > > > On Wed, 9 Feb 2022 12:03:47 +0000
> > > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > > 
> > > > Hi,
> > > >     
> > > > > On Wed, 2022-02-09 at 10:45 +0000, Andre Przywara wrote:    
> > > > > > On Wed, 9 Feb 2022 08:35:04 +0000
> > > > > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > > > > 
> > > > > > Hi,
> > > > > >       
> > > > > > > On Wed, 2022-02-09 at 00:33 +0000, Andre Przywara wrote:      
> > > > > > > > On Tue, 8 Feb 2022 22:05:00 +0000
> > > > > > > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > > > > > > 
> > > > > > > > Hi Joakim,
> > > > > > > >         
> > > > > > > > > Trying to figure out how I should map the MMU for normal RAM so it acessible
> > > > > > > > > from all ELx security states.        
> > > > > > > > 
> > > > > > > >        ^^^^^^^
> > > > > > > > 
> > > > > > > > This does not make much sense. U-Boot is typically running in one
> > > > > > > > exception level only, and sets up the page table for exactly that EL.
> > > > > > > > Each EL uses a separate translation regime (with some twists for stage
> > > > > > > > 2 EL2 and combined EL1/0, plus VHE). If you map your memory in EL3, then
> > > > > > > > drop to EL2, the EL3 page tables become irrelevant.
> > > > > > > > 
> > > > > > > > So in U-Boot we just set up the page tables for the EL we are running
> > > > > > > > in, and leave the paging for the lower exception levels to be set up at
> > > > > > > > the discretion of our payloads (kernels, hypervisors).
> > > > > > > > 
> > > > > > > > Please not that *secure* memory is a separate concept, and handled by
> > > > > > > > external hardware, typically using regions, not page tables.        
> > > > > > > 
> > > > > > > I am a beginner w.r.t ARM and Secure/Non secure so thank you for above.
> > > > > > > 
> > > > > > > The problem I have is that I boot a custom SOC into u-boot and when u-boot tries
> > > > > > > to boot linux I get an error exception when u-boot calls armv8_switch_to_el2 to enter linux.      
> > > > > > 
> > > > > > So that means that U-Boot runs in EL3, is that the first and only firmware
> > > > > > that you run? I think the EL3 part of U-Boot is not widely used and tested
> > > > > > beyond the very few platforms that use it.      
> > > > > 
> > > > > Yes, u-boot is first firmware and runs in EL3(ATM, may change once initial bringup is complete) 
> > > > > Maybe u-boot then lacks some critical init? Do you have an example of a board in u-boot
> > > > > that starts in EL3(from reset) using an A53 cpu?     
> > > > 
> > > > As you have probably figured out by now, the whole Layerscape family uses
> > > > that approach. However most other platforms go with Trusted-Firmware as the
> > > > EL3 setup and secure runtime service provider, so the U-Boot EL3 code in
> > > > here is not well tested or looked after. For initial bringup it might be
> > > > OK, but maybe the problems you run into are due to issues in this code.
> > > >     
> > > > > > Do you have the exact address that fails? That should be in ELR, it would
> > > > > > be great if you can pinpoint the exact instruction in macro.h that fails.      
> > > > > 
> > > > > Yes, the address is the first address where kernel is loaded and you can branch there without problems.    
> > > > 
> > > > You mean if you load the kernel and branch to the entry point, it starts
> > > > running, but crashes as soon as it realises that in runs in EL3?
> > > >     
> > > > > It is the eret instruction(last insn in macro armv8_switch_to_el2_m) that fails.    
> > > > 
> > > > Interesting. Maybe there is something missing in the EL2 setup, but my
> > > > understanding is that this is the part that is actually used by
> > > > Layerscape, for instance.
> > > >     
> > > > > > > I think the exception means "Instruction Abort taken without a change in Exception level."
> > > > > > > I was thinking it could be some privilege missing in MMU map.      
> > > > > > 
> > > > > > Could be. One thing that made me wonder is your rather miserly mapping of
> > > > > > only 32MB, which sounds a bit on the small side. Typically we just map the      
> > > > > 
> > > > > We only have 32 MB ATM :( a bit small but it may increase to 64MB    
> > > > 
> > > > That sounds very miserly. Can you actually run an arm64 Linux kernel with
> > > > that little RAM? IIRC for QEMU we need at least 128 MB, and I haven't seen
> > > > an ARMv8 hardware platform with less than 512MB (maybe 256MB) DRAM yet.
> > > >     
> > > > > > whole first DRAM bank, regardless of whether you actually have memory
> > > > > > there or not. U-Boot should know how much DRAM you have, so will not go
> > > > > > beyond that. Having page tables covering more address space does not
> > > > > > really hurt, but avoids all kind of problems.
> > > > > > And please note that U-Boot loves to move things around: itself from the
> > > > > > load address to the end of DRAM (that it knows of); possibly the kernel,
> > > > > > when the alignment is not right, or the DT and initrd if it sees fit.
> > > > > > So there is little point in mapping just portions of the memory.      
> > > > > 
> > > > > U-boot moves around a lot, I know :) In this case u-boot lives
> > > > > in is own 4MB SRAM but kernel lives in a 32MB HyperRAM.    
> > > > 
> > > > Interesting. I wonder if this works well with U-Boot's memory management,
> > > > which assumes it has quite some DRAM to play with.    
> > > 
> > > Found it, all memory spaces were set to secure mode, the req. spec does not agree :(  
> > 
> > Ah, yes, if the DRAM is configured as secure only, running in EL2
> > (always non-secure on the A53) will not end well.
> >   
> > > Anyhow, now kernel enters into EL2 then EL1 to EL0, all is well until kernel tries
> > > to do simple cache ops like dc ivac, x0 or mrs x3,ctr_el0 when I again just get an error exception:
> > >   EXC [0x400] Synchronous Lower EL using AArch64  
> > 
> > Was this with Linux, or some other kernel? IIRC cache maintenance  
> 
> Yes, 5.14.x

Ah, I see. And that really runs with 32MB? I think we need at least
64MB. Maybe the issues you see are related to that? IIRC the effects can
look rather random.

> > instructions in EL0 need to be enabled in SCTLR_EL1 (.UCI and .DZE, for
> > instance, plus maybe more registers), and those and other operations
> > should not be trapped to EL2 as well.  
> 
> SCTLR_EL1 is 0x30500800 and does not seem to match with above. looks like it is kernel that sets this reg?
> how can kernel get that wrong ?

That can't be really the kernel value, because the MMU needs to be on
(bit 0). Is this the reset value, read in U-Boot? The kernel sets those
bits, check the definition of INIT_SCTLR_EL1_MMU_ON in the kernel
source.
Maybe (the generic EL3) U-Boot code misses to set some EL3 registers,
so some stuff is blocked already there, and the kernel is helpless?

> > > so there is something missing here but I cannot figure out what.
> > > u-boot really don't know about the 32 MB RAM kernel is using,  
> > 
> > But you did map this RAM, and probably loaded the kernel in there? So
> > U-Boot would need to touch this memory. Maybe it left something behind?
> > PSCI handlers? UEFI runtime services?  
> 
> u-boot has this mapped and can branch to it. Kernel is an ITS img loaded so
> kernel exec XIP.

It's not really XIP per se, since it's in DRAM. arm64 Linux doesn't
support real XIP (from ROM) anyway. And U-Boot would still move it
around if it's not properly aligned. TEXT_OFFSET in the arm64 5.14
kernel header is probably 0, so you would need to load it at an 2MB
aligned address (so either right at the start or 2MB into DRAM) to avoid
copying. The old 512K offset is not anymore with modern kernels.

> No PSCI or UEFI configured.

I guessed so ...

Cheers,
Andre

> > > it booted just its own 4MB
> > > and I guessing u-boot then missed to configure something for the 32 MB RAM but what ?  
> > 
> > I really don't understand enough of your setup to be of help here.
> > 
> > Cheers,
> > Andre  
> 