ARM A53 and initial MMU mapping for EL0/1/2/3 ?

[Joakim Tjernlund]

On Thu, 2022-02-10 at 10:22 +0000, Andre Przywara wrote:
> On Wed, 9 Feb 2022 12:03:47 +0000
> Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> 
> Hi,
> 
> > On Wed, 2022-02-09 at 10:45 +0000, Andre Przywara wrote:
> > > On Wed, 9 Feb 2022 08:35:04 +0000
> > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > 
> > > Hi,
> > >   
> > > > On Wed, 2022-02-09 at 00:33 +0000, Andre Przywara wrote:  
> > > > > On Tue, 8 Feb 2022 22:05:00 +0000
> > > > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > > > 
> > > > > Hi Joakim,
> > > > >     
> > > > > > Trying to figure out how I should map the MMU for normal RAM so it acessible
> > > > > > from all ELx security states.    
> > > > > 
> > > > >        ^^^^^^^
> > > > > 
> > > > > This does not make much sense. U-Boot is typically running in one
> > > > > exception level only, and sets up the page table for exactly that EL.
> > > > > Each EL uses a separate translation regime (with some twists for stage
> > > > > 2 EL2 and combined EL1/0, plus VHE). If you map your memory in EL3, then
> > > > > drop to EL2, the EL3 page tables become irrelevant.
> > > > > 
> > > > > So in U-Boot we just set up the page tables for the EL we are running
> > > > > in, and leave the paging for the lower exception levels to be set up at
> > > > > the discretion of our payloads (kernels, hypervisors).
> > > > > 
> > > > > Please not that *secure* memory is a separate concept, and handled by
> > > > > external hardware, typically using regions, not page tables.    
> > > > 
> > > > I am a beginner w.r.t ARM and Secure/Non secure so thank you for above.
> > > > 
> > > > The problem I have is that I boot a custom SOC into u-boot and when u-boot tries
> > > > to boot linux I get an error exception when u-boot calls armv8_switch_to_el2 to enter linux.  
> > > 
> > > So that means that U-Boot runs in EL3, is that the first and only firmware
> > > that you run? I think the EL3 part of U-Boot is not widely used and tested
> > > beyond the very few platforms that use it.  
> > 
> > Yes, u-boot is first firmware and runs in EL3(ATM, may change once initial bringup is complete) 
> > Maybe u-boot then lacks some critical init? Do you have an example of a board in u-boot
> > that starts in EL3(from reset) using an A53 cpu? 
> 
> As you have probably figured out by now, the whole Layerscape family uses
> that approach. However most other platforms go with Trusted-Firmware as the
> EL3 setup and secure runtime service provider, so the U-Boot EL3 code in
> here is not well tested or looked after. For initial bringup it might be
> OK, but maybe the problems you run into are due to issues in this code.
> 
> > > Do you have the exact address that fails? That should be in ELR, it would
> > > be great if you can pinpoint the exact instruction in macro.h that fails.  
> > 
> > Yes, the address is the first address where kernel is loaded and you can branch there without problems.
> 
> You mean if you load the kernel and branch to the entry point, it starts
> running, but crashes as soon as it realises that in runs in EL3?
> 
> > It is the eret instruction(last insn in macro armv8_switch_to_el2_m) that fails.
> 
> Interesting. Maybe there is something missing in the EL2 setup, but my
> understanding is that this is the part that is actually used by
> Layerscape, for instance.
> 
> > > > I think the exception means "Instruction Abort taken without a change in Exception level."
> > > > I was thinking it could be some privilege missing in MMU map.  
> > > 
> > > Could be. One thing that made me wonder is your rather miserly mapping of
> > > only 32MB, which sounds a bit on the small side. Typically we just map the  
> > 
> > We only have 32 MB ATM :( a bit small but it may increase to 64MB
> 
> That sounds very miserly. Can you actually run an arm64 Linux kernel with
> that little RAM? IIRC for QEMU we need at least 128 MB, and I haven't seen
> an ARMv8 hardware platform with less than 512MB (maybe 256MB) DRAM yet.
> 
> > > whole first DRAM bank, regardless of whether you actually have memory
> > > there or not. U-Boot should know how much DRAM you have, so will not go
> > > beyond that. Having page tables covering more address space does not
> > > really hurt, but avoids all kind of problems.
> > > And please note that U-Boot loves to move things around: itself from the
> > > load address to the end of DRAM (that it knows of); possibly the kernel,
> > > when the alignment is not right, or the DT and initrd if it sees fit.
> > > So there is little point in mapping just portions of the memory.  
> > 
> > U-boot moves around a lot, I know :) In this case u-boot lives
> > in is own 4MB SRAM but kernel lives in a 32MB HyperRAM.
> 
> Interesting. I wonder if this works well with U-Boot's memory management,
> which assumes it has quite some DRAM to play with.

Found it, all memory spaces were set to secure mode, the req. spec does not agree :(

Anyhow, now kernel enters into EL2 then EL1 to EL0, all is well until kernel tries
to do simple cache ops like dc ivac, x0 or mrs x3,ctr_el0 when I again just get an error exception:
  EXC [0x400] Synchronous Lower EL using AArch64
so there is something missing here but I cannot figure out what.
u-boot really don't know about the 32 MB RAM kernel is using, it booted just its own 4MB
and I guessing u-boot then missed to configure something for the 32 MB RAM but what ?

 Jocke