ARM A53 and initial MMU mapping for EL0/1/2/3 ?

[Joakim Tjernlund]

On Wed, 2022-02-09 at 10:45 +0000, Andre Przywara wrote:
> On Wed, 9 Feb 2022 08:35:04 +0000
> Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> 
> Hi,
> 
> > On Wed, 2022-02-09 at 00:33 +0000, Andre Przywara wrote:
> > > On Tue, 8 Feb 2022 22:05:00 +0000
> > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > 
> > > Hi Joakim,
> > >   
> > > > Trying to figure out how I should map the MMU for normal RAM so it acessible
> > > > from all ELx security states.  
> > > 
> > >        ^^^^^^^
> > > 
> > > This does not make much sense. U-Boot is typically running in one
> > > exception level only, and sets up the page table for exactly that EL.
> > > Each EL uses a separate translation regime (with some twists for stage
> > > 2 EL2 and combined EL1/0, plus VHE). If you map your memory in EL3, then
> > > drop to EL2, the EL3 page tables become irrelevant.
> > > 
> > > So in U-Boot we just set up the page tables for the EL we are running
> > > in, and leave the paging for the lower exception levels to be set up at
> > > the discretion of our payloads (kernels, hypervisors).
> > > 
> > > Please not that *secure* memory is a separate concept, and handled by
> > > external hardware, typically using regions, not page tables.  
> > 
> > I am a beginner w.r.t ARM and Secure/Non secure so thank you for above.
> > 
> > The problem I have is that I boot a custom SOC into u-boot and when u-boot tries
> > to boot linux I get an error exception when u-boot calls armv8_switch_to_el2 to enter linux.
> 
> So that means that U-Boot runs in EL3, is that the first and only firmware
> that you run? I think the EL3 part of U-Boot is not widely used and tested
> beyond the very few platforms that use it.

Yes, u-boot is first firmware and runs in EL3(ATM, may change once initial bringup is complete) 
Maybe u-boot then lacks some critical init? Do you have an example of a board in u-boot
that starts in EL3(from reset) using an A53 cpu? 
> 
> Do you have the exact address that fails? That should be in ELR, it would
> be great if you can pinpoint the exact instruction in macro.h that fails.

Yes, the address is the first address where kernel is loaded and you can branch there without problems.
It is the eret instruction(last insn in macro armv8_switch_to_el2_m) that fails.

> 
> > I think the exception means "Instruction Abort taken without a change in Exception level."
> > I was thinking it could be some privilege missing in MMU map.
> 
> Could be. One thing that made me wonder is your rather miserly mapping of
> only 32MB, which sounds a bit on the small side. Typically we just map the

We only have 32 MB ATM :( a bit small but it may increase to 64MB

> whole first DRAM bank, regardless of whether you actually have memory
> there or not. U-Boot should know how much DRAM you have, so will not go
> beyond that. Having page tables covering more address space does not
> really hurt, but avoids all kind of problems.
> And please note that U-Boot loves to move things around: itself from the
> load address to the end of DRAM (that it knows of); possibly the kernel,
> when the alignment is not right, or the DT and initrd if it sees fit.
> So there is little point in mapping just portions of the memory.

U-boot moves around a lot, I know :) In this case u-boot lives
in is own 4MB SRAM but kernel lives in a 32MB HyperRAM.

> 
> Cheers,
> Andre
> 
> > Note that kernel boots from a different memory region than u-boot but those two regions
> > are mapped the same way.
> > 
> >  Jocke
> > 
> > PS.
> > I can cheat/hack u-boot to branch to the linux kernel address and that works until kernel
> > tries to switch EL.
> > 
> > >   
> > > > So far I have this mem_map: 
> > > > 
> > > > 		/* memory mapped RAM. 32MB  */
> > > > 		.virt = 0x60000000UL,
> > > > 		.phys = 0x60000000UL,
> > > > 		.size = 0x02000000UL,
> > > > 		.attrs = PTE_BLOCK_MEMTYPE(MT_NORMAL) | PTE_BLOCK_INNER_SHARE
> > > > 
> > > > but starting to doubt that is correct, can someone suggest what to put in the .attrs field?  
> > > 
> > > Those are the correct attributes for normal ("cache-able") memory.
> > > However you probably need at least another mapping for MMIO accesses,
> > > which MUST NOT be cacheable (MT_DEVICE_NGNRNE), but device memory.
> > > See the beginning of arch/arm/mach-sunxi/board.c for an example.
> > > 
> > > Cheers,
> > > Andre  
> > 
>