[U-Boot] Exception Level switching seems broken on RK3399

[Vicente Bergas]

On Thu, May 24, 2018 at 7:05 PM, Dr. Philipp Tomsich
<philipp.tomsich at theobroma-systems.com> wrote:
> Vincente,
>
> On 24 May 2018, at 18:48, Vicente Bergas <vicencb at gmail.com> wrote:
>
> Hello Philipp,
> your answer is much appreciated.
>
> On Thu, May 24, 2018 at 1:07 PM, Dr. Philipp Tomsich
> <philipp.tomsich at theobroma-systems.com> wrote:
>
> Vincente,
>
> On 19 May 2018, at 16:58, Vicente Bergas <vicencb at gmail.com> wrote:
>
> Hello,
> I am writing this from a standalone Sapphire board [1],
> that is, without the Excavator base board.
> The CPU is the Rockchip RK3399, which implements ARMv8.0-A.
>
> Currently the boot process is:
> 1.- Boot ROM
> 2.- SPL, provided as closed source binary blob [2]
>
>
> SPL-support is available in mainline U-Boot.  We developed this for
> the RK3399-Q7 and it has been successfully used on other RK3399
> boards (e.g. I know that some Firefly-users are using this).
>
>
> Thank you!
>
>
> 3.- ATF, closed source binary blob [3]
>    (not using the one from [2] because of stability issues)
>
>
> Why use the closed-source blob, if the RK3399 is supported in the ATF
> mainline and an ATF can be compiled from source?
>
>
> Currently I am using both binary blobs (SPL and ATF) because I could
> not make it work another way. I'll give it another try.
>
>
> 4.- Mainline u-boot, master branch
> 5.- Mainline linux, master branch
>
> I would like to use an opensource boot process.
> As a first approach I try to completely remove the ATF and
> replace the SPL with the one from u-boot.
> The modified boot process looks like:
> 1.- Boot ROM
> 2.- SPL, from mainline u-boot, master branch
> 3.- Mainline u-boot, master branch
> 4.- Mainline linux, master branch
> But it is not working.
>
> The replaced SPL works fine and loads u-boot.
> U-boot also works fine, loads linux and jumps into it.
>
>
> Yes, we’ve done some work to enable us to run U-Boot in EL3 on
> the RK3399 (as we use it for programming the secure e-fuses on
> the RK3399-Q7 in our factory programming setup).
>
>
> I can indeed confirm that U-Boot runs fine in EL3.
>
>
> But then, linux never gets executed.
>
> I have traced the issue to: arch/arm/include/asm/macro.h
> 202: msr  spsr_el3, \tmp
> 203: msr  elr_el3, \ep
> 204: eret // This is the last instruction executed
>
> For testing, I have also set CONFIG_ARMV8_SWITCH_TO_EL1 and
> checked that switch_to_el1 from arch/arm/lib/bootm.c is not reached.
>
> At this point I have a few questions:
> 1.- Is my first approach feasible? That is, is it possible to boot
>    this CPU without ATF?
>
>
> It is feasible (i.e.: requires implementation work) but not recommended:
> Linux will use PSCI to bring up the secondary CPUs.  We have run Linux
> (limited to a single CPU) in EL3 on this CPU during our own board bringup,
> but I would strongly discourage this as it will entail unnecessary effort.
>
>
> There is a misunderstanding here. My intention was to run U-Boot in EL3,
> then switch to EL2 or EL1 from within U-Boot and afterwards run Linux
> in the lower EL.
>
>
> 2.- If so, what should I do to make it work? Probably it is just
>    a configuration issue, but I do not know what to check. [4]
> 3.- Else, why do I need ATF?
>
>
> ATF is the secure monitor on ARMv8 and provides services such as PSCI
> to start up secondary CPUs.  It will usually also be part of
> power-management
> on most SoCs (after all: power configuration needs to be done in the secure
> envelope).
>
>
> Do you mean that without ATF I can only run a single CPU core?
>
>
> Linux (on the RK3399) uses PSCI to start the secondary CPUs and uses SMC
> (secure monitor call) to call into PSCI.  The PSCI handlers thus live within
> ATF.
>
> While it is technically possible to start up the other cores using other
> mechanisms,
> it is the ‘road less travelled’.
>
> Cheers,
> Philipp.
>
>
> Regards,
>  Vicenç.
>
>
> Regards,
> Philipp.
>
>
> Regards,
> Vicenç.

Hello Philipp,
I have managed to make it work. It turns out that support for
this platform was added to mainline after my first attempt.

In doing so I have modified the configuration this way:
--- a/configs/evb-rk3399_defconfig
+++ b/configs/evb-rk3399_defconfig
@@ -11,7 +11,7 @@
 CONFIG_DEBUG_UART=y
 CONFIG_FIT=y
 CONFIG_SPL_LOAD_FIT=y
-CONFIG_SPL_FIT_GENERATOR="arch/arm/mach-rockchip/make_fit_atf.py"
+CONFIG_SPL_FIT_GENERATOR="tools/make_fit_atf"
 # CONFIG_DISPLAY_CPUINFO is not set
 CONFIG_DISPLAY_BOARDINFO_LATE=y
 CONFIG_SPL_STACK_R=y
@@ -27,7 +27,7 @@
 CONFIG_CMD_TIME=y
 CONFIG_SPL_OF_CONTROL=y
 CONFIG_OF_SPL_REMOVE_PROPS="pinctrl-0 pinctrl-names clock-names
interrupt-parent assigned-clocks assigned-clock-rates
assigned-clock-parents"
-CONFIG_SPL_OF_PLATDATA=y
+CONFIG_MKIMAGE_DTC_PATH="scripts/dtc/dtc"
 CONFIG_ENV_IS_IN_MMC=y
 CONFIG_NET_RANDOM_ETHADDR=y
 CONFIG_REGMAP=y

SPL_FIT_GENERATOR and SPL_OF_PLATDATA require python.
In order to remove this dependency:
 1.- I have written a C version for SPL_FIT_GENERATOR.
 2.- Disabled SPL_OF_PLATDATA, it just works.

MKIMAGE_DTC_PATH requires dtc in the PATH.
In order to remove this dependency, I have changed it to use the built-in one.

If there is interest in those changes, I can post the full patch.

Regards,
  Vicenç.