[U-Boot] Exception Level switching seems broken on RK3399

[Dr. Philipp Tomsich]

Vincente,

> On 1 Jun 2018, at 00:28, Vicente Bergas <vicencb at gmail.com> wrote:
> 
> On Thu, May 24, 2018 at 7:05 PM, Dr. Philipp Tomsich
> <philipp.tomsich at theobroma-systems.com> wrote:
>> Vincente,
>> 
>> On 24 May 2018, at 18:48, Vicente Bergas <vicencb at gmail.com> wrote:
>> 
>> Hello Philipp,
>> your answer is much appreciated.
>> 
>> On Thu, May 24, 2018 at 1:07 PM, Dr. Philipp Tomsich
>> <philipp.tomsich at theobroma-systems.com> wrote:
>> 
>> Vincente,
>> 
>> On 19 May 2018, at 16:58, Vicente Bergas <vicencb at gmail.com> wrote:
>> 
>> Hello,
>> I am writing this from a standalone Sapphire board [1],
>> that is, without the Excavator base board.
>> The CPU is the Rockchip RK3399, which implements ARMv8.0-A.
>> 
>> Currently the boot process is:
>> 1.- Boot ROM
>> 2.- SPL, provided as closed source binary blob [2]
>> 
>> 
>> SPL-support is available in mainline U-Boot.  We developed this for
>> the RK3399-Q7 and it has been successfully used on other RK3399
>> boards (e.g. I know that some Firefly-users are using this).
>> 
>> 
>> Thank you!
>> 
>> 
>> 3.- ATF, closed source binary blob [3]
>>   (not using the one from [2] because of stability issues)
>> 
>> 
>> Why use the closed-source blob, if the RK3399 is supported in the ATF
>> mainline and an ATF can be compiled from source?
>> 
>> 
>> Currently I am using both binary blobs (SPL and ATF) because I could
>> not make it work another way. I'll give it another try.
>> 
>> 
>> 4.- Mainline u-boot, master branch
>> 5.- Mainline linux, master branch
>> 
>> I would like to use an opensource boot process.
>> As a first approach I try to completely remove the ATF and
>> replace the SPL with the one from u-boot.
>> The modified boot process looks like:
>> 1.- Boot ROM
>> 2.- SPL, from mainline u-boot, master branch
>> 3.- Mainline u-boot, master branch
>> 4.- Mainline linux, master branch
>> But it is not working.
>> 
>> The replaced SPL works fine and loads u-boot.
>> U-boot also works fine, loads linux and jumps into it.
>> 
>> 
>> Yes, we’ve done some work to enable us to run U-Boot in EL3 on
>> the RK3399 (as we use it for programming the secure e-fuses on
>> the RK3399-Q7 in our factory programming setup).
>> 
>> 
>> I can indeed confirm that U-Boot runs fine in EL3.
>> 
>> 
>> But then, linux never gets executed.
>> 
>> I have traced the issue to: arch/arm/include/asm/macro.h
>> 202: msr  spsr_el3, \tmp
>> 203: msr  elr_el3, \ep
>> 204: eret // This is the last instruction executed
>> 
>> For testing, I have also set CONFIG_ARMV8_SWITCH_TO_EL1 and
>> checked that switch_to_el1 from arch/arm/lib/bootm.c is not reached.
>> 
>> At this point I have a few questions:
>> 1.- Is my first approach feasible? That is, is it possible to boot
>>   this CPU without ATF?
>> 
>> 
>> It is feasible (i.e.: requires implementation work) but not recommended:
>> Linux will use PSCI to bring up the secondary CPUs.  We have run Linux
>> (limited to a single CPU) in EL3 on this CPU during our own board bringup,
>> but I would strongly discourage this as it will entail unnecessary effort.
>> 
>> 
>> There is a misunderstanding here. My intention was to run U-Boot in EL3,
>> then switch to EL2 or EL1 from within U-Boot and afterwards run Linux
>> in the lower EL.
>> 
>> 
>> 2.- If so, what should I do to make it work? Probably it is just
>>   a configuration issue, but I do not know what to check. [4]
>> 3.- Else, why do I need ATF?
>> 
>> 
>> ATF is the secure monitor on ARMv8 and provides services such as PSCI
>> to start up secondary CPUs.  It will usually also be part of
>> power-management
>> on most SoCs (after all: power configuration needs to be done in the secure
>> envelope).
>> 
>> 
>> Do you mean that without ATF I can only run a single CPU core?
>> 
>> 
>> Linux (on the RK3399) uses PSCI to start the secondary CPUs and uses SMC
>> (secure monitor call) to call into PSCI.  The PSCI handlers thus live within
>> ATF.
>> 
>> While it is technically possible to start up the other cores using other
>> mechanisms,
>> it is the ‘road less travelled’.
>> 
>> Cheers,
>> Philipp.
>> 
>> 
>> Regards,
>> Vicenç.
>> 
>> 
>> Regards,
>> Philipp.
>> 
>> 
>> Regards,
>> Vicenç.
> 
> Hello Philipp,
> I have managed to make it work. It turns out that support for
> this platform was added to mainline after my first attempt.
> 
> In doing so I have modified the configuration this way:
> --- a/configs/evb-rk3399_defconfig
> +++ b/configs/evb-rk3399_defconfig
> @@ -11,7 +11,7 @@
> CONFIG_DEBUG_UART=y
> CONFIG_FIT=y
> CONFIG_SPL_LOAD_FIT=y
> -CONFIG_SPL_FIT_GENERATOR="arch/arm/mach-rockchip/make_fit_atf.py"
> +CONFIG_SPL_FIT_GENERATOR="tools/make_fit_atf"
> # CONFIG_DISPLAY_CPUINFO is not set
> CONFIG_DISPLAY_BOARDINFO_LATE=y
> CONFIG_SPL_STACK_R=y
> @@ -27,7 +27,7 @@
> CONFIG_CMD_TIME=y
> CONFIG_SPL_OF_CONTROL=y
> CONFIG_OF_SPL_REMOVE_PROPS="pinctrl-0 pinctrl-names clock-names
> interrupt-parent assigned-clocks assigned-clock-rates
> assigned-clock-parents"
> -CONFIG_SPL_OF_PLATDATA=y
> +CONFIG_MKIMAGE_DTC_PATH="scripts/dtc/dtc"
> CONFIG_ENV_IS_IN_MMC=y
> CONFIG_NET_RANDOM_ETHADDR=y
> CONFIG_REGMAP=y
> 
> SPL_FIT_GENERATOR and SPL_OF_PLATDATA require python.
> In order to remove this dependency:
> 1.- I have written a C version for SPL_FIT_GENERATOR.
> 2.- Disabled SPL_OF_PLATDATA, it just works.
> 
> MKIMAGE_DTC_PATH requires dtc in the PATH.
> In order to remove this dependency, I have changed it to use the built-in one.
> 
> If there is interest in those changes, I can post the full patch.

Please submit a patch—it certainly looks worthwhile.

Thanks,
Philipp.