[U-Boot] Exception Level switching seems broken on RK3399

Thu May 24 17:05:47 UTC 2018

Vincente,

> On 24 May 2018, at 18:48, Vicente Bergas <vicencb at gmail.com> wrote:
> 
> Hello Philipp,
> your answer is much appreciated.
> 
> On Thu, May 24, 2018 at 1:07 PM, Dr. Philipp Tomsich
> <philipp.tomsich at theobroma-systems.com <mailto:philipp.tomsich at theobroma-systems.com>> wrote:
>> Vincente,
>> 
>>> On 19 May 2018, at 16:58, Vicente Bergas <vicencb at gmail.com> wrote:
>>> 
>>> Hello,
>>> I am writing this from a standalone Sapphire board [1],
>>> that is, without the Excavator base board.
>>> The CPU is the Rockchip RK3399, which implements ARMv8.0-A.
>>> 
>>> Currently the boot process is:
>>> 1.- Boot ROM
>>> 2.- SPL, provided as closed source binary blob [2]
>> 
>> SPL-support is available in mainline U-Boot.  We developed this for
>> the RK3399-Q7 and it has been successfully used on other RK3399
>> boards (e.g. I know that some Firefly-users are using this).
> 
> Thank you!
> 
>> 
>>> 3.- ATF, closed source binary blob [3]
>>>    (not using the one from [2] because of stability issues)
>> 
>> Why use the closed-source blob, if the RK3399 is supported in the ATF
>> mainline and an ATF can be compiled from source?
> 
> Currently I am using both binary blobs (SPL and ATF) because I could
> not make it work another way. I'll give it another try.
> 
>> 
>>> 4.- Mainline u-boot, master branch
>>> 5.- Mainline linux, master branch
>>> 
>>> I would like to use an opensource boot process.
>>> As a first approach I try to completely remove the ATF and
>>> replace the SPL with the one from u-boot.
>>> The modified boot process looks like:
>>> 1.- Boot ROM
>>> 2.- SPL, from mainline u-boot, master branch
>>> 3.- Mainline u-boot, master branch
>>> 4.- Mainline linux, master branch
>>> But it is not working.
>>> 
>>> The replaced SPL works fine and loads u-boot.
>>> U-boot also works fine, loads linux and jumps into it.
>> 
>> Yes, we’ve done some work to enable us to run U-Boot in EL3 on
>> the RK3399 (as we use it for programming the secure e-fuses on
>> the RK3399-Q7 in our factory programming setup).
> 
> I can indeed confirm that U-Boot runs fine in EL3.
> 
>> 
>>> But then, linux never gets executed.
>>> 
>>> I have traced the issue to: arch/arm/include/asm/macro.h
>>> 202: msr  spsr_el3, \tmp
>>> 203: msr  elr_el3, \ep
>>> 204: eret // This is the last instruction executed
>>> 
>>> For testing, I have also set CONFIG_ARMV8_SWITCH_TO_EL1 and
>>> checked that switch_to_el1 from arch/arm/lib/bootm.c is not reached.
>>> 
>>> At this point I have a few questions:
>>> 1.- Is my first approach feasible? That is, is it possible to boot
>>>    this CPU without ATF?
>> 
>> It is feasible (i.e.: requires implementation work) but not recommended:
>> Linux will use PSCI to bring up the secondary CPUs.  We have run Linux
>> (limited to a single CPU) in EL3 on this CPU during our own board bringup,
>> but I would strongly discourage this as it will entail unnecessary effort.
> 
> There is a misunderstanding here. My intention was to run U-Boot in EL3,
> then switch to EL2 or EL1 from within U-Boot and afterwards run Linux
> in the lower EL.
> 
>> 
>>> 2.- If so, what should I do to make it work? Probably it is just
>>>    a configuration issue, but I do not know what to check. [4]
>>> 3.- Else, why do I need ATF?
>> 
>> ATF is the secure monitor on ARMv8 and provides services such as PSCI
>> to start up secondary CPUs.  It will usually also be part of power-management
>> on most SoCs (after all: power configuration needs to be done in the secure
>> envelope).
> 
> Do you mean that without ATF I can only run a single CPU core?

Linux (on the RK3399) uses PSCI to start the secondary CPUs and uses SMC
(secure monitor call) to call into PSCI.  The PSCI handlers thus live within ATF.

While it is technically possible to start up the other cores using other mechanisms,
it is the ‘road less travelled’.

Cheers,
Philipp.

> 
> Regards,
>  Vicenç.
> 
>> 
>> Regards,
>> Philipp.
>> 
>>> 
>>> Regards,
>>> Vicenç.
>>> 
>>> [1] http://opensource.rock-chips.com/wiki_Excavator_sapphire_board
>>> [2] https://github.com/rockchip-linux/rkbin/tree/29mirror
>>> [3] https://raw.githubusercontent.com/nightseas/armbian-build/master/packages/blobs/rockchip64/trust.img
>>> [4]
>>> CONFIG_ARM=y
>>> CONFIG_ARCH_ROCKCHIP=y
>>> CONFIG_SYS_TEXT_BASE=0x00200000
>>> CONFIG_SPL_LIBCOMMON_SUPPORT=y
>>> CONFIG_SPL_LIBGENERIC_SUPPORT=y
>>> CONFIG_SYS_MALLOC_F_LEN=0x4000
>>> CONFIG_ROCKCHIP_RK3399=y
>>> CONFIG_ROCKCHIP_SPL_RESERVE_IRAM=0x4000
>>> CONFIG_SPL_STACK_R_ADDR=0x80000
>>> # CONFIG_PSCI_RESET is not set
>>> CONFIG_CMD_HDMIDETECT=y
>>> CONFIG_DEFAULT_DEVICE_TREE="rk3399-evb"
>>> CONFIG_DEBUG_UART=y
>>> CONFIG_BOOTDELAY=1
>>> # CONFIG_DISPLAY_CPUINFO is not set
>>> # CONFIG_DISPLAY_BOARDINFO is not set
>>> CONFIG_SPL_STACK_R=y
>>> CONFIG_SPL_STACK_R_MALLOC_SIMPLE_LEN=0x4000
>>> # CONFIG_CMD_IMI is not set
>>> # CONFIG_CMD_XIMG is not set
>>> # CONFIG_CMD_SAVEENV is not set
>>> # CONFIG_CMD_FLASH is not set
>>> # CONFIG_CMD_LOADB is not set
>>> # CONFIG_CMD_LOADS is not set
>>> CONFIG_CMD_MMC=y
>>> CONFIG_CMD_USB=y
>>> # CONFIG_CMD_SETEXPR is not set
>>> # CONFIG_CMD_NET is not set
>>> # CONFIG_DOS_PARTITION is not set
>>> # CONFIG_ISO_PARTITION is not set
>>> CONFIG_SPL_OF_CONTROL=y
>>> CONFIG_REGMAP=y
>>> CONFIG_SPL_REGMAP=y
>>> CONFIG_SYSCON=y
>>> CONFIG_SPL_SYSCON=y
>>> CONFIG_CLK=y
>>> CONFIG_SPL_CLK=y
>>> CONFIG_ROCKCHIP_GPIO=y
>>> CONFIG_SYS_I2C_ROCKCHIP=y
>>> # CONFIG_MMC_WRITE is not set
>>> # CONFIG_MMC_HW_PARTITIONING is not set
>>> CONFIG_MMC_DW=y
>>> CONFIG_MMC_SDHCI=y
>>> CONFIG_MMC_SDHCI_ROCKCHIP=y
>>> CONFIG_DM_ETH=y
>>> # CONFIG_NETDEVICES is not set
>>> CONFIG_PINCTRL=y
>>> CONFIG_SPL_PINCTRL=y
>>> CONFIG_PINCTRL_ROCKCHIP_RK3399=y
>>> CONFIG_DM_PMIC=y
>>> CONFIG_PMIC_RK8XX=y
>>> CONFIG_REGULATOR_PWM=y
>>> CONFIG_DM_REGULATOR_FIXED=y
>>> CONFIG_REGULATOR_RK8XX=y
>>> CONFIG_PWM_ROCKCHIP=y
>>> CONFIG_RAM=y
>>> CONFIG_SPL_RAM=y
>>> CONFIG_BAUDRATE=1500000
>>> CONFIG_DEBUG_UART_BASE=0xFF1A0000
>>> CONFIG_DEBUG_UART_CLOCK=24000000
>>> CONFIG_DEBUG_UART_SHIFT=2
>>> CONFIG_SYSRESET=y
>>> CONFIG_USB=y
>>> CONFIG_USB_XHCI_HCD=y
>>> CONFIG_USB_XHCI_DWC3=y
>>> CONFIG_USB_EHCI_HCD=y
>>> CONFIG_USB_EHCI_GENERIC=y
>>> CONFIG_USB_STORAGE=y
>>> CONFIG_USB_KEYBOARD=y
>>> CONFIG_USB_HOST_ETHER=y
>>> CONFIG_USB_ETHER_ASIX=y
>>> CONFIG_USB_ETHER_ASIX88179=y
>>> CONFIG_USB_ETHER_MCS7830=y
>>> CONFIG_USB_ETHER_RTL8152=y
>>> CONFIG_USB_ETHER_SMSC95XX=y
>>> CONFIG_DM_VIDEO=y
>>> # CONFIG_BACKLIGHT_PWM is not set
>>> # CONFIG_VIDEO_BPP8 is not set
>>> # CONFIG_VIDEO_BPP16 is not set
>>> # CONFIG_VIDEO_BPP32 is not set
>>> # CONFIG_VIDEO_ANSI is not set
>>> # CONFIG_SYS_WHITE_ON_BLACK is not set
>>> CONFIG_DISPLAY=y
>>> CONFIG_VIDEO_ROCKCHIP=y
>>> CONFIG_DISPLAY_ROCKCHIP_HDMI=y
>>> # CONFIG_FAT_WRITE is not set
>>> CONFIG_USE_TINY_PRINTF=y
>>> CONFIG_ERRNO_STR=y
>>> # CONFIG_EFI_LOADER is not set