[RFC PATCH 00/16] arm: Add Rockchip RK3588 support

Mark Kettenis mark.kettenis at xs4all.nl
Mon Mar 13 16:34:02 CET 2023 

> Date: Mon, 13 Mar 2023 17:21:05 +0200
> From: Eugen Hristev <eugen.hristev at collabora.com>
> 
> On 3/13/23 17:07, Mark Kettenis wrote:
> >> Date: Mon, 13 Mar 2023 16:21:36 +0200
> >> From: Eugen Hristev <eugen.hristev at collabora.com>
> >>
> >> On 3/13/23 12:00, Jonas Karlman wrote:
> >>> On 2023-03-13 09:42, Eugen Hristev wrote:
> >>>> On 3/13/23 00:34, Jonas Karlman wrote:
> >>>>> Hi Eugen,
> >>>>>
> >>>>> On 2023-03-08 09:57, Eugen Hristev wrote:
> >>>>>> On 1/29/23 11:04, Jonas Karlman wrote:
> >>>>>>> On 2023-01-27 14:21, Jagan Teki wrote:
> >>>>>>>> On Fri, 27 Jan 2023 at 05:13, Jonas Karlman <jonas at kwiboo.se> wrote:
> >>>>>>>>>
> >>>>>>>>> On 2023-01-26 23:16, Jonas Karlman wrote:
> >>>>>>>>>> Hi Jagan,
> >>>>>>>>>> On 2023-01-26 20:17, Jagan Teki wrote:
> >>>>>>>>>>> On Fri, 27 Jan 2023 at 00:33, Jonas Karlman <jonas at kwiboo.se> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 2023-01-26 19:26, Jagan Teki wrote:
> >>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, 26 Jan 2023 at 23:34, Simon Glass <sjg at chromium.org> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jagan,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 10:42, Jagan Teki <jagan at edgeble.ai> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 22:28, Jonas Karlman <jonas at kwiboo.se> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jagan,
> >>>>>>>>>>>>>>>> On 2023-01-26 17:51, Jagan Teki wrote:
> >>>>>>>>>>>>>>>>> Hi Jonas,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, 26 Jan 2023 at 04:17, Jonas Karlman <jonas at kwiboo.se> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jagan,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 2023-01-25 23:27, Jagan Teki wrote:
> >>>>>>>>>>>>>>>>>>> This series support Rockchip RK3588. All the device tree files are
> >>>>>>>>>>>>>>>>>>> synced from linux-next with the proper SHA1 mentioned in the commit
> >>>>>>>>>>>>>>>>>>> messages.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Unfortunately, the BL31 from rkbin is not compatible with U-Boot so
> >>>>>>>>>>>>>>>>>>> it is failing to load ATF entry from SPL and hang.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Verified below BL31 versions,
> >>>>>>>>>>>>>>>>>>>       bl31-v1.15
> >>>>>>>>>>>>>>>>>>>       bl31-v1.21
> >>>>>>>>>>>>>>>>>>>       bl31-v1.22
> >>>>>>>>>>>>>>>>>>>       bl31-v1.23
> >>>>>>>>>>>>>>>>>>>       bl31-v1.24
> >>>>>>>>>>>>>>>>>>>       bl31-v1.25
> >>>>>>>>>>>>>>>>>>>       bl31-v1.26
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> < snip >
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> As you can see in the logs above there is timeout waiting for data.
> >>>>>>>>>>
> >>>>>>>>>> I managed to find the issue and have a workaround that gets me longer
> >>>>>>>>>> in the boot process, there still seem to be other issue with the rk3588
> >>>>>>>>>> startup.
> >>>>>>>>>>
> >>>>>>>>>> --------
> >>>>>>>>>> U-Boot SPL 2023.01 (Jan 26 2023 - 22:03:00 +0000)
> >>>>>>>>>> Trying to boot from MMC1
> >>>>>>>>>> ## Checking hash(es) for config config-1 ... OK
> >>>>>>>>>> ## Checking hash(es) for Image atf-1 ... sha256+ OK
> >>>>>>>>>> ## Checking hash(es) for Image u-boot ... sha256+ OK
> >>>>>>>>>> ## Checking hash(es) for Image fdt-1 ... sha256+ OK
> >>>>>>>>>> ## Checking hash(es) for Image atf-2 ... sha256+ OK
> >>>>>>>>>> ## Checking hash(es) for Image atf-3 ... sha256+ OK
> >>>>>>>>>> INFO:    Preloader serial: 2
> >>>>>>>>>> NOTICE:  BL31: v2.3():v2.3-468-ge529a2760:derrick.huang
> >>>>>>>>>> NOTICE:  BL31: Built : 09:59:49, Nov 21 2022
> >>>>>>>>>> INFO:    spec: 0x1
> >>>>>>>>>> INFO:    ext 32k is not valid
> >>>>>>>>>> INFO:    ddr: stride-en 4CH
> >>>>>>>>>> INFO:    GICv3 without legacy support detected.
> >>>>>>>>>> INFO:    ARM GICv3 driver initialized in EL3
> >>>>>>>>>> INFO:    valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0
> >>>>>>>>>> INFO:    system boots from cpu-hwid-0
> >>>>>>>>>> INFO:    idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001
> >>>>>>>>>> INFO:    dfs DDR fsp_params[0].freq_mhz= 2112MHz
> >>>>>>>>>> INFO:    dfs DDR fsp_params[1].freq_mhz= 528MHz
> >>>>>>>>>> INFO:    dfs DDR fsp_params[2].freq_mhz= 1068MHz
> >>>>>>>>>> INFO:    dfs DDR fsp_params[3].freq_mhz= 1560MHz
> >>>>>>>>>> INFO:    BL31: Initialising Exception Handling Framework
> >>>>>>>>>> INFO:    BL31: Initializing runtime services
> >>>>>>>>>> WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
> >>>>>>>>>> ERROR:   Error initializing runtime service opteed_fast
> >>>>>>>>>> INFO:    BL31: Preparing for EL3 exit to normal world
> >>>>>>>>>> INFO:    Entry point address = 0xa00000
> >>>>>>>>>> INFO:    SPSR = 0x3c9
> >>>>>>>>>> "Synchronous Abort" handler, esr 0x96000000
> >>>>>>>>>> elr: 0000000000a23650 lr : 0000000000a24d9c
> >>>>>>>>>> x0 : 0000000000b7fbe8 x1 : 350003402a0003f3
> >>>>>>>>>> x2 : 0000000000000000 x3 : 0000000000b80ff0
> >>>>>>>>>> x4 : 0000000000b80ff0 x5 : 0000000000b80e88
> >>>>>>>>>> x6 : 0000000000000054 x7 : 0000000000000044
> >>>>>>>>>> x8 : 000000000000000a x9 : 0000000000000000
> >>>>>>>>>> x10: 0000000000000034 x11: 0000000000000002
> >>>>>>>>>> x12: 0000000000001988 x13: 0000000000b7fadc
> >>>>>>>>>> x14: 0000000000a7e808 x15: 0000000000a7e808
> >>>>>>>>>> x16: 0000000000000000 x17: 0000000000000000
> >>>>>>>>>> x18: 0000000000b7fe50 x19: 0000000000b7fbe8
> >>>>>>>>>> x20: 000000003c14dc00 x21: 000000003c14dc00
> >>>>>>>>>> x22: 0000000000a7e808 x23: 0000000000000000
> >>>>>>>>>> x24: 0000000000000000 x25: 0000000000000000
> >>>>>>>>>> x26: 0000000000000000 x27: 0000000000000000
> >>>>>>>>>> x28: 0000000000000000 x29: 0000000000b7fb80
> >>>>>>>>>>
> >>>>>>>>>> Code: f90013f5 f9400001 b4000201 f9400021 (f9403435)
> >>>>>>>>>> Resetting CPU ...
> >>>>>>>>>> --------
> >>>>>>>>>>
> >>>>>>>>>> This was running on top of u-boot-dm/master 060a65e899859dcbf42049a18be20ce7118e7c0e
> >>>>>>>>>> with some rk3568 patches and this series, see [1].
> >>>>>>>>>>
> >>>>>>>>>> The last 3 commits contains workaround to issue with sdmmc clock.
> >>>>>>>>>> dwmmc driver set sclk to (uint)-2, my workaround just adds a
> >>>>>>>>>> fallback to default 400khz clock.
> >>>>>>>>>>
> >>>>>>>>>> Next issue is the sync abort, looks it happens when u-boot
> >>>>>>>>>> tries to set clock rates based on devicetree. this is the
> >>>>>>>>>> last debug line before the crash.
> >>>>>>>>>>
> >>>>>>>>>> clk_set_rate(clk=0000000000b7fba8, rate=1008000000)
> >>>>>>>>>
> >>>>>>>>> With the commit at [2] I can now successfully run U-Boot proper.
> >>>>>>>>>
> >>>>>>>>> Source of the two main issues to get this series to run have been the scmi clocks.
> >>>>>>>>> Vendor u-boot first load its scmi driver before trying to set the cpu clocks.
> >>>>>>>>> We can just remove it and leave setting a faster cpu rate to linux.
> >>>>>>>>>
> >>>>>>>>> I also noticed that my sdram size series only detect the first two channels of memory,
> >>>>>>>>> will respin a v2 of that series to add detection of all 4 channels of memory.
> >>>>>>>>>
> >>>>>>>>> [2] https://github.com/Kwiboo/u-boot-rockchip/commit/7e4fee945e725c0ec9ef3905c41ce367e77833b3
> >>>>>>>>>
> >>>>>>>> Okay. We need to find a way to handle the clock value 400Khz
> >>>>>>>> generically via the CLK framework or eMMC can be worth checking as it
> >>>>>>>> doesn't involve SCMI and have a working patch set before MW. I did
> >>>>>>>> that and was able to detect eMMC in U-Boot proper but got some issues
> >>>>>>>> while booting from eMMC [3]
> >>>>>>>
> >>>>>>> I have an updated branch at [4] that should support booting from sdmmc and sdhci.
> >>>>>>> Tested booting from both sd-card and emmc on my Radxa ROCK 5 Model B.
> >>>>>>>
> >>>>>>> This branch is based on u-boot/master f147aa80f52989c7455022ca1ab959e8545feccc
> >>>>>>> and I have included some rk3568 patches and your rk3588 rfc series.
> >>>>>>> I added a few fixup on top of that and a few additional patches, please see commit message
> >>>>>>> for a very brief note on why the change was needed.
> >>>>>>> Feel free to squash fixups and pick commits up to and possible including
> >>>>>>> "board: rockchip: Sync evb-rk3568_defconfig with evb-rk3588_defconfig"
> >>>>>>> for a v2 of this series.
> >>>>>>>
> >>>>>>> The remaining sdhci patches needs a little bit more work,
> >>>>>>> I can send a separate series with emmc patches once they are fully ready.
> >>>>>>>
> >>>>>>> The last commit adds a hack to mkimage to keep the data for atf-3 embedded in the FIT.
> >>>>>>> This ensures we do not run into problems trying to use dma from emmc into secure pmu sram.
> >>>>>>> I think this is a more appropriate way to work around this issue, instead of patching
> >>>>>>> u-boot spl_fit or sdhci core to use bounce buffers in a very special case.
> >>>>>>>
> >>>>>>> [4] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-master-v2
> >>>>>>>
> >>>>>> Hello Jagan, Jonas,
> >>>>>>
> >>>>>> I wanted to chip into this discussion, to ask whether you did anything
> >>>>>> more on the SD clock matter ?
> >>>>>
> >>>>> I have been busy this past week but have now had time to take a new look
> >>>>> at the sdmmc issue, along with completing some other fixes.
> >>>>>
> >>>>>>
> >>>>>> I am currently using your workaround that creates dummy clocks in the DT
> >>>>>> and then the SD node just uses those, and it works, in the SPL, if and
> >>>>>> only if the bootrom also loads the SPL from SD.
> >>>>>> I tried to have the SPL inside the SPI flash, or load it to DDR using
> >>>>>> the rockusb protocol, and then, initializing the SD-Card from SPL fails
> >>>>>> utterly, namely, it cannot communicate with the card.
> >>>>>> I did some changes to have the pinctrl working at SPL level, which
> >>>>>> appears to be fine, but I would like to see what can be done about the
> >>>>>> clocks.
> >>>>>> Having the cru and all the required drivers at SPL level is the way to
> >>>>>> go ? The SPL should run before SCMI is required so it should be able to
> >>>>>> change all the clocks at the clock controller level ?
> >>>>>
> >>>>> I fully agree that we should have some sort of scmi clk driver so that
> >>>>> we can control the sdmmc clk in both SPL and U-Boot proper.
> >>>>>
> >>>>> As you have noticed the current workaround only works because bootrom
> >>>>> leave the clocks in a working state after it has loaded TPL/SPL from
> >>>>> the sdmmc device. When TPL/SPL is loaded from any other source it is
> >>>>> not be possible to read from the sdmmc device in u-boot.
> >>>>>
> >>>>> After having played around with the scmi agent driver and being inspired
> >>>>> by the dummy scmi clk driver in vendor u-boot I have managed to create
> >>>>> something that could work. See top three commits from [5] for a working
> >>>>> proof-of-concept.
> >>>>>
> >>>>> What I did was to enable the scmi agent driver for use in u-boot proper
> >>>>> and keeping it disabled in SPL build. Then added a new scmi clk driver
> >>>>> that is enabled in SPL build, the rk3588 clk driver hooks the new clk
> >>>>> driver to the scmi_clk ofnode. I only implemented the get/set rate ops
> >>>>> for the two sdmmc clocks. With this both SPL and U-Boot proper should
> >>>>> be able to configure the sdmmc clocks.
> >>>>>
> >>>>> The initial fixes commits in that branch should hit the list soon.
> >>>>> Will send the sdmmc related commits once I have had some time to do
> >>>>> more testing.
> >>>>>
> >>>>> [5] https://github.com/Kwiboo/u-boot-rockchip/commits/rk35xx-fixes
> >>>>>
> >>>> Hi Jonas,
> >>>>
> >>>> I managed to get it working in SPL as well, with a similar code inspired
> >>>> from the vendor tree.
> >>>> About the solution, I am not sure whether it should be called a scmi
> >>>> clock, and whether it should be part of clk_rk3588
> >>>> Honestly by looking at the DT node, I find the description of the clock
> >>>> tree wrong to begin with.
> >>>> The SD should not have the 2 clocks tied to the scmi agent, because this
> >>>> is true only if it's running in normal world with a secure world agent
> >>>> that will act as a middle man. So if we are running at a higher EL, as
> >>>> in the SPL case, the clocks should be tied directly to the CRU.
> >>>> So the node itself is described bad IMO.
> >>>>
> >>>> Anyway, if the node is set in stone from Linux, there isn;t much we can do.
> >>>> I am also thinking whether we should have the SCMI enabled in SPL as
> >>>> well, but the agent/clock should just access the CRU directly, meaning
> >>>> to do something at the agent level :
> >>>> #ifdef CONFIG_SPL_BUILD
> >>>> -> access CRU
> >>>> #else
> >>>> -> send SCMI message
> >>>> #endif
> >>>> Does this make more sense ?
> >>>>
> >>>> Or have some kind of wrapper driver that would act as a dispatcher
> >>>> depending on the SPL/proper build ?
> >>>
> >>> Looks like the sdmmc clock is controlled by the securecru registers and I
> >>> expect that this can only be configured in secure world, not sure how this
> >>> could have been modeled differently.
> >>
> >> Since it can work with a clock given by either scmi or securecru, I
> >> would expect all clocks to be in the list, and the driver could start
> >> the needed clock as per the EL level it's running into :)
> >>
> >> e.g.
> >> clocks = <&scru CCLK_SD>, <&scru HCLK_SD>, <&scmi CCLK_SD>, <&scmi
> >> HCLK_SD> , <&cru basic other clocks..>
> >>
> >> Just my vision of how it could be modeled
> > 
> > That is a change to the device tree bindings.  The goal is to have
> > U-Boot use the same bindings and device tree as the official device
> > tree (which currently is the one in mainline Linux).
> 
> That's right. A change in the bindings (and Linux )

And OpenBSD, and NetBSD, and ...

The device tree bindings are ABI.  You can't change them unless they
are really relly broken.

> >>> With the approach that I took I think the normal clk framework will behave
> >>> as such dispatcher and clock gets tied to cru in SPL.
> >>>
> >>> #ifdef CONFIG_SPL_BUILD
> >>> -> bind protocol at 14 to rk3588 securecru driver -> read/write securecru regs
> >>> #else
> >>> -> bind protocol at 14 to scmi clk driver -> send SCMI message
> >>> #endif
> >>
> >> This looks right, just that we would have to bring a lot of bloat to SPL
> >> , firmware subsystem and things we may not want.
> > 
> > I don't see why this would bring in a lot of bloat into SPL.  The SPL
> > device tree will grow a little bit since it will have to include the
> > scmi nodes.  And a little bit of additional code in the rk3588 clock
> > driver.  Navigating the Kconfig stuff is a bit hard, but I don't think
> > this needs to pull in the firmware subsystem in SPL.
> 
> When I tried this, I noticed that only the scmi agents look inside the 
> subnodes (e.g. protocol at 14) which do not have a compatible, and then 
> they bind the subnodes to a driver found by a hardcoded search for 
> 'scmi_clock'.
> So, to get there, an agent is required, and firmware node, probed 
> firmware, probed 'scmi clock' . Then, probing it requires additional 
> things, because the agent wants a shared memory 'shmem' node, and you 
> end up also probing a SRAM area, allocating it to the space... that's 
> what I mean when I say that there might be much more bloat added than 
> just an additional clock set/get for securecru clocks.

No, Jonas added code to find the clock protocol node and bind the
clock driver to it:

https://github.com/Kwiboo/u-boot-rockchip/blob/3209167d7a518291f912964186ee90f10b555084/drivers/clk/rockchip/clk_rk3588.c#L1953

> >>> Naming and placement of the SPL securecru driver could be improved.
> >>> Not sure if any other soc beside rk3588 will need this at this moment.
> >>
> >> securecru sounds better for me, I don't think we can have a scmi clock
> >> fake driver like in vendor uboot, it has nothing 'scmi' about it.
> >>
> >>>> Meanwhile, I will test your patches to see how they work on my setup, I
> >>>> also have some things in progress including pinctrl in SPL for the rock5b.
> >>>> Thanks for your detailed description,
> >>>
> >>> Thanks for the hint at pinctrl, I made some updates to [5] and could verify
> >>> that sdmmc works when booting from emmc thanks to your pinctrl commit.
> >>> Also enabled SPI NOR and eMMC for rock5b in [6], and could do a sf probe
> >>> and build a spi firmware image.
> >>>
> >>> There is an issue with non-DMA access in sdhci that requires the HACK commit
> >>> for proper loading of atf, was hoping to disable SDMA in SPL like what is done
> >>> using fifo-mode for sdmmc. But only first sector is read successfully then it
> >>> fails, see below. Will take a closer look before I post eMMC series.
> >>
> >> In your patches, I see this :
> >> https://github.com/Kwiboo/u-boot-rockchip/blob/rk35xx-fixes/arch/arm/dts/rk3588s-u-boot.dtsi#L61
> >> Perhaps the fifo-mode is intended for the sdhci(emmc) and not for
> >> sdmmc(sd-card) ? My understanding was that sdhci cannot do DMA to SRAM,
> >> but the sdmmc can ?
> >> one possible workaround is to have DMA to DRAM and then relocate it to
> >> SRAM using the CPU. Having DMA disabled for the whole IP may have
> >> downsides, but this is U-boot, we don't expect to have anything else to
> >> do with the CPU while the DMA master works its magic, and the CPU should
> >> be faster.
> >>
> >> I tested your patches together with the series I sent now
> >> ( https://marc.info/?l=u-boot&m=167871206530072&w=2 ) and it appears to
> >> boot correctly from SD-Card
> >> Once you send them to the ML I can retest them.
> >> Thanks !
> >>
> >>>
> >>>     Trying to boot from MMC2
> >>>     1
> >>>        - 0 'mmc at fe2c0000'
> >>>        - 1 'mmc at fe2e0000'
> >>>        - found
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2c0000.blk, 67, 0
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2e0000.blk, 67, 1
> >>>     clock is disabled (0Hz)
> >>>     clock is enabled (400000Hz)
> >>>     clk_set_rate(clk=5000d8, rate=400000)
> >>>     size=200, ptr=3b8, limit=100000: 5001b8
> >>>     clock is enabled (25000000Hz)
> >>>     clk_set_rate(clk=5000d8, rate=25000000)
> >>>     clk_set_rate(clk=5000d8, rate=25000000)
> >>>     clock is enabled (52000000Hz)
> >>>     clk_set_rate(clk=5000d8, rate=52000000)
> >>>     spl: mmc boot mode: raw
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2c0000.blk, 67, 0
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2e0000.blk, 67, 1
> >>>     hdr read sector 4000, count=1
> >>>     Found FIT
> >>>     size=a00, ptr=dc0, limit=100000: aligned to 5003c0
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2c0000.blk, 67, 0
> >>>     blk_find_device: uclass_id=67, devnum=1: mmc at fe2e0000.blk, 67, 1
> >>>     fit read sector 4000, sectors=5, dst=5003c0, count=0, size=0xa00
> >>>     mmc_load_image_raw_sector: mmc block read error
> >>>     spl: mmc boot mode: fs
> >>>     Trying to boot from MMC1
> >>>
> >>> [6] https://github.com/Kwiboo/u-boot-rockchip/commits/rk3588-emmc
> >>>
> >>> Regards,
> >>> Jonas
> >>>
> >>>
> >>>> Eugen
> >>>>
> >>>>>
> >>>>> Regards,
> >>>>> Jonas
> >>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Eugen
> >>>>>>
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Jonas
> >>>>>>>
> >>>>>>>>
> >>>>>>>> [3] https://github.com/edgeble/u-boot/commit/1389822228ac3dfe8d3cd3513db6a32a5c898b7f
> >>>>>>>>
> >>>>>>>> Jagan.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> 
>

More information about the U-Boot mailing list