[U-Boot] UBoot running UBoot - is it possible?

Tue Nov 13 13:11:46 UTC 2018

Hi Allan,

On 13/11/18 02:38, Allan Chandler wrote:
> Hello, bods.
> 
> We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.

ok - think about that a *safe* update of the bootloader with all
components (SPL, u-boot.img, environment) must be considered and
designed at the early beginning because it could require hardware support.

The i.MX6 can boot from several devices in a sequence, and this can be
used to update the bootloader. For example, it can boot from a SPI NOR
flash if booting from eMMC fails. Of course, you have added a SPI NOR
(or another second source) in your design.

Even in this case, please consider that the BootROM searches for another
source if the first one is definely wroing, that means that the i.MX
header is corrupted. If the image (SPL) is correct, the BootROM cannot
detect it.

This means that with multiple devices where to boot you cannot be sure,
because a valid SPL was written but board does not boot for some reasons.

You could use the EXT CSD register in the eMMC (it looks like you have
just eMMC on board) to switch the boot partition. This is transparent to
the i.MX and eMMC since eMMC 4.3 supports two partitions. This can allow
to write the second bootloader and to be power-off safe, but there is no
fallback. Anyway, the rick can be mitigated with testing, but cannot be
dropped at all. If the second bootloader is valid, the BootROM will
always try to boot from it and there is no fallback if it goes wrong.

To be absolutely sure, a (hw) mechanism that switch between two SPLs,
maybe driven if watchdog expires, could be implemented. Of course,
complexity increases.

The thing is that each project must balance risks again costs. In some
cases, it is also accepted that bootloader is updated with a percentage
of risks that the board could be returned because broken. This can be
cheaper as implementing a 100% safe update for the bootloader, but of
course it cannot be used in critical applications.

You could also plan to update U-Boot, that is u-boot.img on i.MX,
without updating SPL. This is also a used path, with SPL fixed on the
board and checking for two copies of u-boot.img.

However, this is broken. SPL is responsible for the first initialization
and first of all of the DRAM controller on i.MX. If something was
broken, it cannot be fixed just by updating the u-boot.img image. I do
not know if it is worth to have it, it looks like a half-upate where
just a very limited number of issues can be fixed.

> 
> However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
> 
> What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
> 
> 1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
> 
> 2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.

Check if version differs is a *must*. I do this in SWUpdate and the
bootloader is always part of each delivery, but it is updated only if
version differs.

> 
> The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
> 
> We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
> 
> So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.

Think again: you are updating the bootloader, but you rely on u-boot
scripts that are part of u-boot itself. Scripts can be corrupted, too,
or changed by a previous u-boot version, or...

It sounds like a hack, but of course, it could work with a percentage of
risks that things can go wrong. I mean, a percentage without taking into
account Murphy's law :-D

> 
> 1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot:
>             mmcboot=run check_uboot; <rest of original mmcboot>
>             other_uboot=0
> 
> 2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
> 
> 3/ We defined check_uboot as:
>             if test ${other_uboot} -eq 1; then
>                         setenv other_uboot 0
>                         saveenv
>                         ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg
>                         go 0x17800000
>             fi

This means that the *old* U-Boot works pretty well, because it can go
until this point and load something else...

There are a lot of assumption here, because you are not really updating
the bootloader, but simply adding a new actor in the chain. And then,
both can be buggy.....

> The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
> 
> 4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
> 
> Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.

The second bootloader must be fine tuned and changed from original
because it is not allowed to set again the DRAM controller, and maybe
even some closks, and maybe...

> 
> I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
> 
> So I guess my questions are as follows:
> 
> a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?

Define safely. 100% error prone, 97%, ..

100% safe could mean additional hardware and additional complexity, and
at the end, additional risk that something can go wrong.

That means: some projects decide to add hw support or more to be
absolutely sure that nothing can happen, some other projects take the
risk and accept the possibility to have a low RTB. You decide, at the end...

> b/ Does anyone have any ideas on how I could achieve this?

I think that each solution is quite project specific. Other SOCs allow
to switch automatically SPL (that means, the source for the bootloader).
On i.MX6, it can be reached using a sequence of devices (see BOOT_CFG in
manual), but there is no fallback.

> 
> I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
> 
> Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.

Best regards,
Stefano Babic

> 
> Cheers,
> Pax
> 
> Allan Chandler | Software Engineer
> DTI Group Ltd | Integrated Transit Technologies
> 31 Affleck Road, Perth Airport, WA 6105, Australia
> P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler at dti.com.au<mailto:allan.chandler at dti.com.au>
> Visit our website www.dti.com.au<http://www.dti.com.au>
> The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot
> 

-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================