[U-Boot] Hart lottery and CONFIG_XIP

[Auer, Lukas]

On Tue, 2019-06-04 at 09:42 +0800, Rick Chen wrote:
> Hi Troy
> 
> Bin Meng <bmeng.cn at gmail.com> 於 2019年6月3日 週一 下午10:56寫道：
> > Hi Troy,
> > 
> > On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes
> > <troy.benjegerdes at sifive.com> wrote:
> > > 
> > > 
> > > > On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn at gmail.com> wrote:
> > > > 
> > > > +Anup
> > > > 
> > > > Hi Troy,
> > > > 
> > > > On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> > > > <troy.benjegerdes at sifive.com> wrote:
> > > > > 
> > > > > 
> > > > > > On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36 at gmail.com> wrote:
> > > > > > 
> > > > > > Hi Troy
> > > > > > 
> > > > > > > > From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> > > > > > > > Sent: Saturday, June 01, 2019 12:24 AM
> > > > > > > > To: Auer, Lukas
> > > > > > > > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> > > > > > > > Subject: Re: Hart lottery and CONFIG_XIP
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > > On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer at aisec.fraunhofer.de>
> > > > > > > > wrote:
> > > > > > > > > Hi Troy,
> > > > > > > > > 
> > > > > > > > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> > > > > > > > > > Wouldn't the following line in head.S fail when running from flash
> > > > > > > > > > (although maybe not in a way that prevents booting)
> > > > > > > > > > 
> > > > > > > > > >      /* save the boot hart id to global_data */
> > > > > > > > > >      SREG    tp, GD_BOOT_HART(gp)
> > > > > > > > > > 
> > > > > > > > > > Shouldn’t this be protected by CONFIG_XIP?
> > > > > > > > > 
> > > > > > > > > The boot hart ID is stored in global data, which is allocated from the
> > > > > > > > > stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> > > > > > > > > won't cause any issues when running from flash.
> > > > > > > > 
> > > > > > > > Sorry about the confusion, I was reading it wrong earlier.
> > > > > > > > 
> > > > > > > > I’m hoping to have a couple of patches ready later today to change the
> > > > > > > > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> > > > > > > > remove the potential indeterminism of which hart wins the lottery when doing
> > > > > > > > board bringup and debugging.
> > > > > > 
> > > > > > I am OK with that.
> > > > > > Actually my preliminary patch about
> > > > > > [PATCH 0/4] AE350 support SMP boot from flash
> > > > > > did as your wish.
> > > > > > 
> > > > > > You can refer it :
> > > > > > [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> > > > > > https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
> > > > > > 
> > > > > > Rick
> > > > > 
> > > > > To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
> > > > > 
> > > > > It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
> > > > > 
> > > > 
> > > > I am not sure what caused the confusion. CONFIG_XIP was added to
> > > > support U-Boot executing from ROM directly.
> > > > 
> > > 
> > > The confusion is use cases where you don’t necessarily need CONFIG_XIP,
> > > but you do want deterministic SMP booting, and the code is a lot more understandable
> > > with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
> > > warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
> > > 
> > 
> > OK, as I pointed out, the deterministic SMP booting was originally
> > posted, but later was changed to the lottery mechanism per Anup's
> > review comments. I am personally fine with both.
> 
> Yes. I am agree with that.
> But it shall also support booting form ROM and RAM as now.

Just to give a bit of context information. Anup's reasoning was that
with a fixed boot hart, we have to rely on it to work. If there is a
hardware failure in just the boot hart, the system won't be able to
boot anymore. With a hart lottery that is not a problem.

With that in mind, it makes sense to have to option of a fixed boot
hart, but I would not enable that option in any of the upstream
defconfigs. That way it can be manually enabled if required for the
application.

Thanks,
Lukas

> > > > > From an system testing and validation point of view, I would find it much better
> > > > > (especially at very early boot stages, where U-boot might be the first non-ROM code
> > > > > running) to have a deterministic process to determine what core runs U-boot. This
> > > > 
> > > > I remember when SMP patches were submitted by Lukas for the first time
> > > > it was deterministic using some macros like CONFIG_BOOT_HART, however
> > > > per Anup's request, the hart lottery feature, similar to what Linux
> > > > has, was added.
> > > 
> > > I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly
> > > for system validation testing. Also along those lines, if we are going to use a lottery
> > > to determine what CPU boots the system, how do we know afterwards which one
> > > it was?
> > > 
> > 
> > Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then
> > look at which cpu node has a child node of "riscv_timer".
> 
> Or you can check gd->arch.boot_hart by gdb
> 
> (gdb) p/x gd->arch.boot_hart
> $3 = 0x0
> 
> B.R
> Rick
> 
> > Regards,
> > Bin