[PATCH v4 00/33] Initial implementation of standard boot

[Tom Rini]

On Thu, Mar 24, 2022 at 12:33:38PM -0600, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 24 Mar 2022 at 08:46, Tom Rini <trini at konsulko.com> wrote:
> >
> > On Wed, Mar 23, 2022 at 08:13:42PM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Wed, 23 Mar 2022 at 14:09, Tom Rini <trini at konsulko.com> wrote:
> > > >
> > > > On Wed, Mar 23, 2022 at 12:45:50PM -0600, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Wed, 23 Mar 2022 at 08:05, Tom Rini <trini at konsulko.com> wrote:
> > > > > >
> > > > > > On Sun, Mar 06, 2022 at 05:49:43AM -0700, Simon Glass wrote:
> > > > > > >
> > > > > > > The bootflow feature provide a built-in way for U-Boot to automatically
> > > > > > > boot an Operating System without custom scripting and other customisation.
> > > > > > > This is called 'standard boot' since it provides a standard way for
> > > > > > > U-Boot to boot a distro, without scripting.
> > > > > > >
> > > > > > > It introduces the following concepts:
> > > > > > >
> > > > > > >    - bootdev - a device which can hold a distro
> > > > > > >    - bootmeth - a method to scan a bootdev to find bootflows (owned by
> > > > > > >                 U-Boot)
> > > > > > >    - bootflow - a description of how to boot (owned by the distro)
> > > > > > >
> > > > > > > This series provides an implementation of these, enabled to scan for
> > > > > > > bootflows from MMC, USB and Ethernet. It supports the existing distro
> > > > > > > boot as well as the EFI loader flow (bootefi/bootmgr). It works
> > > > > > > similiarly to the existing script-based approach, but is native to
> > > > > > > U-Boot.
> > > > > > >
> > > > > > > With this we can boot on a Raspberry Pi 3 with just one command:
> > > > > > >
> > > > > > >    bootflow scan -lb
> > > > > > >
> > > > > > > which means to scan, listing (-l) each bootflow and trying to boot each
> > > > > > > one (-b). The final patch shows this.
> > > > > > >
> > > > > > > With a standard way to identify boot devices, booting become easier. It
> > > > > > > also should be possible to support U-Boot scripts, for backwards
> > > > > > > compatibility only.
> > > > > > >
> > > > > > > This series relies on the PXE clean-up series, posted here:
> > > > > > >
> > > > > > >    https://patchwork.ozlabs.org/project/uboot/list/?series=267078
> > > > > > >
> > > > > > > For documentation, see the 'doc' patch.
> > > > > > >
> > > > > > > For version 2, a new naming scheme is used as above:
> > > > > > >
> > > > > > >    - bootdev is used instead of bootdevice, because 'device' is overused,
> > > > > > >        is everywhere in U-Boot, can be confused with udevice
> > > > > > >    - bootmeth - because 'method' is too vanilla, appears 1300 times in
> > > > > > >        U-Boot
> > > > > > >
> > > > > > > Also in version 2, drivers are introduced for the boot methods, to make
> > > > > > > it more extensible. Booting a custom OS is simply a matter of creating a
> > > > > > > bootmeth for it and implementing the read_file() and boot() methods.
> > > > > > >
> > > > > > > Version 4 makes some minor improvements and leaves out the RFC patch for
> > > > > > > rpi conversion, in the hope of getting the base support applied sooner
> > > > > > > rather than later.
> > > > > > >
> > > > > > > The design is described in these two documents:
> > > > > > >
> > > > > > > https://drive.google.com/file/d/1ggW0KJpUOR__vBkj3l61L2dav4ZkNC12/view?usp=sharing
> > > > > > >
> > > > > > > https://drive.google.com/file/d/1kTrflO9vvGlKp-ZH_jlgb9TY3WYG6FF9/view?usp=sharing
> > > > > >
> > > > > > I keep putting off commenting more here, but, I still feel this is the
> > > > > > wrong direction.  What problems do we have today with distro boot?
> > > > > > Well, we haven't figured out how to move configuring it out of the board
> > > > > > config.h file.  But that's just one of a half dozen or so examples of
> > > > > > how we haven't figured out a good solution to configuring the default
> > > > > > environment.  And only some of those other examples are boot related
> > > > > > (the NXP chain of trust booting stuff is another boot example, ETHPRIME,
> > > > > > HOSTNAME, etc, are non-boot examples).
> > > > > >
> > > > > > We also aren't improving testing of "can we boot" here, because what
> > > > > > THAT needs is setting up LAVA and booting some installers on some
> > > > > > hardware (and some QEMU).  That's testing that Linux boot works.  Today
> > > > > > we have tests for hush parsing, and if distro boot makes use of
> > > > > > something we don't have a test for, we need a test for it.  This adds
> > > > > > tests for itself, which is good.
> > > > > >
> > > > > > And I still don't see an example of where this demonstrates that
> > > > > > existing non-UEFI boot cases are now easier to handle or cleaner to
> > > > > > handle or otherwise better.
> > > > > >
> > > > > > In that this is an attempt to tackle one of the long standing needed
> > > > > > migrations (be able to drop board config.h files), something here needs
> > > > > > doing.  But I don't see this as the right direction, sorry.
> > > > >
> > > > > Does anyone have a better idea for all of this? This is a solid base
> > > > > we can build on but we can't make any progress while this is just
> > > > > patches. What not apply it and we can move forward?
> > > > >
> > > > > - solves the env problem for distro boot in that we don't need the scripts
> > > >
> > > > Solves what problem?  That distro boot is done in hush?  Yes, we have
> > > > many examples of needing to define the default environment in a
> > > > non-static way.  This is not a solution to that problem.
> > >
> > > It removes the need for the boot scripts that are currently in the
> > > environment. I'm not sure what you mean by defining a default
> > > environment in a non-static way.
> >
> > I'm not sure what you're missing.  We have a number of examples today of
> > modifying the default environment based on CONFIG options.  How we do
> > the generic distro boot is one of those examples, and you're trying to
> > replace it.  It should be able to address the NXP (fsl_...) secure boot
> > scripts as well.  It doesn't address at all the non-boot related things.
> 
> The text environment does support CONFIG options.

Then I misremembered or misunderstood something you had said previously,
sorry.

> > > > > - gets rid of the scripts which are a confusing mess
> > > >
> > > > Yes, a generic "figure out where a bootable might be, boot it" is
> > > > complex.  That's basically why the UEFI boot manager code exists, and
> > > > replaces that logic.
> > >
> > > I thought we already decided that U-Boot was going to continue to
> > > support non-UEFI booting?
> >
> > Yes?  CONFIG_BOOTCOMMAND can be whatever you like.  No one is talking at
> > all anywhere about removing anything that's not bootefi.  I don't
> > understand your point here.

I want to come back here for a moment because maybe we're talking at
cross purposes.  I absolutely want there to be CONFIG_BOOTCOMMAND and/or
non-hush-based (for device security) ways to run whatever, to boot the
OS/next stage.

But for something to deal with the combinatorial matrix hell of
heuristic and then some configurability (or not!) of find out which
arbitrary devices to boot from, or not boot from, or override booting
from and so on, no, I don't want to DIY something, I want the UEFI
standard for that.  If there's some other standard for dealing with all
of that as well, OK, lets talk about implementing it.  If UEFI standard
could be improved in some way, well, lets push back and get it improved.
But I do not want another implementation of find an arbitrary thing and
boot it.  I'm not sure who is asking for that, either.

> > > > > - provides proper high-level concepts of boot device and boot method
> > > >
> > > > I don't see the inherent value.  Or maybe to put things another way, the
> > > > "now go boot" API has always been implemented via hush, even when the
> > > > interactive part is disabled.  You're providing a programatic way of
> > > > doing it instead.  And I'm not seeing it being more flexible / easier to
> > > > implement.  It's really sounded harder, not easier, than #define
> > > > BOOT_TARGET_DEVICES.
> > >
> > > It is easier in that you don't need to define the boot targets. It is
> > > done automatically using the devices that are available.
> >
> > So you're replacing defining the system to be correct with heuristics to
> > guess that it's correct?
> >
> > > > > - allows testing of the U-Boot part of 'can we boot' because we have
> > > > > tests for all the cases - we can expand this over time
> > > >
> > > > No, it doesn't.  It provides a different interface and tests for itself.
> > > > If it's not LAVA (or something like that) booting real Linux
> > > > distribution images it's not a real can we boot test.
> > >
> > > I didn't say it would test booting linux or real distros. You
> > > obviously need LAVA (or something like that) for doing that.
> > >
> > > But it does provide tests for the boot flow which are completely
> > > lacking at present. So at least we are testing the code booting code.
> >
> > We have hush tests, the distro boot logic is therefore tested as well as
> > what you're adding here as neither of them are today testing that we can
> > actually boot something.
> >
> > > > > - allows non-UEFI boot cases like Chrome OS, which is currently just a
> > > > > hack for one board[1]
> > > >
> > > > I don't see a Chrome OS example in your series.
> > >
> > > Would that help? I don't mind doing it if we can make some progress,
> > > but your previous objection was about code size. I addressed that and
> > > have not heard any comment about that.
> >
> > I object to the whole thing.  The lack of implementations where this is
> > showing that it's actually better than what we have today is just one of
> > the problems.  That we replace something that's basically size-free and
> > replace it with several kilobytes is another problem.
> >
> > > > > - provides a programmatic base for A/B boot, etc.
> > > >
> > > > And I've noted before that I don't see how this makes life easier for
> > > > any of the existing A/B mechanisms.  Some of which are just going with
> > > > "Hey, just UEFI boot and let grub deal with it".
> > >
> > > It makes it much easier since we now have the means to locate and read
> > > in boot flows. Then it is just a case of selecting the one we want,
> > > e.g. by examining the list of bootflows and selecting one. It is a
> > > helpful division of labour.
> >
> > That's not the hard part of integrating an A/B update mechanism or
> > implementing it for a given board.
> >
> > > > > I feel the same way with Takahiro's series, which has been out-of-tree
> > > > > for too long.
> > > > >
> > > > > Please reconsider this. What do we have to lose?
> > > >
> > > > What we'll gain seems to be another chunk of code that needs boards to
> > > > migrate to, maybe.  And I don't see examples where this, now, or in the
> > > > next iteration, provides a smaller, or clearer implementation of
> > > > something.
> > >
> > > Yes it requires boards to migrate, although in fact the migration can
> > > be driven by features rather than board-by-board. For example, we
> > > could migrate boards that use USB, MMC, etc. first, and then migrate
> > > more as we add more features.
> > >
> > > But at least it is a chunk of code, with full tests. The scripts are
> > > just a pain.
> >
> > So we just add unused code until someone makes use of it?  We already
> > have that with the environment rework that can't be really used because
> > it doesn't handle distro boot support, which this attempts to handle by
> > removing that functionality from the environment and to C code.
> 
> My summary of the above is that you see a lot of other/bigger problems
> and want to solve those instead. But that is not the purpose of this
> series.
> 
> >
> > And as I'm testing some Kconfig migrations in another window and seeing
> > what feels like an endless scroll of "migrate to ..." messages flow by,
> > I have little confidence in more than a few boards getting migrated to
> > this functionality, while leaving the to me bigger problems entirely
> > unresolved (how do we delete board config.h files when the environment
> > needs CONFIG based tweaking, which is more than just distro boot).
> 
> This seems to be the nub of it. I am reluctant to invest more time in
> this when there is so little mindshare about the problems we face. You
> are reluctant to move it forward when you worry it will just be
> another migration effort that leads nowhere.
> 
> I am more than happy to commit to migrating all boards over to bootstd
> and removing the distro scripts from the environment. This is a
> per-feature migration, rather than per-board.
> 
> Does that help?

I see a large number of problems that need to be solved, most urgently,
in U-Boot.  A good subset of these are DM migration related and you're
in the best position to make some initial passes at.  I'm also not
convinced that this series has been worked far enough to be an
improvement on the boot cases that are NOT "just use UEFI boot manager".
And the time / effort for getting there I'd personally rather see put in
to some other migrations / improvements on the DM front.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20220324/253589da/attachment.sig>