[PATCH 10/19] buildman: Incorporate the genboardscfg.py tool

[Simon Glass]

Hi Tom,

On Wed, 20 Jul 2022 at 11:16, Tom Rini <trini at konsulko.com> wrote:
>
> On Wed, Jul 20, 2022 at 09:01:04AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Mon, 18 Jul 2022 at 06:11, Tom Rini <trini at konsulko.com> wrote:
> > >
> > > On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini at konsulko.com> wrote:
> > > > >
> > > > > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini at konsulko.com> wrote:
> > > > > > >
> > > > > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > > > > doing another build in the same working directory. If it is out of date
> > > > > > > > with respect to the Kconfig, it is updated.
> > > > > > > >
> > > > > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > > > > but:
> > > > > > > >
> > > > > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > > > >   while, in case other tools use it
> > > > > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > > > >   a particular SoC (similar to 'buildman -nv <soc>'
> > > > > > >
> > > > > > > While I don't think other tools still use boards.cfg, this will make it
> > > > > > > easier to find out that I'm wrong.  Perhaps once the CONFIG to Kconfig
> > > > > > > migration is done we can move to just pickle'ing the data or similar
> > > > > > > since I find the main use of what was in boards.cfg can be figured out
> > > > > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > > > > trying to convert stuff.  Thanks for doing this.
> > > > > >
> > > > > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > > > > forecast as to when?
> > > > >
> > > > > Not yet.  I'm auditing CONFIG_SYS_* now, with a notion to move
> > > > > everything that's not really configurable just out of CONFIG namespace
> > > > > as the starting point.  That'll drop us down to ~500 to migrate, which
> > > > > feels a bit less daunting.
> > > > >
> > > > > > One thing we could to is provide an option for buildman to spit out
> > > > > > the various fields that go into boards.cfg
> > > > >
> > > > > Right.  So I might not have said this before, but one reason I wanted
> > > > > buildman to natively know kconfiglib and have everything was that while
> > > > > we can do a lot of good matching on what to build, it would be amazingly
> > > > > good to be able to say "build every platform with NVME_PCI set" (and if
> > > > > it's not too hard hex/int options with a specific value).
> > > >
> > > > Ah OK. At present moveconfig has the functionality to list the boards
> > > > that have particular options (-b and -f). It is expensive to build the
> > > > database though - over a minute on a 32-thread machine. So we would
> > > > have to cache it. Also just about any change would invalidate the
> > > > cache and I'm not sure if it possible to detect which changes have no
> > > > effect on which cache entries...
> > >
> > > Ah, maybe it will take some more thinking about then.  Maybe an
> > > "advanced" match option, and also seeing how to have Azure generate the db
> > > in one job and pass it as an artifact to every other job in the world
> > > build stage.  Not an immediate need.
> >
> > Well I suppose having that logic in moveconfig doesn't make a lot of
> > sense. So we could move it to buildman and have a way of creating the
> > database, as you say. But bear in mind that every commit being built
> > has the potential to change the Kconfig. It may be possible to hash
> > the files or detect changes using timestamps.
>
> Yeah, we can revisit this later on down the line.  I don't think (and we
> don't do it today either, fwiw) we need to ensure the list of boards to
> build is right for each step of the commit, and might even be
> counter-productive, at least for my use case.  Build the list of boards
> on top of tree, tell me how it changed over N commits (which commit
> caused the size change) is how I use things.  Then the CI case is only
> for top of tree anyhow.

OK, yes, actually that's the decision I came to for the original board
selection, i.e. the board selection comes from the current commits.

Regards,
Simon