[PATCH 10/19] buildman: Incorporate the genboardscfg.py tool

[Tom Rini]

On Wed, Jul 20, 2022 at 09:01:04AM -0600, Simon Glass wrote:
> Hi Tom,
> 
> On Mon, 18 Jul 2022 at 06:11, Tom Rini <trini at konsulko.com> wrote:
> >
> > On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini at konsulko.com> wrote:
> > > >
> > > > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini at konsulko.com> wrote:
> > > > > >
> > > > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > > > doing another build in the same working directory. If it is out of date
> > > > > > > with respect to the Kconfig, it is updated.
> > > > > > >
> > > > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > > > but:
> > > > > > >
> > > > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > > >   while, in case other tools use it
> > > > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > > >   a particular SoC (similar to 'buildman -nv <soc>'
> > > > > >
> > > > > > While I don't think other tools still use boards.cfg, this will make it
> > > > > > easier to find out that I'm wrong.  Perhaps once the CONFIG to Kconfig
> > > > > > migration is done we can move to just pickle'ing the data or similar
> > > > > > since I find the main use of what was in boards.cfg can be figured out
> > > > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > > > trying to convert stuff.  Thanks for doing this.
> > > > >
> > > > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > > > forecast as to when?
> > > >
> > > > Not yet.  I'm auditing CONFIG_SYS_* now, with a notion to move
> > > > everything that's not really configurable just out of CONFIG namespace
> > > > as the starting point.  That'll drop us down to ~500 to migrate, which
> > > > feels a bit less daunting.
> > > >
> > > > > One thing we could to is provide an option for buildman to spit out
> > > > > the various fields that go into boards.cfg
> > > >
> > > > Right.  So I might not have said this before, but one reason I wanted
> > > > buildman to natively know kconfiglib and have everything was that while
> > > > we can do a lot of good matching on what to build, it would be amazingly
> > > > good to be able to say "build every platform with NVME_PCI set" (and if
> > > > it's not too hard hex/int options with a specific value).
> > >
> > > Ah OK. At present moveconfig has the functionality to list the boards
> > > that have particular options (-b and -f). It is expensive to build the
> > > database though - over a minute on a 32-thread machine. So we would
> > > have to cache it. Also just about any change would invalidate the
> > > cache and I'm not sure if it possible to detect which changes have no
> > > effect on which cache entries...
> >
> > Ah, maybe it will take some more thinking about then.  Maybe an
> > "advanced" match option, and also seeing how to have Azure generate the db
> > in one job and pass it as an artifact to every other job in the world
> > build stage.  Not an immediate need.
> 
> Well I suppose having that logic in moveconfig doesn't make a lot of
> sense. So we could move it to buildman and have a way of creating the
> database, as you say. But bear in mind that every commit being built
> has the potential to change the Kconfig. It may be possible to hash
> the files or detect changes using timestamps.

Yeah, we can revisit this later on down the line.  I don't think (and we
don't do it today either, fwiw) we need to ensure the list of boards to
build is right for each step of the commit, and might even be
counter-productive, at least for my use case.  Build the list of boards
on top of tree, tell me how it changed over N commits (which commit
caused the size change) is how I use things.  Then the CI case is only
for top of tree anyhow.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20220720/f18aa832/attachment.sig>