[PATCH] CI: Add gitlab tags to builds which can use all CPUs

[Simon Glass]

Hi Tom,

On Thu, 28 Nov 2024 at 10:01, Tom Rini <trini at konsulko.com> wrote:
>
> On Thu, Nov 28, 2024 at 09:00:10AM -0700, Simon Glass wrote:
>
> > Most gitlab runners are mostly idle at present and could happily run
> > several jobs at once.
> >
> > There are a few exceptions, such as the world builds.
> >
> > Add a 'single' tag for the exceptions so that they are dealt with
> > separately, with no concurrency. This is not quite perfect, since it
> > means that a 'single' job can be combined with a number of normal jobs,
> > but in practice this doesn't seem to matter.
> >
> > To use this, for each existing machine, add a new, separate runner, with
> > the tag 'single'. Set 'limit = 1' in its config.toml file and set
> > 'concurrent = 10' (for example) in the global section. The machine will
> > then pick up one 'single' job and up to 9 other jobs at once.
> >
> > In my testing this reduces the time to complete a CI pipeline.
> >
> > My choice of which jobs to mark as single is mostly down to how much CPU
> > they need. For the 'docs' job, it runs for a short time, so doesn't seem
> > worth marking it as a 'single' job.
> >
> > Signed-off-by: Simon Glass <sjg at chromium.org>
>
> This is an interesting approach. What will work better long term is
> adding more runners with the "all" tag because as part of the rework of
> the Gitlab job I've done and you can see here:
> https://source.denx.de/u-boot/u-boot/-/pipelines/23588
> (but isn't ready to post as Ilias wants to look in to the sandbox+arm64
> issue) means that every runner needs some tags and for runners which can
> handle multiple "small" jobs at once (as the first two stages are
> building a board and then testing, or similar) we can make use of that,
> and then restrict the world builds to the fastest nodes only.
> This also means that things like the Oracle "always free" tier cloud
> instances could be used for example as an "all" jobs node, I suspect.
> And the same for any other cloud provider with an equivalent tier.

For reference I had this idea a year ago [1] and it didn't go anywhere
at the time.

For me this patch reduces the CI time to 35mins best case. So I'm
going to pick up this patch for the ci tree. Once you have patches for
your approach (which seems better to me as well), we can go with that.

Regards,
Simon

[1] CAHFG_=USRrEytULOdnezrCmeAcV_KGO7M=ieWCoi1mCzHwmHRQ at mail.gmail.com/
https://lore.kernel.org/u-boot/CAPnjgZ0Gj==Q-YOU+EoD=s3h25Waw5EYa0DEOD_comoAy_bRMw@mail.gmail.com/