Two jobs at once on denx-vulcan?

[Simon Glass]

Hi Tom,

On Fri, 24 Sept 2021 at 08:20, Tom Rini <trini at konsulko.com> wrote:
>
> On Fri, Sep 24, 2021 at 04:01:21PM +0200, Harald Seiler wrote:
> > Hi Simon,
> >
> > On Mon, 2021-09-20 at 08:06 -0600, Simon Glass wrote:
> > > Hi Harald,
> > >
> > > On Mon, 20 Sept 2021 at 02:12, Harald Seiler <hws at denx.de> wrote:
> > > >
> > > > Hi,
> > > >
> > > > On Sat, 2021-09-18 at 10:37 -0600, Simon Glass wrote:
> > > > > Hi,
> > > > >
> > > > > Is there something screwy with this? It seems that denx-vulcan does
> > > > > two builds at once?
> > > > >
> > > > > https://source.denx.de/u-boot/custodians/u-boot-dm/-/jobs/323540
> > > >
> > > > Hm, I did some changes to the vulcan runner which might have caused
> > > > this... But still, even if it is running multiple jobs in parallel, they
> > > > should still be isolated, so how does this lead to a build failure?
> > >
> > > I'm not sure that it does, but I do see this at the above link:
> > >
> > > Error: Unable to create
> > > '/builds/u-boot/custodians/u-boot-dm/.git/logs/HEAD.lock': File
> > > exists.
> >
> > This is super strange... Each build should be running in its own
> > container so there should never be a way for such a race to occur.  No
> > clue what is going on here...
>
> I know this from having to track down a different oddball failure with
> konsulko-bootbake.  It comes down to something along the lines of
> volumes being re-used.  Good in that it means that every job every time
> isn't doing a whole clone of the u-boot tree.  Bad in that just in case
> the job gets wedged/killed in a crazy spot you end up with problems like
> this.  If you run a 'find' on vulcan you'll figure out which overlay has
> a problem.  Or you can stop the runner for a moment and tell docker to
> purge unused volumes and it'll clear it up.
>
> > > Re doing multiple builds, have you set it up so it doesn't take on the
> > > very large builds? I would love to enable multiple builds for the qemu
> > > steps since they mostly use a single CPU, but am not sure how to do
> > > it.
> >
> > Actually, this was more a mistake than an intentional change.  I updated
> > the runner on vulcan to also take jobs for some other repos and wanted
> > those jobs to run in parallel.  It looks like I just forgot setting the
> > `limit = 1` option for the U-Boot runner.
> >
> > Now, I think doing what you suggest is possible.  We need to tag build
> > and "test" jobs differently and then define multiple runners with
> > different limits.  E.g. in `.gitlab-ci.yml`:
> >
> >       build all 32bit ARM platforms:
> >         stage: world build
> >         tags:
> >           - build
> >
> >       cppcheck:
> >         stage: testsuites
> >         tags:
> >           - test
> >
> > And then define two runners in `/etc/gitlab-runner/config.toml`:
> >
> >       concurrent = 4
> >
> >       [[runners]]
> >         name = "u-boot builder on vulcan"
> >         limit = 1
> >         ...
> >
> >       [[runners]]
> >         name = "u-boot tester on vulcan"
> >         limit = 4
> >         ...
> >
> > and during registration they get the `build` and `test` tags
> > respectively.  This would allow running (in this example) up to 4 test
> > jobs concurrently, but only ever one large build job at once.
>
> Yes, but this would also make it harder for people to use the CI as-is
> with their own runners.  For example, the only thing stopping people
> from using the free gitlab CI runners on their own is that squashfs
> test being broken.

Thanks for the info Harald.

Would it just mean that they would need to add both 'build' and 'test'
tags to their running? If so that does not sound onerous.

I believe it would speed up CI quite a bit.

Regards,
Simon