buildman stops (crashed) on current master

[Stefano Babic]

Hi Simon,

On 07.10.21 15:43, Simon Glass wrote:
> Hi Stefano,
> 
> On Thu, 7 Oct 2021 at 04:37, Stefano Babic <sbabic at denx.de> wrote:
>>
>> Hi all,
>>
>> CI stops by building aarch64 without notice, for reference:
>>
>> https://source.denx.de/u-boot/custodians/u-boot-imx/-/jobs/332319
>>
>> There is no error, just process is killed. It looks like it stops at
>> xilinx_zynqmp_virt,
>>
>> ./tools/buildman/buildman -o /tmp -P -E -W aarch64but board can be built
>> without issues.
>>
>> If I build on my host (not in docker, anyway), it generally builds fine
>> - but it crashes sometimes, too. On gitlab instance , it crashes.
>> Issue does not seem that depends on merged patches, and introduces
>> boards were already built successfully. Any hint ? I have also no idea
>> what I should look as what I see is just
>>
>> "usr/bin/bash: line 104:    24 Killed
>> ./tools/buildman/buildman -o /tmp -P -E -W aarch64"
> 
> I cannot see that link.

Verified with Wolfgang. The CI's results are available only after having 
logged in to source.denxe.de. Without account, they are not shown and 
the server return not found (I was not aware of this).

> I am not sure what is going on. Does it say
> what signal killed it?

Nothing, that makes an investigation difficult. It looks like it is 
bound (resource issue ?) with the runner or docker instance, because if 
I run the same buildman command on my host, it works until end and no 
errors are reported:

    aarch64:  w+   xilinx_zynqmp_virt 

+===================== WARNING ======================
+This board uses CONFIG_SPL_FIT_GENERATOR. Please migrate
+to binman instead, to avoid the proliferation of
+arch-specific scripts with no tests.
+====================================================
+WARNING: BL31 file bl31.bin NOT found, U-Boot will run in EL3
   125  181    0 /306            xilinx_zynqmp_virt

gitlab stops exactly with the last board, as if all work was done, but 
rather after a timeout and pipeline is set to failed (just a feeling).

> 
> Does it sit there for an hour and timeout? If so, then I  did see that
> myself once recently, when the Kconfig needed stdin, but I could not
> quitetie it down. I think buildman would provide it, but sometimes
> not, apparently. So it can happen when there is an existing build
> there and your new one which adds Kconfig options that don't have
> defaults, or something like that?

But anytime there is a new docker instance on gitlab runner, so this 
could happen only on local host.

> 
> If that is it, you can repeat it by clearing out your .bm-work
> directory then building just that board for one commit, then the next
> (with the Kconfig change).
> 
> Buildman is supposed to handle this, of course. I'm not sure what has changed.

Ok, let's see.

Regards,
Stefano


-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================