[PATCH] CI: Add automatic retry for test.py jobs
Tom Rini
trini at konsulko.com
Thu Jul 27 22:35:15 CEST 2023
On Thu, Jul 27, 2023 at 01:18:12PM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Sun, 16 Jul 2023 at 12:18, Tom Rini <trini at konsulko.com> wrote:
> >
> > On Sat, Jul 15, 2023 at 05:40:25PM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Thu, 13 Jul 2023 at 15:57, Tom Rini <trini at konsulko.com> wrote:
> > > >
> > > > On Thu, Jul 13, 2023 at 03:03:57PM -0600, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Wed, 12 Jul 2023 at 14:38, Tom Rini <trini at konsulko.com> wrote:
> > > > > >
> > > > > > On Wed, Jul 12, 2023 at 02:32:18PM -0600, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Wed, 12 Jul 2023 at 11:09, Tom Rini <trini at konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jul 12, 2023 at 08:00:23AM -0600, Simon Glass wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > On Tue, 11 Jul 2023 at 20:33, Tom Rini <trini at konsulko.com> wrote:
> > > > > > > > > >
> > > > > > > > > > It is not uncommon for some of the QEMU-based jobs to fail not because
> > > > > > > > > > of a code issue but rather because of a timing issue or similar problem
> > > > > > > > > > that is out of our control. Make use of the keywords that Azure and
> > > > > > > > > > GitLab provide so that we will automatically re-run these when they fail
> > > > > > > > > > 2 times. If they fail that often it is likely we have found a real issue
> > > > > > > > > > to investigate.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Tom Rini <trini at konsulko.com>
> > > > > > > > > > ---
> > > > > > > > > > .azure-pipelines.yml | 1 +
> > > > > > > > > > .gitlab-ci.yml | 1 +
> > > > > > > > > > 2 files changed, 2 insertions(+)
> > > > > > > > >
> > > > > > > > > This seems like a slippery slope. Do we know why things fail? I wonder
> > > > > > > > > if we should disable the tests / builders instead, until it can be
> > > > > > > > > corrected?
> > > > > > > >
> > > > > > > > It happens in Azure, so it's not just the broken runner problem we have
> > > > > > > > in GitLab. And the problem is timing, as I said in the commit.
> > > > > > > > Sometimes we still get the RTC test failing. Other times we don't get
> > > > > > > > QEMU + U-Boot spawned in time (most often m68k, but sometimes x86).
> > > > > > >
> > > > > > > How do we keep this list from growing?
> > > > > >
> > > > > > Do we need to? The problem is in essence since we rely on free
> > > > > > resources, sometimes some heavy lifts take longer. That's what this
> > > > > > flag is for.
> > > > >
> > > > > I'm fairly sure the RTC thing could be made deterministic.
> > > >
> > > > We've already tried that once, and it happens a lot less often. If we
> > > > make it even looser we risk making the test itself useless.
> > >
> > > For sleep, yes, but for rtc it should be deterministic now...next time
> > > you get a failure could you send me the trace?
> >
> > Found one:
> > https://dev.azure.com/u-boot/u-boot/_build/results?buildId=6592&view=logs&j=b6c47816-145c-5bfe-20a7-c6a2572e6c41&t=0929c28c-6e32-5635-9624-54eaa917d713&l=599
>
> I don't seem to have access to that...but it is rtc or sleep?
It was the RTC one, and has since rolled off and been deleted.
> > And note that we have a different set of timeout problems that may or may not
> > be configurable, which is in the upload of the pytest results. I haven't seen
> > if there's a knob for this one yet, within Azure (or the python project we're
> > adding for it).
>
> Oh dear.
>
> >
> > > > > The spawning thing...is there a timeout for that? What actually fails?
> > > >
> > > > It doesn't spawn in time for the framework to get to the prompt. We
> > > > could maybe increase the timeout value. It's always the version test
> > > > that fails.
> > >
> > > Ah OK, yes increasing the timeout makes sense.
> > >
> > > >
> > > > > > > > > I'll note that we don't have this problem with sandbox tests.
> > > > > > > >
> > > > > > > > OK, but that's not relevant?
> > > > > > >
> > > > > > > It is relevant to the discussion about using QEMU instead of sandbox,
> > > > > > > e.g. with the TPM. I recall a discussion with Ilias a while back.
> > > > > >
> > > > > > I'm sure we could make sandbox take too long to start as well, if enough
> > > > > > other things are going on with the system. And sandbox has its own set
> > > > > > of super frustrating issues instead, so I don't think this is a great
> > > > > > argument to have right here (I have to run it in docker, to get around
> > > > > > some application version requirements and exclude event_dump, bootmgr,
> > > > > > abootimg and gpt tests, which could otherwise run, but fail for me).
> > > > >
> > > > > I haven't heard about this before. Is there anything that could be done?
> > > >
> > > > I have no idea what could be done about it since I believe all of them
> > > > run fine in CI, including on this very host, when gitlab invokes it
> > > > rather than when I invoke it. My point here is that sandbox tests are
> > > > just a different kind of picky about things and need their own kind of
> > > > "just hit retry".
> > >
> > > Perhaps this is Python dependencies? I'm not sure, but if you see it
> > > again, please let me know in case we can actually fix this.
> >
> > Alright. So the first pass I took at running sandbox pytest with as
> > little hand-holding as possible I hit the known issue of /boot/vmlinu*
> > being 0400 in Ubuntu. I fixed that and then re-ran and:
> > test/py/tests/test_cleanup_build.py F
> >
> > ========================================== FAILURES ===========================================
> > _________________________________________ test_clean __________________________________________
> > test/py/tests/test_cleanup_build.py:94: in test_clean
> > assert not leftovers, f"leftovers: {', '.join(map(str, leftovers))}"
> > E AssertionError: leftovers: fdt-out.dtb, sha1-pad/sandbox-u-boot.dtb, sha1-pad/sandbox-kernel.dtb, sha1-basic/sandbox-u-boot.dtb, sha1-basic/sandbox-kernel.dtb, sha384-basic/sandbox-u-boot.dtb, sha384-basic/sandbox-kernel.dtb, algo-arg/sandbox-u-boot.dtb, algo-arg/sandbox-kernel.dtb, sha1-pss/sandbox-u-boot.dtb, sha1-pss/sandbox-kernel.dtb, sha256-pad/sandbox-u-boot.dtb, sha256-pad/sandbox-kernel.dtb, sha256-global-sign/sandbox-binman.dtb, sha256-global-sign/sandbox-u-boot.dtb, sha256-global-sign/sandbox-u-boot-global.dtb, sha256-global-sign/sandbox-kernel.dtb, sha256-global-sign-pss/sandbox-binman-pss.dtb, sha256-global-sign-pss/sandbox-u-boot.dtb, sha256-global-sign-pss/sandbox-kernel.dtb, sha256-global-sign-pss/sandbox-u-boot-global-pss.dtb, auto_fit/dt-1.dtb, auto_fit/dt-2.dtb, sha256-pss/sandbox-u-boot.dtb, sha256-pss/sandbox-kernel.dtb, sha256-pss-pad/sandbox-u-boot.dtb, sha256-pss-pad/sandbox-kernel.dtb, hashes/sandbox-kernel.dtb, sha256-basic/sandbox-u-boot.dtb, sha256-basic/sandbox-kernel.dtb, sha1-pss-pad/sandbox-u-boot.dtb, sha1-pss-pad/sandbox-kernel.dtb, sha384-pad/sandbox-u-boot.dtb, sha384-pad/sandbox-kernel.dtb, sha256-pss-pad-required/sandbox-u-boot.dtb, sha256-pss-pad-required/sandbox-kernel.dtb, ecdsa/sandbox-kernel.dtb, sha256-pss-required/sandbox-u-boot.dtb, sha256-pss-required/sandbox-kernel.dtb
> > E assert not [PosixPath('fdt-out.dtb'), PosixPath('sha1-pad/sandbox-u-boot.dtb'), PosixPath('sha1-pad/sandbox-kernel.dtb'), PosixPa...ic/sandbox-u-boot.dtb'), PosixPath('sha1-basic/sandbox-kernel.dtb'), PosixPath('sha384-basic/sandbox-u-boot.dtb'), ...]
> > ------------------------------------ Captured stdout call -------------------------------------
> > +make O=/tmp/pytest-of-trini/pytest-231/test_clean0 clean
> > make[1]: Entering directory '/tmp/pytest-of-trini/pytest-231/test_clean0'
> > CLEAN cmd
> > CLEAN dts/../arch/sandbox/dts
> > CLEAN dts
> > CLEAN lib
> > CLEAN tools
> > CLEAN tools/generated
> > CLEAN include/bmp_logo.h include/bmp_logo_data.h include/generated/env.in include/generated/env.txt drivers/video/u_boot_logo.S u-boot u-boot-dtb.bin u-boot-initial-env u-boot-nodtb.bin u-boot.bin u-boot.cfg u-boot.dtb u-boot.dtb.gz u-boot.dtb.out u-boot.dts u-boot.lds u-boot.map u-boot.srec u-boot.sym System.map image.map keep-syms-lto.c lib/efi_loader/helloworld_efi.S
> > make[1]: Leaving directory '/tmp/pytest-of-trini/pytest-231/test_clean0'
> > =================================== short test summary info ===================================
> > FAILED test/py/tests/test_cleanup_build.py::test_clean - AssertionError: leftovers: fdt-out....
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > ================================= 1 failed, 6 passed in 6.42s =================================
>
> That test never passes for me locally, because as you say we add a lot
> of files to the build directory and there is no tracking of them such
> that 'make clean' could remove them. We could fix that, e.g.:
>
> 1. Have binman record all its output filenames in a binman.clean file
> 2. Have tests always use a 'testfiles' subdir for files they create
It sounds like this is showing some bugs in how we use binman since
"make clean" should result in a clean tree, and I believe we get a few
patches now and again about removing leftover files.
> > Fixing that manually with an rm -rf of /tmp/pytest-of-trini and now it's
> > stuck. I've rm -rf'd that and git clean -dfx and just repeat that
> > failure. I'm hopeful that when I reboot whatever magic is broken will
> > be cleaned out. Moving things in to a docker container again, I get:
> > =========================================== ERRORS ============================================
> > _______________________________ ERROR at setup of test_gpt_read _______________________________
> > /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:74: in state_disk_image
> > ???
> > /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:37: in __init__
> > ???
> > test/py/u_boot_utils.py:279: in __enter__
> > self.module_filename = module.__file__
> > E AttributeError: 'NoneType' object has no attribute '__file__'
> > =================================== short test summary info ===================================
> > ERROR test/py/tests/test_gpt.py::test_gpt_read - AttributeError: 'NoneType' object has no at...
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > ========================== 41 passed, 45 skipped, 1 error in 19.29s ===========================
> >
> > And then ignoring that one with "-k not gpt":
> > test/py/tests/test_android/test_ab.py E
> >
> > =========================================== ERRORS ============================================
> > __________________________________ ERROR at setup of test_ab __________________________________
> > /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:54: in ab_disk_image
> > ???
> > /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:28: in __init__
> > ???
> > test/py/u_boot_utils.py:279: in __enter__
> > self.module_filename = module.__file__
> > E AttributeError: 'NoneType' object has no attribute '__file__'
> > =================================== short test summary info ===================================
> > ERROR test/py/tests/test_android/test_ab.py::test_ab - AttributeError: 'NoneType' object has...
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > ============= 908 passed, 75 skipped, 10 deselected, 1 error in 159.17s (0:02:39) =============
>
> These two are the same error. It looks like somehow it is unable to
> obtain the module with:
>
> frame = inspect.stack()[1]
> module = inspect.getmodule(frame[0])
>
> i.e. module is None
>
> +Stephen Warren who may know
>
> What Python version is this?
It's the docker container we use for CI, where these tests pass every
time they're run normally, whatever is in Ubuntu "Jammy".
> > Now, funny things. If I git clean -dfx, I can then get that test to
> > pass. So I guess something else isn't cleaning up / is writing to a
> > common area? I intentionally build within the source tree, but in a
> > subdirectory of that, and indeed a lot of tests write to the source
> > directory itself.
>
> Wow that really is strange. The logic in that class is pretty clever.
> Do you see a message like 'Waiting for generated file timestamp to
> increase' at any point?
>
> BTW these problems don't have anything to do with sandbox, which I
> think was your original complaint. The more stuff we bring into tests
> (Python included) the harder it gets.
The original complaint, as I saw it, was that "sandbox pytests don't
randomly fail". My point is that sandbox pytests randomly fail all the
time. QEMU isn't any worse. I can't say it's better since my local
loop is sandbox for sanity then on to hardware.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20230727/776c6ecb/attachment.sig>
More information about the U-Boot
mailing list