[PATCH] CI: Add automatic retry for test.py jobs
Simon Glass
sjg at google.com
Thu Aug 17 15:41:36 CEST 2023
Hi Tom,
On Thu, 27 Jul 2023 at 14:35, Tom Rini <trini at konsulko.com> wrote:
>
> On Thu, Jul 27, 2023 at 01:18:12PM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Sun, 16 Jul 2023 at 12:18, Tom Rini <trini at konsulko.com> wrote:
> > >
> > > On Sat, Jul 15, 2023 at 05:40:25PM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Thu, 13 Jul 2023 at 15:57, Tom Rini <trini at konsulko.com> wrote:
> > > > >
> > > > > On Thu, Jul 13, 2023 at 03:03:57PM -0600, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Wed, 12 Jul 2023 at 14:38, Tom Rini <trini at konsulko.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jul 12, 2023 at 02:32:18PM -0600, Simon Glass wrote:
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > On Wed, 12 Jul 2023 at 11:09, Tom Rini <trini at konsulko.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Jul 12, 2023 at 08:00:23AM -0600, Simon Glass wrote:
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > On Tue, 11 Jul 2023 at 20:33, Tom Rini <trini at konsulko.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > It is not uncommon for some of the QEMU-based jobs to fail not because
> > > > > > > > > > > of a code issue but rather because of a timing issue or similar problem
> > > > > > > > > > > that is out of our control. Make use of the keywords that Azure and
> > > > > > > > > > > GitLab provide so that we will automatically re-run these when they fail
> > > > > > > > > > > 2 times. If they fail that often it is likely we have found a real issue
> > > > > > > > > > > to investigate.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Tom Rini <trini at konsulko.com>
> > > > > > > > > > > ---
> > > > > > > > > > > .azure-pipelines.yml | 1 +
> > > > > > > > > > > .gitlab-ci.yml | 1 +
> > > > > > > > > > > 2 files changed, 2 insertions(+)
> > > > > > > > > >
> > > > > > > > > > This seems like a slippery slope. Do we know why things fail? I wonder
> > > > > > > > > > if we should disable the tests / builders instead, until it can be
> > > > > > > > > > corrected?
> > > > > > > > >
> > > > > > > > > It happens in Azure, so it's not just the broken runner problem we have
> > > > > > > > > in GitLab. And the problem is timing, as I said in the commit.
> > > > > > > > > Sometimes we still get the RTC test failing. Other times we don't get
> > > > > > > > > QEMU + U-Boot spawned in time (most often m68k, but sometimes x86).
> > > > > > > >
> > > > > > > > How do we keep this list from growing?
> > > > > > >
> > > > > > > Do we need to? The problem is in essence since we rely on free
> > > > > > > resources, sometimes some heavy lifts take longer. That's what this
> > > > > > > flag is for.
> > > > > >
> > > > > > I'm fairly sure the RTC thing could be made deterministic.
> > > > >
> > > > > We've already tried that once, and it happens a lot less often. If we
> > > > > make it even looser we risk making the test itself useless.
> > > >
> > > > For sleep, yes, but for rtc it should be deterministic now...next time
> > > > you get a failure could you send me the trace?
> > >
> > > Found one:
> > > https://dev.azure.com/u-boot/u-boot/_build/results?buildId=6592&view=logs&j=b6c47816-145c-5bfe-20a7-c6a2572e6c41&t=0929c28c-6e32-5635-9624-54eaa917d713&l=599
> >
> > I don't seem to have access to that...but it is rtc or sleep?
>
> It was the RTC one, and has since rolled off and been deleted.
>
> > > And note that we have a different set of timeout problems that may or may not
> > > be configurable, which is in the upload of the pytest results. I haven't seen
> > > if there's a knob for this one yet, within Azure (or the python project we're
> > > adding for it).
> >
> > Oh dear.
> >
> > >
> > > > > > The spawning thing...is there a timeout for that? What actually fails?
> > > > >
> > > > > It doesn't spawn in time for the framework to get to the prompt. We
> > > > > could maybe increase the timeout value. It's always the version test
> > > > > that fails.
> > > >
> > > > Ah OK, yes increasing the timeout makes sense.
> > > >
> > > > >
> > > > > > > > > > I'll note that we don't have this problem with sandbox tests.
> > > > > > > > >
> > > > > > > > > OK, but that's not relevant?
> > > > > > > >
> > > > > > > > It is relevant to the discussion about using QEMU instead of sandbox,
> > > > > > > > e.g. with the TPM. I recall a discussion with Ilias a while back.
> > > > > > >
> > > > > > > I'm sure we could make sandbox take too long to start as well, if enough
> > > > > > > other things are going on with the system. And sandbox has its own set
> > > > > > > of super frustrating issues instead, so I don't think this is a great
> > > > > > > argument to have right here (I have to run it in docker, to get around
> > > > > > > some application version requirements and exclude event_dump, bootmgr,
> > > > > > > abootimg and gpt tests, which could otherwise run, but fail for me).
> > > > > >
> > > > > > I haven't heard about this before. Is there anything that could be done?
> > > > >
> > > > > I have no idea what could be done about it since I believe all of them
> > > > > run fine in CI, including on this very host, when gitlab invokes it
> > > > > rather than when I invoke it. My point here is that sandbox tests are
> > > > > just a different kind of picky about things and need their own kind of
> > > > > "just hit retry".
> > > >
> > > > Perhaps this is Python dependencies? I'm not sure, but if you see it
> > > > again, please let me know in case we can actually fix this.
> > >
> > > Alright. So the first pass I took at running sandbox pytest with as
> > > little hand-holding as possible I hit the known issue of /boot/vmlinu*
> > > being 0400 in Ubuntu. I fixed that and then re-ran and:
> > > test/py/tests/test_cleanup_build.py F
> > >
> > > ========================================== FAILURES ===========================================
> > > _________________________________________ test_clean __________________________________________
> > > test/py/tests/test_cleanup_build.py:94: in test_clean
> > > assert not leftovers, f"leftovers: {', '.join(map(str, leftovers))}"
> > > E AssertionError: leftovers: fdt-out.dtb, sha1-pad/sandbox-u-boot.dtb, sha1-pad/sandbox-kernel.dtb, sha1-basic/sandbox-u-boot.dtb, sha1-basic/sandbox-kernel.dtb, sha384-basic/sandbox-u-boot.dtb, sha384-basic/sandbox-kernel.dtb, algo-arg/sandbox-u-boot.dtb, algo-arg/sandbox-kernel.dtb, sha1-pss/sandbox-u-boot.dtb, sha1-pss/sandbox-kernel.dtb, sha256-pad/sandbox-u-boot.dtb, sha256-pad/sandbox-kernel.dtb, sha256-global-sign/sandbox-binman.dtb, sha256-global-sign/sandbox-u-boot.dtb, sha256-global-sign/sandbox-u-boot-global.dtb, sha256-global-sign/sandbox-kernel.dtb, sha256-global-sign-pss/sandbox-binman-pss.dtb, sha256-global-sign-pss/sandbox-u-boot.dtb, sha256-global-sign-pss/sandbox-kernel.dtb, sha256-global-sign-pss/sandbox-u-boot-global-pss.dtb, auto_fit/dt-1.dtb, auto_fit/dt-2.dtb, sha256-pss/sandbox-u-boot.dtb, sha256-pss/sandbox-kernel.dtb, sha256-pss-pad/sandbox-u-boot.dtb, sha256-pss-pad/sandbox-kernel.dtb, hashes/sandbox-kernel.dtb, sha256-basic/sandbox-u-boot.dtb, sha256-basic/sandbox-kernel.dtb, sha1-pss-pad/sandbox-u-boot.dtb, sha1-pss-pad/sandbox-kernel.dtb, sha384-pad/sandbox-u-boot.dtb, sha384-pad/sandbox-kernel.dtb, sha256-pss-pad-required/sandbox-u-boot.dtb, sha256-pss-pad-required/sandbox-kernel.dtb, ecdsa/sandbox-kernel.dtb, sha256-pss-required/sandbox-u-boot.dtb, sha256-pss-required/sandbox-kernel.dtb
> > > E assert not [PosixPath('fdt-out.dtb'), PosixPath('sha1-pad/sandbox-u-boot.dtb'), PosixPath('sha1-pad/sandbox-kernel.dtb'), PosixPa...ic/sandbox-u-boot.dtb'), PosixPath('sha1-basic/sandbox-kernel.dtb'), PosixPath('sha384-basic/sandbox-u-boot.dtb'), ...]
> > > ------------------------------------ Captured stdout call -------------------------------------
> > > +make O=/tmp/pytest-of-trini/pytest-231/test_clean0 clean
> > > make[1]: Entering directory '/tmp/pytest-of-trini/pytest-231/test_clean0'
> > > CLEAN cmd
> > > CLEAN dts/../arch/sandbox/dts
> > > CLEAN dts
> > > CLEAN lib
> > > CLEAN tools
> > > CLEAN tools/generated
> > > CLEAN include/bmp_logo.h include/bmp_logo_data.h include/generated/env.in include/generated/env.txt drivers/video/u_boot_logo.S u-boot u-boot-dtb.bin u-boot-initial-env u-boot-nodtb.bin u-boot.bin u-boot.cfg u-boot.dtb u-boot.dtb.gz u-boot.dtb.out u-boot.dts u-boot.lds u-boot.map u-boot.srec u-boot.sym System.map image.map keep-syms-lto.c lib/efi_loader/helloworld_efi.S
> > > make[1]: Leaving directory '/tmp/pytest-of-trini/pytest-231/test_clean0'
> > > =================================== short test summary info ===================================
> > > FAILED test/py/tests/test_cleanup_build.py::test_clean - AssertionError: leftovers: fdt-out....
> > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > ================================= 1 failed, 6 passed in 6.42s =================================
> >
> > That test never passes for me locally, because as you say we add a lot
> > of files to the build directory and there is no tracking of them such
> > that 'make clean' could remove them. We could fix that, e.g.:
> >
> > 1. Have binman record all its output filenames in a binman.clean file
> > 2. Have tests always use a 'testfiles' subdir for files they create
>
> It sounds like this is showing some bugs in how we use binman since
> "make clean" should result in a clean tree, and I believe we get a few
> patches now and again about removing leftover files.
>
> > > Fixing that manually with an rm -rf of /tmp/pytest-of-trini and now it's
> > > stuck. I've rm -rf'd that and git clean -dfx and just repeat that
> > > failure. I'm hopeful that when I reboot whatever magic is broken will
> > > be cleaned out. Moving things in to a docker container again, I get:
> > > =========================================== ERRORS ============================================
> > > _______________________________ ERROR at setup of test_gpt_read _______________________________
> > > /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:74: in state_disk_image
> > > ???
> > > /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:37: in __init__
> > > ???
> > > test/py/u_boot_utils.py:279: in __enter__
> > > self.module_filename = module.__file__
> > > E AttributeError: 'NoneType' object has no attribute '__file__'
> > > =================================== short test summary info ===================================
> > > ERROR test/py/tests/test_gpt.py::test_gpt_read - AttributeError: 'NoneType' object has no at...
> > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > ========================== 41 passed, 45 skipped, 1 error in 19.29s ===========================
> > >
> > > And then ignoring that one with "-k not gpt":
> > > test/py/tests/test_android/test_ab.py E
> > >
> > > =========================================== ERRORS ============================================
> > > __________________________________ ERROR at setup of test_ab __________________________________
> > > /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:54: in ab_disk_image
> > > ???
> > > /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:28: in __init__
> > > ???
> > > test/py/u_boot_utils.py:279: in __enter__
> > > self.module_filename = module.__file__
> > > E AttributeError: 'NoneType' object has no attribute '__file__'
> > > =================================== short test summary info ===================================
> > > ERROR test/py/tests/test_android/test_ab.py::test_ab - AttributeError: 'NoneType' object has...
> > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > ============= 908 passed, 75 skipped, 10 deselected, 1 error in 159.17s (0:02:39) =============
> >
> > These two are the same error. It looks like somehow it is unable to
> > obtain the module with:
> >
> > frame = inspect.stack()[1]
> > module = inspect.getmodule(frame[0])
> >
> > i.e. module is None
> >
> > +Stephen Warren who may know
> >
> > What Python version is this?
>
> It's the docker container we use for CI, where these tests pass every
> time they're run normally, whatever is in Ubuntu "Jammy".
>
> > > Now, funny things. If I git clean -dfx, I can then get that test to
> > > pass. So I guess something else isn't cleaning up / is writing to a
> > > common area? I intentionally build within the source tree, but in a
> > > subdirectory of that, and indeed a lot of tests write to the source
> > > directory itself.
> >
> > Wow that really is strange. The logic in that class is pretty clever.
> > Do you see a message like 'Waiting for generated file timestamp to
> > increase' at any point?
> >
> > BTW these problems don't have anything to do with sandbox, which I
> > think was your original complaint. The more stuff we bring into tests
> > (Python included) the harder it gets.
>
> The original complaint, as I saw it, was that "sandbox pytests don't
> randomly fail". My point is that sandbox pytests randomly fail all the
> time. QEMU isn't any worse. I can't say it's better since my local
> loop is sandbox for sanity then on to hardware.
The way I see it, terms of flakiness, speed and ease of debugging,
from best to worse, we have:
- sandbox
- Python wrapper around sandbox
- Python wrapper around QEMU
Regards,
Simon
More information about the U-Boot
mailing list