[PATCH 1/1] gitlab: show skipped Python tests

[Tom Rini]

On Mon, Jun 22, 2020 at 12:23:35PM -0600, Simon Glass wrote:
> Hi Heinrich,
> 
> On Mon, 22 Jun 2020 at 10:40, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> >
> > On 22.06.20 18:17, Simon Glass wrote:
> > > Hi Heinrich,
> > >
> > > On Mon, 22 Jun 2020 at 10:07, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> > >>
> > >> Call pytest3 with argument -ra to display reason why Python tests are
> > >> skipped.
> > >>
> > >> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
> > >> ---
> > >>  .gitlab-ci.yml | 2 +-
> > >>  1 file changed, 1 insertion(+), 1 deletion(-)
> > >>
> > >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > >> index f2e491c117..f53098ea5f 100644
> > >> --- a/.gitlab-ci.yml
> > >> +++ b/.gitlab-ci.yml
> > >> @@ -46,7 +46,7 @@ stages:
> > >>      # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
> > >>      - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
> > >>        export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
> > >> -      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
> > >> +      ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
> > >>          ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
> > >>          --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
> > >
> > > Do you have a link showing the current output with this patch?
> >
> > Hello Simon,
> >
> > here is an example output:
> >
> > https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112385
> 
> That's what I was afraid of. The skip output is more than the normal
> output, and if we don't intend to fix it, I'd rather not have
> unactionable warnings in the output.
> 
> Having said that, we need to enable SPI flash, FPGA and MMC
> environment tests by the look of it.

So, looking at the output.  40 lines of test output, 32 lines of
explanation of why skips.  I think it's still readable.  It also shows
where perhaps we could group / re-org things a little as it already does
smart things like:
SKIPPED [131] /builds/u-boot/custodians/u-boot-efi/test/py/conftest.py:468: board "qemu_arm64" not supported

Looking at the spi flash tests (and also the mmc read/write tests) we
should have a generic check in the tests for them be configured and bail
if not, as that would reduce the skip summary lines and collect them all
in one spot.

> > Some of the skips are ok. But some we should really fix like:
> >
> > SKIPPED [1]
> > /builds/u-boot/custodians/u-boot-efi/test/py/tests/test_efi_loader.py:100:
> > No static network configuration is defined
> 
> Also, I suspect that the warnings will be huge when running on real hardware.

Well, that's beside the point with this patch.  It's very specifically
only doing it on GitLab (and if/when we add this, v2 should cover
travis/Azure) as it's CI and logs are cheap but recreation can be
annoying.  But it's also not true at least here.  I see a few more lines
than the above example on my am335x_evm config, but I also have a few
more skips.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200622/c4f9c9f7/attachment.sig>