[PATCH 1/1] gitlab: show skipped Python tests

[Simon Glass]

Hi,

On Mon, 22 Jun 2020 at 12:46, Tom Rini <trini at konsulko.com> wrote:
>
> On Mon, Jun 22, 2020 at 12:23:35PM -0600, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Mon, 22 Jun 2020 at 10:40, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> > >
> > > On 22.06.20 18:17, Simon Glass wrote:
> > > > Hi Heinrich,
> > > >
> > > > On Mon, 22 Jun 2020 at 10:07, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> > > >>
> > > >> Call pytest3 with argument -ra to display reason why Python tests are
> > > >> skipped.
> > > >>
> > > >> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
> > > >> ---
> > > >>  .gitlab-ci.yml | 2 +-
> > > >>  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >>
> > > >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > >> index f2e491c117..f53098ea5f 100644
> > > >> --- a/.gitlab-ci.yml
> > > >> +++ b/.gitlab-ci.yml
> > > >> @@ -46,7 +46,7 @@ stages:
> > > >>      # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
> > > >>      - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
> > > >>        export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
> > > >> -      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
> > > >> +      ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
> > > >>          ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
> > > >>          --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
> > > >
> > > > Do you have a link showing the current output with this patch?
> > >
> > > Hello Simon,
> > >
> > > here is an example output:
> > >
> > > https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112385
> >
> > That's what I was afraid of. The skip output is more than the normal
> > output, and if we don't intend to fix it, I'd rather not have
> > unactionable warnings in the output.
> >
> > Having said that, we need to enable SPI flash, FPGA and MMC
> > environment tests by the look of it.
>
> On a different note, I think we should look at:
> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112376
> and:
> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112380
>
> as it shows that the reason we probably skip the test_fs/test_mkdir.py
> tests is that since board is literal, we don't match sandbox on
> sandbox_flattree.  That answers one outstanding question about why we
> skip some tests and not others at least.

Hmm yes.

It is definitely good to have this output so we can figure out what
should not be skipped.

But outputting things which we know should be skipped just means we
won't notice those that are not supposed to be skipped. How do we
handle that?

Regards,
Simon