[PATCH 1/1] gitlab: show skipped Python tests

[Simon Glass]

Hi Heinrich,

On Wed, 24 Jun 2020 at 07:56, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 24.06.20 15:49, Simon Glass wrote:
> > Hi,
> >
> > On Mon, 22 Jun 2020 at 12:46, Tom Rini <trini at konsulko.com> wrote:
> >>
> >> On Mon, Jun 22, 2020 at 12:23:35PM -0600, Simon Glass wrote:
> >>> Hi Heinrich,
> >>>
> >>> On Mon, 22 Jun 2020 at 10:40, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> >>>>
> >>>> On 22.06.20 18:17, Simon Glass wrote:
> >>>>> Hi Heinrich,
> >>>>>
> >>>>> On Mon, 22 Jun 2020 at 10:07, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> >>>>>>
> >>>>>> Call pytest3 with argument -ra to display reason why Python tests are
> >>>>>> skipped.
> >>>>>>
> >>>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
> >>>>>> ---
> >>>>>>  .gitlab-ci.yml | 2 +-
> >>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>>>>
> >>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> >>>>>> index f2e491c117..f53098ea5f 100644
> >>>>>> --- a/.gitlab-ci.yml
> >>>>>> +++ b/.gitlab-ci.yml
> >>>>>> @@ -46,7 +46,7 @@ stages:
> >>>>>>      # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
> >>>>>>      - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
> >>>>>>        export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
> >>>>>> -      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
> >>>>>> +      ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
> >>>>>>          ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
> >>>>>>          --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
> >>>>>
> >>>>> Do you have a link showing the current output with this patch?
> >>>>
> >>>> Hello Simon,
> >>>>
> >>>> here is an example output:
> >>>>
> >>>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112385
> >>>
> >>> That's what I was afraid of. The skip output is more than the normal
> >>> output, and if we don't intend to fix it, I'd rather not have
> >>> unactionable warnings in the output.
> >>>
> >>> Having said that, we need to enable SPI flash, FPGA and MMC
> >>> environment tests by the look of it.
> >>
> >> On a different note, I think we should look at:
> >> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112376
> >> and:
> >> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112380
> >>
> >> as it shows that the reason we probably skip the test_fs/test_mkdir.py
> >> tests is that since board is literal, we don't match sandbox on
> >> sandbox_flattree.  That answers one outstanding question about why we
> >> skip some tests and not others at least.
> >
> > Hmm yes.
> >
> > It is definitely good to have this output so we can figure out what
> > should not be skipped.
> >
> > But outputting things which we know should be skipped just means we
> > won't notice those that are not supposed to be skipped. How do we
> > handle that?
> >
> > Regards,
> > Simon
> >
> If you have a lines like:
>
> .config feature "cmd_fpga_loadbp" not enabled
> board "qemu_arm64" not supported
>
> you know the test is skipped due to configuration.

OK then can we avoid printing this useless information by default?

>
> Other messages clearly tell you that something is not correctly set up:
>
> No env__efi_loader_grub_file binary specified in environment
> got empty parameter set ['env__mmc_dev_config']

OK then this is what we should display.

Regards,
Simon