[PATCH 1/1] gitlab: show skipped Python tests

[Heinrich Schuchardt]

On 24.06.20 15:49, Simon Glass wrote:
> Hi,
>
> On Mon, 22 Jun 2020 at 12:46, Tom Rini <trini at konsulko.com> wrote:
>>
>> On Mon, Jun 22, 2020 at 12:23:35PM -0600, Simon Glass wrote:
>>> Hi Heinrich,
>>>
>>> On Mon, 22 Jun 2020 at 10:40, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>>>
>>>> On 22.06.20 18:17, Simon Glass wrote:
>>>>> Hi Heinrich,
>>>>>
>>>>> On Mon, 22 Jun 2020 at 10:07, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>>>>>
>>>>>> Call pytest3 with argument -ra to display reason why Python tests are
>>>>>> skipped.
>>>>>>
>>>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
>>>>>> ---
>>>>>>  .gitlab-ci.yml | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>>> index f2e491c117..f53098ea5f 100644
>>>>>> --- a/.gitlab-ci.yml
>>>>>> +++ b/.gitlab-ci.yml
>>>>>> @@ -46,7 +46,7 @@ stages:
>>>>>>      # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
>>>>>>      - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
>>>>>>        export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
>>>>>> -      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
>>>>>> +      ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
>>>>>>          ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
>>>>>>          --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
>>>>>
>>>>> Do you have a link showing the current output with this patch?
>>>>
>>>> Hello Simon,
>>>>
>>>> here is an example output:
>>>>
>>>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112385
>>>
>>> That's what I was afraid of. The skip output is more than the normal
>>> output, and if we don't intend to fix it, I'd rather not have
>>> unactionable warnings in the output.
>>>
>>> Having said that, we need to enable SPI flash, FPGA and MMC
>>> environment tests by the look of it.
>>
>> On a different note, I think we should look at:
>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112376
>> and:
>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112380
>>
>> as it shows that the reason we probably skip the test_fs/test_mkdir.py
>> tests is that since board is literal, we don't match sandbox on
>> sandbox_flattree.  That answers one outstanding question about why we
>> skip some tests and not others at least.
>
> Hmm yes.
>
> It is definitely good to have this output so we can figure out what
> should not be skipped.
>
> But outputting things which we know should be skipped just means we
> won't notice those that are not supposed to be skipped. How do we
> handle that?
>
> Regards,
> Simon
>
If you have a lines like:

.config feature "cmd_fpga_loadbp" not enabled
board "qemu_arm64" not supported

you know the test is skipped due to configuration.

Other messages clearly tell you that something is not correctly set up:

No env__efi_loader_grub_file binary specified in environment
got empty parameter set ['env__mmc_dev_config']

Best regards

Heinrich