[PATCH 1/1] gitlab: show skipped Python tests

[Heinrich Schuchardt]

On 24.06.20 17:17, Simon Glass wrote:
> Hi Heinrich,
>
> On Wed, 24 Jun 2020 at 07:56, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>
>> On 24.06.20 15:49, Simon Glass wrote:
>>> Hi,
>>>
>>> On Mon, 22 Jun 2020 at 12:46, Tom Rini <trini at konsulko.com> wrote:
>>>>
>>>> On Mon, Jun 22, 2020 at 12:23:35PM -0600, Simon Glass wrote:
>>>>> Hi Heinrich,
>>>>>
>>>>> On Mon, 22 Jun 2020 at 10:40, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>>>>>
>>>>>> On 22.06.20 18:17, Simon Glass wrote:
>>>>>>> Hi Heinrich,
>>>>>>>
>>>>>>> On Mon, 22 Jun 2020 at 10:07, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>>>>>>>
>>>>>>>> Call pytest3 with argument -ra to display reason why Python tests are
>>>>>>>> skipped.
>>>>>>>>
>>>>>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
>>>>>>>> ---
>>>>>>>>  .gitlab-ci.yml | 2 +-
>>>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>>>>> index f2e491c117..f53098ea5f 100644
>>>>>>>> --- a/.gitlab-ci.yml
>>>>>>>> +++ b/.gitlab-ci.yml
>>>>>>>> @@ -46,7 +46,7 @@ stages:
>>>>>>>>      # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
>>>>>>>>      - export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
>>>>>>>>        export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
>>>>>>>> -      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
>>>>>>>> +      ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
>>>>>>>>          ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
>>>>>>>>          --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
>>>>>>>
>>>>>>> Do you have a link showing the current output with this patch?
>>>>>>
>>>>>> Hello Simon,
>>>>>>
>>>>>> here is an example output:
>>>>>>
>>>>>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112385
>>>>>
>>>>> That's what I was afraid of. The skip output is more than the normal
>>>>> output, and if we don't intend to fix it, I'd rather not have
>>>>> unactionable warnings in the output.
>>>>>
>>>>> Having said that, we need to enable SPI flash, FPGA and MMC
>>>>> environment tests by the look of it.
>>>>
>>>> On a different note, I think we should look at:
>>>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112376
>>>> and:
>>>> https://gitlab.denx.de/u-boot/custodians/u-boot-efi/-/jobs/112380
>>>>
>>>> as it shows that the reason we probably skip the test_fs/test_mkdir.py
>>>> tests is that since board is literal, we don't match sandbox on
>>>> sandbox_flattree.  That answers one outstanding question about why we
>>>> skip some tests and not others at least.
>>>
>>> Hmm yes.
>>>
>>> It is definitely good to have this output so we can figure out what
>>> should not be skipped.
>>>
>>> But outputting things which we know should be skipped just means we
>>> won't notice those that are not supposed to be skipped. How do we
>>> handle that?
>>>
>>> Regards,
>>> Simon
>>>
>> If you have a lines like:
>>
>> .config feature "cmd_fpga_loadbp" not enabled
>> board "qemu_arm64" not supported
>>
>> you know the test is skipped due to configuration.
>
> OK then can we avoid printing this useless information by default?
>
>>
>> Other messages clearly tell you that something is not correctly set up:
>>
>> No env__efi_loader_grub_file binary specified in environment
>> got empty parameter set ['env__mmc_dev_config']
>
> OK then this is what we should display.

There is no switch in pytest to display this selectively.

Best regards

Heinrich