[PATCH] test: make tests should use pytest -ra

[Heinrich Schuchardt]

On 4/20/20 8:32 PM, Tom Rini wrote:
> On Mon, Apr 20, 2020 at 08:23:08PM +0200, Heinrich Schuchardt wrote:
>> On 4/20/20 1:38 AM, Simon Glass wrote:
>>> On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>>>
>>>> By passing -ra to pytest we get a summary indicating which tests failed
>>>> and why tests were skipped.
>>>>
>>>> Here is an example output:
>>>>
>>>> ======================== short test summary info =========================
>>>> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available
>>>> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network
>>>> configuration is defined
>>>> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
>>>>
>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
>>>> ---
>>>>  test/run | 6 +++---
>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>
>>>
>>> This is really noisy - I get lots of extra output. Can we make this an option?
>>
>> When I run 'make tests' I get 41 out of 199 lines explaining skipped
>> and failed tests.
>>
>> Lines like these are noise because there is no actionable information:
>>
>> test/py/tests/test_fs/test_basic.py
>> sssssssssssssssssssssssssssssssssssssss [  0%]
>> test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [  0%]
>> test/py/tests/test_fs/test_mkdir.py ssssssssssss [  0%]
>> test/py/tests/test_fs/test_symlink.py ssss [  0%]
>> test/py/tests/test_fs/test_unlink.py ssssssssssssss [  0%]
>>
>> This new line has actionable information:
>>
>> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
>> filesystem: fat16
>>
>> Next step is to change this line to provide a more useful output, e.g.
>>
>> -    except CalledProcessError:
>> -        pytest.skip('Setup failed for filesystem: ' + fs_type)
>> +    except CalledProcessError as err:
>> +        pytest.skip('Setup failed for filesystem: ' + fs_type + \
>> +            ', {}'.format(err))
>>
>> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
>> filesystem: fat16, Command 'mkfs.vfat -F 16
>> build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit
>> status 127.
>>
>> Now we know that that the test is wrong by assuming that mkfs.vfat is in
>> the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we
>> can fix it.
>>
>> We should get rid of all skipped tests especially on Travis CI and
>> Gitlab CI. Further we should provide instructions to set up a local
>> system to avoid skipping tests.
>>
>> Simon, why do you want to remove the actionable information?
>
> OK, so I ran this through "qcheck" and I got a bit more output, but some
> of which indicates we should figure out how to enable these tests for
> sandbox out of the box.
>
> The follow-up patch I really want to see is passing -ra in the
> travis/gitlab/azure file as I very much agree we should get everything
> running in CI or have a good reason why not (ie the MMC tests might be
> hard to enable on all platforms via qemu but we should make a good try
> at it).
>
> And giving better error messages too would be very helpful so it's
> clearer why we aren't (and why some people would not want to) run the FS
> tests for example as we really haven't figured out a good 100%
> non-root-user method to do those tests.
>

The tests run fine on my Debian system with
https://github.com/xypron2/u-boot/commit/f465d52a5a539761d9e5602331280ce07a1bcbca
where I assume mkfs.* and fsck.ext4 are in /sbin. The same should work
on Ubuntu.

On current distributions there are also symlinks in /usr/sbin/ but
https://askubuntu.com/questions/674847/sudo-usr-sbin-mkfs-ext4-command-not-found
indicates that this does not hold true for elder distributions.

I will submit the test once tested in Travis CI and Gitlab CI.

Best regards

Heinrich