[PATCH] test: make tests should use pytest -ra

[Heinrich Schuchardt]

On 4/20/20 1:38 AM, Simon Glass wrote:
> On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>>
>> By passing -ra to pytest we get a summary indicating which tests failed
>> and why tests were skipped.
>>
>> Here is an example output:
>>
>> ======================== short test summary info =========================
>> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available
>> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network
>> configuration is defined
>> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
>>
>> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
>> ---
>>  test/run | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>
> This is really noisy - I get lots of extra output. Can we make this an option?

When I run 'make tests' I get 41 out of 199 lines explaining skipped
and failed tests.

Lines like these are noise because there is no actionable information:

test/py/tests/test_fs/test_basic.py
sssssssssssssssssssssssssssssssssssssss [  0%]
test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [  0%]
test/py/tests/test_fs/test_mkdir.py ssssssssssss [  0%]
test/py/tests/test_fs/test_symlink.py ssss [  0%]
test/py/tests/test_fs/test_unlink.py ssssssssssssss [  0%]

This new line has actionable information:

SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16

Next step is to change this line to provide a more useful output, e.g.

-    except CalledProcessError:
-        pytest.skip('Setup failed for filesystem: ' + fs_type)
+    except CalledProcessError as err:
+        pytest.skip('Setup failed for filesystem: ' + fs_type + \
+            ', {}'.format(err))

SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16
build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit
status 127.

Now we know that that the test is wrong by assuming that mkfs.vfat is in
the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we
can fix it.

We should get rid of all skipped tests especially on Travis CI and
Gitlab CI. Further we should provide instructions to set up a local
system to avoid skipping tests.

Simon, why do you want to remove the actionable information?

Best regards

Heinrich