[PATCH] test: make tests should use pytest -ra

[Tom Rini]

On Mon, Apr 20, 2020 at 08:23:08PM +0200, Heinrich Schuchardt wrote:
> On 4/20/20 1:38 AM, Simon Glass wrote:
> > On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
> >>
> >> By passing -ra to pytest we get a summary indicating which tests failed
> >> and why tests were skipped.
> >>
> >> Here is an example output:
> >>
> >> ======================== short test summary info =========================
> >> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available
> >> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network
> >> configuration is defined
> >> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
> >>
> >> Signed-off-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
> >> ---
> >>  test/run | 6 +++---
> >>  1 file changed, 3 insertions(+), 3 deletions(-)
> >>
> >
> > This is really noisy - I get lots of extra output. Can we make this an option?
> 
> When I run 'make tests' I get 41 out of 199 lines explaining skipped
> and failed tests.
> 
> Lines like these are noise because there is no actionable information:
> 
> test/py/tests/test_fs/test_basic.py
> sssssssssssssssssssssssssssssssssssssss [  0%]
> test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [  0%]
> test/py/tests/test_fs/test_mkdir.py ssssssssssss [  0%]
> test/py/tests/test_fs/test_symlink.py ssss [  0%]
> test/py/tests/test_fs/test_unlink.py ssssssssssssss [  0%]
> 
> This new line has actionable information:
> 
> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
> filesystem: fat16
> 
> Next step is to change this line to provide a more useful output, e.g.
> 
> -    except CalledProcessError:
> -        pytest.skip('Setup failed for filesystem: ' + fs_type)
> +    except CalledProcessError as err:
> +        pytest.skip('Setup failed for filesystem: ' + fs_type + \
> +            ', {}'.format(err))
> 
> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
> filesystem: fat16, Command 'mkfs.vfat -F 16
> build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit
> status 127.
> 
> Now we know that that the test is wrong by assuming that mkfs.vfat is in
> the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we
> can fix it.
> 
> We should get rid of all skipped tests especially on Travis CI and
> Gitlab CI. Further we should provide instructions to set up a local
> system to avoid skipping tests.
> 
> Simon, why do you want to remove the actionable information?

OK, so I ran this through "qcheck" and I got a bit more output, but some
of which indicates we should figure out how to enable these tests for
sandbox out of the box.

The follow-up patch I really want to see is passing -ra in the
travis/gitlab/azure file as I very much agree we should get everything
running in CI or have a good reason why not (ie the MMC tests might be
hard to enable on all platforms via qemu but we should make a good try
at it).

And giving better error messages too would be very helpful so it's
clearer why we aren't (and why some people would not want to) run the FS
tests for example as we really haven't figured out a good 100%
non-root-user method to do those tests.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200420/4751f612/attachment.sig>