[PATCH 0/3] test: Try to deal with some co-dependent tests

[Heinrich Schuchardt]

On 2/8/21 5:05 AM, Simon Glass wrote:
> Tests are supposed to be independent. With driver model tests, the
> environment is reset before each test, which ensures that.
>
> With Python tests there is no reset of the board between tests, since we
> want to run all the tests as quickly as possible and without needing the
> external scripts running constantly.
>
> In principle the Python tests can be independent if they each put the
> world back the way they found it, but it turns out that some are not.
> This means that some tests cannot be run unless another test is run
> first. It also means that tests cannot be run in parallel, e.g. on
> sandbox.
>
> This series fixes some of them. Those that remain:
>
>     test_gpt_swap_partitions - not sure?
>     test_pinmux_status - not sure?
>     test_sqfs_load - cannot be run more than once!
>     test_bind_unbind_with_uclass - relies on previous test
>
> The last one would be much better done as a C test, so it doesn't have
> to deal with the changing driver tree. There isn't a lot of value in
> running the test on a real board, since sandbox should find any bugs
> in driver model or the 'bind' command.
>
> If the above can be resolved we can enable parallel tests. On my test
> machine (32 threads) it reduces the time from 38 seconds to 7.5s
>
> To use this feature:
>     pip3 install pytest-xdist
>
>     test/py/test.py -B sandbox --build-dir /tmp/xx -q -k 'not slow' -n32

Thanks for looking into parallelization.

What I am missing in this series is a patch for
doc/develop/py_testing.rst describing how parallelization of Python
tests is controlled.

I have seen that test/py/tests/test_fs/test_basic.py test_fs1() is
always failing on my machine because the test file 2.5GB.file is
truncated. It is not truncated if I add some waiting time.

Could this be caused by parallelization?

Package 'python3-pytest-xdist' is not installed on my system.

Best regards

Heinrich

>
>
> Simon Glass (3):
>    test: Allow vboot tests to run in parallel
>    test: Allow hush tests to run in parallel
>    test: Allow tpm2 tests to run in parallel
>
>   test/py/tests/test_hush_if_test.py | 20 ++++--------
>   test/py/tests/test_tpm2.py         | 52 ++++++++++++++++++++++++++----
>   test/py/tests/test_vboot.py        | 30 +++++++++--------
>   3 files changed, 69 insertions(+), 33 deletions(-)
>