[PATCH 0/3] test: Try to deal with some co-dependent tests

[Simon Glass]

Hi Heinrich,

On Mon, 8 Feb 2021 at 00:25, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 2/8/21 5:05 AM, Simon Glass wrote:
> > Tests are supposed to be independent. With driver model tests, the
> > environment is reset before each test, which ensures that.
> >
> > With Python tests there is no reset of the board between tests, since we
> > want to run all the tests as quickly as possible and without needing the
> > external scripts running constantly.
> >
> > In principle the Python tests can be independent if they each put the
> > world back the way they found it, but it turns out that some are not.
> > This means that some tests cannot be run unless another test is run
> > first. It also means that tests cannot be run in parallel, e.g. on
> > sandbox.
> >
> > This series fixes some of them. Those that remain:
> >
> >     test_gpt_swap_partitions - not sure?
> >     test_pinmux_status - not sure?
> >     test_sqfs_load - cannot be run more than once!
> >     test_bind_unbind_with_uclass - relies on previous test
> >
> > The last one would be much better done as a C test, so it doesn't have
> > to deal with the changing driver tree. There isn't a lot of value in
> > running the test on a real board, since sandbox should find any bugs
> > in driver model or the 'bind' command.
> >
> > If the above can be resolved we can enable parallel tests. On my test
> > machine (32 threads) it reduces the time from 38 seconds to 7.5s
> >
> > To use this feature:
> >     pip3 install pytest-xdist
> >
> >     test/py/test.py -B sandbox --build-dir /tmp/xx -q -k 'not slow' -n32
>
> Thanks for looking into parallelization.
>
> What I am missing in this series is a patch for
> doc/develop/py_testing.rst describing how parallelization of Python
> tests is controlled.

I don't think we can mention that yet, as it doesn't actually work.

>
> I have seen that test/py/tests/test_fs/test_basic.py test_fs1() is
> always failing on my machine because the test file 2.5GB.file is
> truncated. It is not truncated if I add some waiting time.
>
> Could this be caused by parallelization?

Not at present since we don't use it. I seldom run those tests. I
wonder whether they work once and then not again, i.e. something needs
to be reset at the start of that test?

Also I have not even tried to parallelise those tests. For me they use
too much memory.

>
> Package 'python3-pytest-xdist' is not installed on my system.

OK. Then the -n flag is not available.

Regards,
Simon