[PATCH 15/17] pytest: Collect SPL unit tests

[Simon Glass]

Hi Stephen,

On Mon, 5 Oct 2020 at 15:35, Stephen Warren <swarren at wwwdotorg.org> wrote:
>
> On 10/5/20 1:51 PM, Simon Glass wrote:
> > Hi Stephen,
> >
> > On Mon, 5 Oct 2020 at 13:39, Stephen Warren <swarren at wwwdotorg.org> wrote:
> >>
> >> On 10/3/20 9:25 AM, Simon Glass wrote:
> >>> Add a new test_spl fixture to handle running SPL unit tests.
> >>
> >>> diff --git a/test/py/conftest.py b/test/py/conftest.py
> >>
> >>> @@ -317,10 +318,13 @@ def pytest_generate_tests(metafunc):
> >>>      Returns:
> >>>          Nothing.
> >>>      """
> >>> -
> >>> +    #print('name', metafunc.fixturenames)
> >>
> >> Revert that debug change?
> >
> > Will do.
> >
> >>
> >>> diff --git a/test/py/tests/test_spl.py b/test/py/tests/test_spl.py
> >>
> >>> +    cons.restart_uboot_with_flags(['-u', ut_spl_subtest])
> >>
> >> How is that change reverted when the test runs, so that subsequent tests
> >> are run on the main U-Boot rather than this restarted U-Boot?
> >
> > Well actually at the moment it just continues into U-Boot. It will
> > mostly pass the tests, but probably not all of them.
>
> Looking at existing tests, there are quite a few that do this:
>
> try:
>     test body which might do something nasty to U-Boot
> finally:
>     u_boot_console.restart_uboot()
>
> ... so that might be applicable.
>
> >> It feels like it'd be better to start a separate top-level test run for
> >> this purpose, rather than swap out the U-Boot process in the middle of a
> >> test run.
> >
> > I was hoping that the fixture stuff would take care of that. How would
> > I do a separate top-level test run?
>
> That'd simply be just running test/py/test.py and passing it the
> relevant U-Boot binary/args rather than the main binary. I assume you'd
> want to pass relevant -k option to restrict the set of tests run to
> something relevant, and an appropriate --build-dir option to point at
> the binary.

I'm fiddling with this a bit. I do already have a separate run of the
SPL tests using the mechanism you describe here, in test/run :

# Run tests which require sandbox_spl
run_test "sandbox_spl" ./test/py/test.py --bd sandbox_spl --build \
   -k 'test_ofplatdata or test_handoff or test_spl'

Also, the test_spl() collection function in this patch is used to read
out tests from spl/u-boot-spl and these tests are not present in
u-boot since they are only built for SPL (Makefile ensures this).

So yes I can add a try...finally thing, but in fact all that does is
make every SPL test be followed by a reboot into U-Boot proper, for no
purpose.

Could you please have another look at this?

Regards,
Simon