[PATCH v3 00/41] test: Refactor tests to have a single test runner

[Simon Glass]

Hi Tom,

On Wed, 3 Mar 2021 at 15:37, Tom Rini <trini at konsulko.com> wrote:
>
> On Wed, Mar 03, 2021 at 03:18:15PM -0500, Tom Rini wrote:
> > On Wed, Mar 03, 2021 at 02:37:51PM -0500, Tom Rini wrote:
> > > On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote:
> > >
> > > > At present U-Boot has two broad sets of tests in the C code: driver model
> > > > tests which do a lot of pre-/post-init and command tests which do not.
> > > >
> > > > This separation makes it slightly harder to write a test, since there are
> > > > two different test-state structures and different rules for running the
> > > > two different test types. At present these rules are determined by where
> > > > the test is (actually its prefix).
> > > >
> > > > All unit tests can be run from the command line with the 'ut' command.
> > > > Since SPL does not have commands, it currently calls the test runner
> > > > directly and offers no control of which tests are run.
> > > >
> > > > This seems like a good time to refactor the tests into a unified test
> > > > runner, allowing U-Boot proper and SPL to use the same path, perhaps with
> > > > some different conditions along the way.
> > > >
> > > > This series sets up a unified runner called ut_run_list(), which runs a
> > > > set of tests from a linker_list. Driver model tests are distinguished by
> > > > a new UT_TESTF_DM flag so that the necessary init and cleanup can still
> > > > be done.
> > > >
> > > > The runner is modified to support running SPL tests that are not solely
> > > > for driver model. An example test for FIT loading is added as a
> > > > demonstration.
> > > >
> > > > In addition, some documentation is added to explain how to write tests.
> > > >
> > > > This series is available at u-boot-dm/test-working
> > > >
> > > > Changes in v3:
> > > > - Add new patch to re-enable test_ofplatdata
> > > > - Reword the SPL tests section for clarity
> > > > - Use test_set_state() throughout test-main.c instead of direct assignment
> > > > - Update the pytest collector as well
> > > >
> > > > Changes in v2:
> > > > - Use correct rst format for 'Ad-hoc tests' section
> > > > - Expand docs on how each type of test is marked
> > > > - Put the docs in tests_sandbox since it is more related to sandbox
> > > > - Put in a mention of tests_sandbox in the main testing docs
> > > > - Add a note that SPL tests can in fact be run individualy
> > > > - Document how to run all C tests with 'ut all'
> > > > - Fix 'get list' typo
> > > > - Fix conditions so non-DM SPL tests are actually run
> > > > - Allow for prefix to be NULL, to match function comment
> > > > - Add new patches to cover running an SPL test
> > >
> > > There was a small problem applying one or two of these patches that I
> > > fixed up.  But then when testing locally (Ubuntu 18.04, but virtualenv
> > > setup and populated with test/py/requirements.txt) like 200 tests failed
> > > to run for Sandbox.  Please check this and that CI runs.
> >
> > Er, something has gone odd for me locally, please disregard.
>
> Two problems!  One, this series IS breaking tests for me.  Fresh venv,
> ~260 tests fail to run.  Two, I didn't notice a fail-to-build in my
> build+pytest script, so the binary from this really really failed with
> the current pytest scripts :)

Sorry about that...it needs a fix which I had already brought into
dm/master when I was intending for this series to go to the current
release, but have not sent a pull request for. They are in dm/master
now too and I'll send a pull request for master (that you should also
apply to -next).

http://patchwork.ozlabs.org/project/uboot/patch/20210204042944.2636812-1-sjg@chromium.org/‌
‌

Basically there is a crash in sandbox_spl in a rather obscure case
with of-platdata. Finding that was one of the fruits of adding the SPL
tests.

Regards,
Simon