[PATCH v3 00/41] test: Refactor tests to have a single test runner

[Tom Rini]

On Wed, Mar 03, 2021 at 03:18:15PM -0500, Tom Rini wrote:
> On Wed, Mar 03, 2021 at 02:37:51PM -0500, Tom Rini wrote:
> > On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote:
> > 
> > > At present U-Boot has two broad sets of tests in the C code: driver model
> > > tests which do a lot of pre-/post-init and command tests which do not.
> > > 
> > > This separation makes it slightly harder to write a test, since there are
> > > two different test-state structures and different rules for running the
> > > two different test types. At present these rules are determined by where
> > > the test is (actually its prefix).
> > > 
> > > All unit tests can be run from the command line with the 'ut' command.
> > > Since SPL does not have commands, it currently calls the test runner
> > > directly and offers no control of which tests are run.
> > > 
> > > This seems like a good time to refactor the tests into a unified test
> > > runner, allowing U-Boot proper and SPL to use the same path, perhaps with
> > > some different conditions along the way.
> > > 
> > > This series sets up a unified runner called ut_run_list(), which runs a
> > > set of tests from a linker_list. Driver model tests are distinguished by
> > > a new UT_TESTF_DM flag so that the necessary init and cleanup can still
> > > be done.
> > > 
> > > The runner is modified to support running SPL tests that are not solely
> > > for driver model. An example test for FIT loading is added as a
> > > demonstration.
> > > 
> > > In addition, some documentation is added to explain how to write tests.
> > > 
> > > This series is available at u-boot-dm/test-working
> > > 
> > > Changes in v3:
> > > - Add new patch to re-enable test_ofplatdata
> > > - Reword the SPL tests section for clarity
> > > - Use test_set_state() throughout test-main.c instead of direct assignment
> > > - Update the pytest collector as well
> > > 
> > > Changes in v2:
> > > - Use correct rst format for 'Ad-hoc tests' section
> > > - Expand docs on how each type of test is marked
> > > - Put the docs in tests_sandbox since it is more related to sandbox
> > > - Put in a mention of tests_sandbox in the main testing docs
> > > - Add a note that SPL tests can in fact be run individualy
> > > - Document how to run all C tests with 'ut all'
> > > - Fix 'get list' typo
> > > - Fix conditions so non-DM SPL tests are actually run
> > > - Allow for prefix to be NULL, to match function comment
> > > - Add new patches to cover running an SPL test
> > 
> > There was a small problem applying one or two of these patches that I
> > fixed up.  But then when testing locally (Ubuntu 18.04, but virtualenv
> > setup and populated with test/py/requirements.txt) like 200 tests failed
> > to run for Sandbox.  Please check this and that CI runs.
> 
> Er, something has gone odd for me locally, please disregard.

Two problems!  One, this series IS breaking tests for me.  Fresh venv,
~260 tests fail to run.  Two, I didn't notice a fail-to-build in my
build+pytest script, so the binary from this really really failed with
the current pytest scripts :)

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210303/7885fbee/attachment.sig>