[PATCH 00/43] test: Improvements to ut command and test-suite running

[Simon Glass]

Hi Heinrich,

On Wed, 15 Jan 2025 at 08:33, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 15.01.25 14:30, Simon Glass wrote:
> > The current method of running unit tests relies on subcommands of the
> > ut command. Only the code in each subcommand knows how to find the tests
> > related to that subcomand.
> >
> > This is not ideal and we now have quite a few subcommands which do
> > nothing but locate the relevant tests in a linker list, then call a
> > common function to run them.
> >
> > This series adds a list of test suites, so that these subcommands can be
> > removed.
> >
> > An issue with 'ut all' is that it doesn't record how many tests failed
> > overall, so it is necessary to examine copious amounts of output to look
> > for failures. This series adds a new 'total' feature allow recording the
> > total number of failed tests.
> >
> > To help with 'ut all' a new pytest is created which runs it (as well as
> > 'ut info') and makes sure that all is well. Due to the 'ut all' failures
> > this does not pass, so the test is disabled for now. It is here because
> > it provides security against misnaming a test suite and causing it not
> > to run.
> >
> > Future work may:
> > - get 'ut all' passing
> > - enable test_suite() in CL, to ensure that 'ut all' keeps passing
> > - record duration of each suite
> > - allow running the tests in random order to tease out dependencies
> > - tweak the output to remove common prefixes
> > - getting rid of bootstd, optee and seame 'ut' subcommands
>
> Am I missing a patch for doc/ or will the usage of the ut command remain
> unchanged?

No change to the command, but the output changes, so I can add an
update to the 'examples' section.

>
> How would you specify a test suite like 'lib' in future?

LIB_TEST(...)

If you want to create a new one, you need to add it to the list in
cmd_ut.c as now. But it is now in a new format which is easier to
maintain.

> How would you specify running all tests in future?

'ut -a' as now

[..]

Regards,
Simon