[PATCH v2 14/45] test: Support tests which can only be run manually

[Simon Glass]

Hi Heinrich,

On Thu, 13 Oct 2022 at 10:19, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 10/13/22 14:28, Simon Glass wrote:
> > At present we normally write tests either in Python or in C. But most
> > Python tests end up doing a lot of checks which would be better done in C.
> > Checks done in C are orders of magnitude faster and it is possible to get
> > full access to U-Boot's internal workings, rather than just relying on
> > the command line.
> >
> > The model is to have a Python test set up some things and then use C code
> > (in a unit test) to check that they were done correctly. But we don't want
> > those checks to happen as part of normal test running, since each C unit
> > tests is dependent on the associate Python tests, so cannot run without
> > it.
> >
> > To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
> > tests, so that they can be skipped by default when the 'ut' command is
> > used. Require that tests have a name ending with '_norun', so that pytest
>
> Why do want to use a naming convention?

See the line immediately above. The pytests don't have access to the
flags as they are compiled into the executable.

> What will we do when we want more flags like e.g. "slow"?

We already have that in the pytest side. This 'manual' thing is
unusual in that both pytest and C test need to have it.

>
> Adding more fields to struct unit_test would be more future prove.

Well a flag is more compact and suits the purpose. See above for why
we need to add something to the name.

Regards,
Simon

[..]