[PATCH 00/43] test: Improvements to ut command and test-suite running

[Simon Glass]

Hi Mattijs,

On Thu, 16 Jan 2025 at 02:47, Mattijs Korpershoek
<mkorpershoek at baylibre.com> wrote:
>
> Hi Simon,
>
> Thank you for the series.
>
> On mer., janv. 15, 2025 at 06:30, Simon Glass <sjg at chromium.org> wrote:
>
> > The current method of running unit tests relies on subcommands of the
> > ut command. Only the code in each subcommand knows how to find the tests
> > related to that subcomand.
> >
> > This is not ideal and we now have quite a few subcommands which do
> > nothing but locate the relevant tests in a linker list, then call a
> > common function to run them.
> >
> > This series adds a list of test suites, so that these subcommands can be
> > removed.
> >
> > An issue with 'ut all' is that it doesn't record how many tests failed
> > overall, so it is necessary to examine copious amounts of output to look
> > for failures. This series adds a new 'total' feature allow recording the
> > total number of failed tests.
> >
> > To help with 'ut all' a new pytest is created which runs it (as well as
> > 'ut info') and makes sure that all is well. Due to the 'ut all' failures
> > this does not pass, so the test is disabled for now. It is here because
> > it provides security against misnaming a test suite and causing it not
> > to run.
>
> I wanted to test this, but it seems it does not apply on both master or
> next:
>
> With master: 178f6ecb21fe ("Merge patch series "bootstd: Support recording images"")
> With next: d6da3dbaef57 ("Merge patch series "cmd: Add support for optee commands."")
>
> When running:
> $ b4 shazam -s -l --check 20250115133114.590375-1-sjg at chromium.org
>
> I see:
>
> """
> [...] snip
>
> Total patches: 43
> ---
> Applying: test: Drop unused suite prototypes
> Applying: test: Rename test_get_state() to ut_get_state()
> Applying: test: Add functions to init and uninit the test state
> Applying: test: Pass the test-state into ut_run_list()
> Applying: Improve support for linker lists in data structures
> Applying: test: Add newlines to hush-test messages
> Applying: test: Rename test suites to match their linker-list name
> Applying: test: Update ut info to show suites
> Applying: test/py: Add a test which runs all unit tests
> Applying: test: Drop the _test suffix on linker lists
> Applying: test: Introduce a better array of test suites
> Patch failed at 0011 test: Introduce a better array of test suites
> error: sha1 information is lacking or useless (test/cmd_ut.c).
> error: could not build fake ancestor
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> hint: When you have resolved this problem, run "git am --continue".
> hint: If you prefer to skip this patch, run "git am --skip" instead.
> hint: To restore the original branch and stop patching, run "git am --abort".
> """
>
> On what base should this be applied if we want to test this?
> Sorry if this is a sily question, usually I get dependencies from the
> cover letter but I could not find any here.

Yes, it is supposed to be based on -next but there are two patches in
-next but not in the tree I used:

90856d695f2 test: sandbox: fix link error with do_ut_bootm if BLOBLIST=n
6970eeb7885 test: sandbox: fix link error with do_ut_bloblist if BLOBLIST=n

The series is available at
sjg.u-boot.org/u-boot/u-boot/-/tree/tes?ref_type=heads

I'll sort this out when I send v2, but will wait for some more feedback.

[..]

Regards,
SImon