[PATCH v2 00/44] test: Improvements to ut command and test-suite running
Simon Glass
sjg at chromium.org
Mon Jan 20 22:25:22 CET 2025
The current method of running unit tests relies on subcommands of the
ut command. Only the code in each subcommand knows how to find the tests
related to that subcomand.
This is not ideal and we now have quite a few subcommands which do
nothing but locate the relevant tests in a linker list, then call a
common function to run them.
This series adds a list of test suites, so that these subcommands can be
removed.
An issue with 'ut all' is that it doesn't record how many tests failed
overall, so it is necessary to examine copious amounts of output to look
for failures. This series adds a new 'total' feature allow recording the
total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as
'ut info') and makes sure that all is well. Due to the 'ut all' failures
this does not pass, so the test is disabled for now. It is here because
it provides security against misnaming a test suite and causing it not
to run.
Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands
Changes in v2:
- Correct linker-list documentation
- Add a little more debugging
- Strip lines before processing
- Correct the comment on collect_info()
- Show a better assertion message with missing/extra suites
- Tidy up the comments for struct suite
- Show a message when a suite is not compiled in
- Adjust ut_report() to take a run-count
- Add missing comment for 'total'
- Add new patch to update ut documentation
- Drop patch to disable test_suite.py
Simon Glass (44):
test: Drop unused suite prototypes
test: Rename test_get_state() to ut_get_state()
test: Add functions to init and uninit the test state
test: Pass the test-state into ut_run_list()
Improve support for linker lists in data structures
test: Add newlines to hush-test messages
test: Rename test suites to match their linker-list name
test: Update ut info to show suites
test/py: Add a test which runs all unit tests
test: Drop the _test suffix on linker lists
test: Introduce a better array of test suites
test: Drop the function for running bdinfo tests
test: Drop the function for running cmd tests
test: Drop the function for running common tests
test: Drop the function for running dm tests
test: Drop the function for running env tests
test: Drop the function for running exit tests
test: Drop the function for running fdt tests
test: Drop the function for running font tests
test: Drop the function for running lib tests
test: Drop the function for running log tests
test: Drop the function for running mbr tests
test: Drop the function for running mem tests
test: Drop the function for running setexpr tests
test: Drop the function for running measurement tests
test: Drop the function for running bloblist tests
test: Drop the function for running bootm tests
test: Drop the function for running addrmap tests
test: Drop the function for running hush tests
test: Drop the function for running loadm tests
test: Drop the function for running pci_mps tests
test: Drop the function for running seama tests
test: Drop the function for running upl tests
test: Drop the info test from the list
test: Drop conditional compilation for suites
test: Pass the test state to cmd_ut_category()
test: Move stats into a struct
test: Keep a track of the numbers of tests run
test: Move stat-printing into its own function
test: Record and show the totals for all test runs
test: Sort the test suites
test: Disable test_suite
test: Move help into the suite declaration
doc: Update ut documentation
arch/sandbox/cpu/spl.c | 6 +-
doc/usage/cmd/ut.rst | 74 +++----
include/dm/test.h | 2 +-
include/linker_lists.h | 57 ++++++
include/test/cmd.h | 2 +-
include/test/common.h | 2 +-
include/test/env.h | 2 +-
include/test/hush.h | 2 +-
include/test/lib.h | 2 +-
include/test/log.h | 4 +-
include/test/optee.h | 2 +-
include/test/overlay.h | 2 +-
include/test/suites.h | 53 ++---
include/test/test.h | 24 ++-
include/test/ut.h | 45 ++++-
test/boot/bootm.c | 11 +-
test/boot/bootstd_common.c | 9 +-
test/boot/bootstd_common.h | 3 +-
test/boot/measurement.c | 12 +-
test/boot/upl.c | 11 +-
test/cmd/Makefile | 4 +-
test/cmd/addrmap.c | 11 +-
test/cmd/bdinfo.c | 10 +-
test/cmd/cmd_ut_cmd.c | 20 --
test/cmd/exit.c | 11 +-
test/cmd/fdt.c | 10 +-
test/cmd/font.c | 10 +-
test/cmd/loadm.c | 11 +-
test/cmd/mbr.c | 10 +-
test/cmd/mem.c | 19 --
test/cmd/mem_copy.c | 2 +-
test/cmd/mem_search.c | 2 +-
test/cmd/pci_mps.c | 12 +-
test/cmd/seama.c | 11 +-
test/cmd/setexpr.c | 11 +-
test/cmd_ut.c | 366 +++++++++++++++++++++-------------
test/common/Makefile | 1 -
test/common/bloblist.c | 12 +-
test/common/cmd_ut_common.c | 21 --
test/dm/Makefile | 2 -
test/dm/test-dm.c | 16 --
test/dm/test-driver.c | 10 +-
test/dm/test-uclass.c | 8 +-
test/env/cmd_ut_env.c | 10 -
test/hush/Makefile | 1 -
test/hush/cmd_ut_hush.c | 19 --
test/hush/dollar.c | 7 +-
test/hush/loop.c | 4 +-
test/lib/Makefile | 1 -
test/lib/cmd_ut_lib.c | 19 --
test/log/Makefile | 2 -
test/log/log_ut.c | 20 --
test/optee/cmd_ut_optee.c | 4 +-
test/overlay/cmd_ut_overlay.c | 14 +-
test/py/conftest.py | 7 +-
test/py/tests/test_spl.py | 2 +-
test/py/tests/test_suite.py | 188 +++++++++++++++++
test/py/tests/test_upl.py | 2 +-
test/py/tests/test_ut.py | 2 +-
test/py/tests/test_vbe.py | 2 +-
test/py/tests/test_vpl.py | 2 +-
test/test-main.c | 106 ++++++----
test/ut.c | 4 +-
63 files changed, 715 insertions(+), 616 deletions(-)
delete mode 100644 test/cmd/cmd_ut_cmd.c
delete mode 100644 test/cmd/mem.c
delete mode 100644 test/common/cmd_ut_common.c
delete mode 100644 test/dm/test-dm.c
delete mode 100644 test/hush/cmd_ut_hush.c
delete mode 100644 test/lib/cmd_ut_lib.c
delete mode 100644 test/log/log_ut.c
create mode 100644 test/py/tests/test_suite.py
--
2.43.0
More information about the U-Boot
mailing list