[PATCH 00/43] test: Improvements to ut command and test-suite running
Heinrich Schuchardt
xypron.glpk at gmx.de
Wed Jan 15 16:32:40 CET 2025
On 15.01.25 14:30, Simon Glass wrote:
> The current method of running unit tests relies on subcommands of the
> ut command. Only the code in each subcommand knows how to find the tests
> related to that subcomand.
>
> This is not ideal and we now have quite a few subcommands which do
> nothing but locate the relevant tests in a linker list, then call a
> common function to run them.
>
> This series adds a list of test suites, so that these subcommands can be
> removed.
>
> An issue with 'ut all' is that it doesn't record how many tests failed
> overall, so it is necessary to examine copious amounts of output to look
> for failures. This series adds a new 'total' feature allow recording the
> total number of failed tests.
>
> To help with 'ut all' a new pytest is created which runs it (as well as
> 'ut info') and makes sure that all is well. Due to the 'ut all' failures
> this does not pass, so the test is disabled for now. It is here because
> it provides security against misnaming a test suite and causing it not
> to run.
>
> Future work may:
> - get 'ut all' passing
> - enable test_suite() in CL, to ensure that 'ut all' keeps passing
> - record duration of each suite
> - allow running the tests in random order to tease out dependencies
> - tweak the output to remove common prefixes
> - getting rid of bootstd, optee and seame 'ut' subcommands
Am I missing a patch for doc/ or will the usage of the ut command remain
unchanged?
How would you specify a test suite like 'lib' in future?
How would you specify running all tests in future?
Best regards
Heinrich
>
>
> Simon Glass (43):
> test: Drop unused suite prototypes
> test: Rename test_get_state() to ut_get_state()
> test: Add functions to init and uninit the test state
> test: Pass the test-state into ut_run_list()
> Improve support for linker lists in data structures
> test: Add newlines to hush-test messages
> test: Rename test suites to match their linker-list name
> test: Update ut info to show suites
> test/py: Add a test which runs all unit tests
> test: Drop the _test suffix on linker lists
> test: Introduce a better array of test suites
> test: Drop the function for running bdinfo tests
> test: Drop the function for running cmd tests
> test: Drop the function for running common tests
> test: Drop the function for running dm tests
> test: Drop the function for running env tests
> test: Drop the function for running exit tests
> test: Drop the function for running fdt tests
> test: Drop the function for running font tests
> test: Drop the function for running lib tests
> test: Drop the function for running log tests
> test: Drop the function for running mbr tests
> test: Drop the function for running mem tests
> test: Drop the function for running setexpr tests
> test: Drop the function for running measurement tests
> test: Drop the function for running bloblist tests
> test: Drop the function for running bootm tests
> test: Drop the function for running addrmap tests
> test: Drop the function for running hush tests
> test: Drop the function for running loadm tests
> test: Drop the function for running pci_mps tests
> test: Drop the function for running seama tests
> test: Drop the function for running upl tests
> test: Drop the info test from the list
> test: Drop conditional compilation for suites
> test: Pass the test state to cmd_ut_category()
> test: Move stats into a struct
> test: Keep a track of the numbers of tests run
> test: Move stat-printing into its own function
> test: Record and show the totals for all test runs
> test: Sort the test suites
> test: Disable test_suite
> test: Move help into the suite declaration
>
> arch/sandbox/cpu/spl.c | 6 +-
> include/dm/test.h | 2 +-
> include/linker_lists.h | 51 +++++
> include/test/cmd.h | 2 +-
> include/test/common.h | 2 +-
> include/test/env.h | 2 +-
> include/test/hush.h | 2 +-
> include/test/lib.h | 2 +-
> include/test/log.h | 4 +-
> include/test/optee.h | 2 +-
> include/test/overlay.h | 2 +-
> include/test/suites.h | 53 ++---
> include/test/test.h | 23 ++-
> include/test/ut.h | 45 ++++-
> test/boot/bootm.c | 11 +-
> test/boot/bootstd_common.c | 9 +-
> test/boot/bootstd_common.h | 3 +-
> test/boot/measurement.c | 12 +-
> test/boot/upl.c | 11 +-
> test/cmd/Makefile | 4 +-
> test/cmd/addrmap.c | 11 +-
> test/cmd/bdinfo.c | 10 +-
> test/cmd/cmd_ut_cmd.c | 20 --
> test/cmd/exit.c | 11 +-
> test/cmd/fdt.c | 10 +-
> test/cmd/font.c | 10 +-
> test/cmd/loadm.c | 11 +-
> test/cmd/mbr.c | 10 +-
> test/cmd/mem.c | 19 --
> test/cmd/mem_copy.c | 2 +-
> test/cmd/mem_search.c | 2 +-
> test/cmd/pci_mps.c | 12 +-
> test/cmd/seama.c | 11 +-
> test/cmd/setexpr.c | 11 +-
> test/cmd_ut.c | 361 +++++++++++++++++++++-------------
> test/common/Makefile | 1 -
> test/common/bloblist.c | 12 +-
> test/common/cmd_ut_common.c | 21 --
> test/dm/Makefile | 2 -
> test/dm/test-dm.c | 16 --
> test/dm/test-driver.c | 10 +-
> test/dm/test-uclass.c | 8 +-
> test/env/cmd_ut_env.c | 10 -
> test/hush/Makefile | 1 -
> test/hush/cmd_ut_hush.c | 19 --
> test/hush/dollar.c | 7 +-
> test/hush/loop.c | 4 +-
> test/lib/Makefile | 1 -
> test/lib/cmd_ut_lib.c | 19 --
> test/log/Makefile | 2 -
> test/log/log_ut.c | 20 --
> test/optee/cmd_ut_optee.c | 4 +-
> test/overlay/cmd_ut_overlay.c | 14 +-
> test/py/conftest.py | 7 +-
> test/py/tests/test_spl.py | 2 +-
> test/py/tests/test_suite.py | 182 +++++++++++++++++
> test/py/tests/test_upl.py | 2 +-
> test/py/tests/test_ut.py | 2 +-
> test/py/tests/test_vbe.py | 2 +-
> test/py/tests/test_vpl.py | 2 +-
> test/test-main.c | 104 ++++++----
> test/ut.c | 4 +-
> 62 files changed, 657 insertions(+), 580 deletions(-)
> delete mode 100644 test/cmd/cmd_ut_cmd.c
> delete mode 100644 test/cmd/mem.c
> delete mode 100644 test/common/cmd_ut_common.c
> delete mode 100644 test/dm/test-dm.c
> delete mode 100644 test/hush/cmd_ut_hush.c
> delete mode 100644 test/lib/cmd_ut_lib.c
> delete mode 100644 test/log/log_ut.c
> create mode 100644 test/py/tests/test_suite.py
>
More information about the U-Boot
mailing list