[PATCH 00/12] test: Add support for passing arguments to C unit tests

[Simon Glass]

Some tests have two parts: a Python test which handles setup, such as
creating filesystem images or configuring the environment, and a C
test which does the bulk of the work including assertions. At present
there is no way to pass information between these two parts, so
things like paths and device names must be hard-coded in the C test.

This tends to make people write tests in Python, but this is not ideal
for several reasons:

- Python lacks access to things like return codes and has to scan the
  console output to figure out what happened
- Python lacks access to internal state, so it cannot directly check
  the result of an operation
- Python is ~100x slower due to console interaction, etc.
- C tests can be stepped through in gdb, which is much harder with
  Python driving the console
- C tests can exercise internal APIs directly (e.g. fs_read(),
  fs_size()) rather than going through the command layer, so they
  test more precisely what they intend to

This series adds infrastructure for passing typed, runtime arguments
from Python to C unit tests, using a name=value format on the ut
command line.

The series includes:

- Regex assertion helpers for matching variable output in tests

- A private buffer in unit_test_state for test-local temporary data

- Argument-type definitions (string, integer, boolean) with optional
  flags and default values

- A new UNIT_TEST_ARGS() macro for declaring tests with typed
  arguments

- Argument parsing in the test framework, accepting name=value format

- Updates to the ut command to pass arguments through to tests

- Type-checked accessor macros ut_str(), ut_int(), and ut_bool()
  with bounds validation

- Tests for the argument feature covering type checking, optional
  arguments, and argument-parsing failures

- Documentation for the test-parameter feature

- C-based filesystem tests as an example of the hybrid approach,
  with Python wrappers that pass filesystem type, image path, and
  expected MD5 values to the C tests


Simon Glass (12):
  test: Add ut_asserteq_regex() for regex pattern matching
  test: Add a helper to check the next line against a regex
  test: Add a private buffer for tests
  test: Add argument-type definitions
  test: Add a macro to declare unit tests with arguments
  test: Add support for passing arguments to C tests
  test: Enhance the ut command to pass test arguments
  test: Add type-checked argument accessor functions
  test: Add tests for unit-test arguments
  test: Add documentation for the test framework
  test: fs: add C-based filesystem tests
  test: fs: Update Python tests to call C implementations

 arch/sandbox/cpu/spl.c              |   3 +-
 doc/develop/tests_writing.rst       | 169 ++++++++++++
 doc/usage/cmd/ut.rst                |  18 +-
 include/test/fs.h                   |  39 +++
 include/test/test.h                 | 107 ++++++++
 include/test/ut.h                   | 123 ++++++++-
 test/Makefile                       |   1 +
 test/cmd_ut.c                       |  32 ++-
 test/common/Makefile                |   1 +
 test/common/test_args.c             | 176 ++++++++++++
 test/fs/Makefile                    |   3 +
 test/fs/fs_basic.c                  | 407 ++++++++++++++++++++++++++++
 test/py/tests/test_fs/conftest.py   |   4 +-
 test/py/tests/test_fs/test_basic.py | 346 +++++++----------------
 test/test-main.c                    | 170 +++++++++++-
 test/ut.c                           | 117 ++++++++
 16 files changed, 1437 insertions(+), 279 deletions(-)
 create mode 100644 include/test/fs.h
 create mode 100644 test/common/test_args.c
 create mode 100644 test/fs/Makefile
 create mode 100644 test/fs/fs_basic.c

-- 
2.43.0

base-commit: e2fa3e570f83ab0f9ce667ddaec9dc738bcf05b9
branch: testa-us