[U-Boot] [PATCH 0/15] buildman: Expand test coverage

[Simon Glass]

Buildman's test coverage is inadequate, particularly in the area of the
core builder threads and logic. As a result it is harder to make changes
than it should be, since verifying correctness manually is tedious.

The existing output test relies on the user to verify that things look
OK. This is getting harder with more output options available, so this
series turns it into a unit test.

A new functional test is provided which tests buildman from the parser
options down to the logic that issues git and make commands. This runs
in a few seconds and provides coverage of the builder logic and threads,
plus most build-related options.

Output formatting is already tested in test.py, and there is also a test
there which checks that errors and warnings are correctly detected by the
build system and reported in the summary.

So overall, with this series, test coverage is now considerably better.


Simon Glass (15):
  patman: Add a way of recording terminal output for testing
  buildman: Send builder output through a function for testing
  buildman: Enhance basic test to check summary output
  patman: RunPipe() should not pipe stdout/stderr unless asked
  buildman: Move the command line code into its own file
  buildman: Move full help code into the control module
  patman: Provide a way to intercept commands for testing
  buildman: Add a functional test
  buildman: Set up bsettings outside the control module
  buildman: Avoid looking at config file or toolchains in tests
  buildman: Allow tests to have their own boards
  buildman: Correct counting of build failures on retry
  buildman: Provide an internal option to clean the outpur dir
  patman: Start with a clean series when needed
  buildman: Add additional functional tests

 tools/buildman/bsettings.py     |  15 +-
 tools/buildman/builder.py       |  58 ++---
 tools/buildman/builderthread.py |  15 +-
 tools/buildman/buildman.py      |  98 +-------
 tools/buildman/cmdline.py       |  85 +++++++
 tools/buildman/control.py       |  60 +++--
 tools/buildman/func_test.py     | 503 ++++++++++++++++++++++++++++++++++++++++
 tools/buildman/test.py          | 101 +++++++-
 tools/buildman/toolchain.py     |   4 +-
 tools/patman/command.py         |  22 ++
 tools/patman/patchstream.py     |   4 +-
 tools/patman/terminal.py        |  61 +++++
 12 files changed, 875 insertions(+), 151 deletions(-)
 create mode 100644 tools/buildman/cmdline.py
 create mode 100644 tools/buildman/func_test.py

-- 
2.1.0.rc2.206.gedb03e5