[U-Boot] [PATCH v3 0/18] buildman: Expand test coverage

[Simon Glass]

Buildman's test coverage is inadequate, particularly in the area of the
core builder threads and logic. As a result it is harder to make changes
than it should be, since verifying correctness manually is tedious.

The existing output test relies on the user to verify that things look
OK. This is getting harder with more output options available, so this
series turns it into a unit test.

A new functional test is provided which tests buildman from the parser
options down to the logic that issues git and make commands. This runs
in a few seconds and provides coverage of the builder logic and threads,
plus most build-related options.

Output formatting is already tested in test.py, and there is also a test
there which checks that errors and warnings are correctly detected by the
build system and reported in the summary.

So overall, with this series, test coverage is now considerably better.

This test also fixes a few bugs, some reported by Steve Rae:
- The -H options hang
- An incorrect 'failure' count value in some cases
- Correct GetMetaDataForList() action
- Permit '/' in branch names
- Ignore conflicting tags (allows building multiple series together)

Changes in v3:
- Add new patch to permit branch names with an embedded '/'
- Add new patch to ignore conflicting tags in buildman

Changes in v2:
- Add a function to print out the terminal output recorded
- Add a comment to _HandleCommandGit
- Make sure the test temporary directory is removed
- Add patch to expand output test to cover directory prefixes

Simon Glass (18):
  patman: Add a way of recording terminal output for testing
  buildman: Send builder output through a function for testing
  buildman: Enhance basic test to check summary output
  patman: RunPipe() should not pipe stdout/stderr unless asked
  buildman: Move the command line code into its own file
  buildman: Move full help code into the control module
  patman: Provide a way to intercept commands for testing
  buildman: Add a functional test
  buildman: Set up bsettings outside the control module
  buildman: Avoid looking at config file or toolchains in tests
  buildman: Allow tests to have their own boards
  buildman: Correct counting of build failures on retry
  buildman: Provide an internal option to clean the outpur dir
  patman: Start with a clean series when needed
  buildman: Add additional functional tests
  buildman: Expand output test to cover directory prefixes
  buildman: Permit branch names with an embedded '/'
  buildman: Ignore conflicting tags

 tools/buildman/bsettings.py     |  15 +-
 tools/buildman/builder.py       |  58 ++---
 tools/buildman/builderthread.py |  15 +-
 tools/buildman/buildman.py      |  98 +-------
 tools/buildman/cmdline.py       |  85 +++++++
 tools/buildman/control.py       |  73 ++++--
 tools/buildman/func_test.py     | 519 ++++++++++++++++++++++++++++++++++++++++
 tools/buildman/test.py          | 153 +++++++++++-
 tools/buildman/toolchain.py     |   4 +-
 tools/patman/command.py         |  22 ++
 tools/patman/patchstream.py     |   6 +-
 tools/patman/terminal.py        |  72 ++++++
 12 files changed, 955 insertions(+), 165 deletions(-)
 create mode 100644 tools/buildman/cmdline.py
 create mode 100644 tools/buildman/func_test.py

-- 
2.1.0.rc2.206.gedb03e5