[PATCH 5/5] patman: test_util: Print test stdout/stderr within test summaries

[Simon Glass]

On Sat, 2 Apr 2022 at 11:06, Alper Nebi Yasak <alpernebiyasak at gmail.com> wrote:
>
> While running tests for a python tool, the tests' outputs get printed in
> whatever order they happen to run, without any indication as to which
> output belongs to which test. Unittest supports capturing these outputs
> and printing them as part of the test summaries, but when a failure or
> error occurs it switches back to printing as the tests run. Testtools
> and subunit tests can do the same as their parts inherit from unittest,
> but they don't outright expose this functionality.
>
> On the unittest side, enable output buffering for the custom test result
> class. Try to avoid ugly outputs by not printing stdout/stderr before
> the test summary for low verbosity levels and for successful tests.
>
> On the subunit side, implement a custom TestProtocolClient that enables
> the same underlying functionality and injects the captured streams as
> additional test details. This causes them to be merged into their test's
> error traceback message, which is later rebuilt into an exception and
> passed to our unittest report class.
>
> Signed-off-by: Alper Nebi Yasak <alpernebiyasak at gmail.com>
> ---
> The way I had to do this for concurrencytest feels hacky, so I am
> ambivalent towards this patch. I'm actually fine with re-running a
> failing binman test alone with --debug -v6 to get better, isolated
> output from it.

Yes that's what I tend to do, but I think we should try this patch. We
can always drop it in a future release if it causes problems.

>
>  tools/concurrencytest/concurrencytest.py | 83 +++++++++++++++++++++++-
>  tools/patman/test_util.py                | 33 +++++++++-
>  2 files changed, 112 insertions(+), 4 deletions(-)

Reviewed-by: Simon Glass <sjg at chromium.org>