[U-Boot] [PATCH 09/14] test/py: Provide a way to check that a command fails

[Teddy Reed]

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass <sjg at chromium.org> wrote:
> Sometimes we want to run a command and check that it fails. Add a function
> to handle this. It can check the return code and also make sure that the
> output contains a given error message.
>
> Signed-off-by: Simon Glass <sjg at chromium.org>
> ---
>
>  test/py/u_boot_utils.py | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
>
> diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py
> index 5d638d9..c834e7b 100644
> --- a/test/py/u_boot_utils.py
> +++ b/test/py/u_boot_utils.py
> @@ -185,6 +185,28 @@ def cmd(u_boot_console, cmd_str):
>      """
>      return run_and_log(u_boot_console, cmd_str.split())
>
> +def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg):
> +    """Run a command which is expected to fail.

nit, in the restrictive case, s/which/that/

> +
> +    This runs a command and checks that it fails with the expected return code
> +    and exception method. If not, an exception is raised.
> +
> +    Args:
> +        u_boot_console: A console connection to U-Boot.
> +        cmd: The command to run, as an array of argv[].
> +        retcode: Expected non-zero return code from the command.
> +        msg: String which should be contained within the command's output.

nit, same as above

> +    """
> +    try:
> +        runner = u_boot_console.log.get_runner(cmd[0], sys.stdout)
> +        runner.run(cmd)
> +    except Exception as e:
> +        assert(msg in runner.output)

I understand why this is a partial match, as requiring a complete
match would result in changing several test cases every time debug
output was altered. Not fun.

That said, it seems dangerous. When performing failure tests there are
often several failing side-effects. For example, if you are testing
hash-mismatches in vboot several code paths may complain about
comparison failures, partial output matching may not be able to
isolate the failing logic. You may encounter a false positive if any
side-effect emits the same expected output.

Would optional parameters make sense? They could allow a tester to
restrict matching to the last line of output, or request a complete
string match?

> +    else:
> +        raise Exception('Expected exception, but not raised')

This is a bit vague, does it make sense to include the expected
(failing) retcode too?

> +    finally:
> +        runner.close()
> +
>  ram_base = None
>  def find_ram_base(u_boot_console):
>      """Find the running U-Boot's RAM location.
> --
> 2.8.0.rc3.226.g39d4020
>
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

Let me know if my comments about output matching are too pedantic.
Since there are few existing test cases, the last thing I want to do
is over-engineer some helpful testing facilities. :)

-- 
Teddy Reed V