[tbot] Handling return code from test case

[Harald Seiler]

Hello Stefano,

On Thu, 2018-11-22 at 10:08 +0100, Stefano Babic wrote:
> Hi,
> 
> I am trying two simple testcases:
> 
> - first testcase boots a board in Linux and run an existing command
> - second testcase boots a board in Linux and run a not existing command
> 
> As desired, tbot reports success for the first one and failure for the
> second one.
> 
> But let's say I want to test that a command fails, that is my tbot
> testcase should be successful if the command is not found.

First of all, you should distinguish between running a command which
fails (retcode != 0) and running a command that does not exist.  I'd suggest
using the following:

	if not lnx.test("which", "<command>"):
	    # Command does not exist
	    ...

If you were to use

	if not lnx.test("<command>"):
	    ...

instead, you can't distinguish between the command not existing and the
command failing.

> 
> When I call exec0, test of course fails but tbot stops. How is this
> managed ?

`exec0`[1] is just a pretty wrapper around `exec`[2]. `exec` returns a tuple:

	(ret_code, console_output)

`exec0` will raise an exception if the return code is not 0.  In my suggested
code above, I use `test`[3], which is a wrapper that returns `True` if the ret-
code is 0 and `False` otherwise.

> 
> Just to make things clear:
> 
> import contextlib
>  import typing
>  import time
>  import tbot
>  from tbot.machine import linux
> 
>  @tbot.testcase
>  def missingcmd(
>      mach: typing.Optional[linux.LinuxMachine] = None,
>  ) -> None:
>      with contextlib.ExitStack() as cx:
>          if mach is None:
>              lh = cx.enter_context(tbot.acquire_lab())
>              b = cx.enter_context(tbot.acquire_board(lh))
>              lnx = cx.enter_context(tbot.acquire_linux(b))
>          else:
>              lnx = mach
> 
>          lnx.exec0("doesnotexist")
> 
> 
> And tbot should be successful if the command cannot be executed (also, I
> want a "reverse" case).

See my above suggestion.  If you want the testcase to fail if the command
exists, use something like this:

	assert not lnx.test("which", "doesnotexist")

> Regards,
> Stefano
> 

[1] https://rahix.de/tbot/module-linux.html#tbot.machine.linux.LinuxMachine.exec0
[2] https://rahix.de/tbot/module-linux.html#tbot.machine.linux.LinuxMachine.exec
[3] https://rahix.de/tbot/module-linux.html#tbot.machine.linux.LinuxMachine.test

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws at denx.de