[tbot] Composition of testcases

[Harald Seiler]

Hi Stefano,

On Mon, 2018-12-03 at 16:13 +0100, Stefano Babic wrote:
> Hi,
> 
> it is not clear to me how I combine different testcases with different
> parameter input. To make a concrete case, I have written a testcase that
> update a board via SWUpdate using the Web interface. A second testcase
> just check SWUpdate's version before running.
> 
> So I have the following testcases:
> 
> @tbot.testcase
> def swupdateweb(
>     mach: typing.Optional[linux.LinuxMachine] = None,
>     ip = None,
>     path = None,
> ) -> None:
> 
> and:
> 
> @tbot.testcase
> def checkswupdate(
>     mach: typing.Optional[linux.LinuxMachine] = None,
> ) -> None:
> 
> 
> to run all of them, I have a "board" testcase where I list them as:
> 
>         tc.testsuite(
>             checkswupdate,
>             swupdateweb, # noqa: F405
>             mach=lh,
>             path=path,
>             ip=ip
>         )
> 
> The thing is that "checkswupdate" fails because it gets unknown
> parameters (path and ip). The update itseld ("swupdateweb") works.

The issue here is that the signatures of all testcases supplied
to tc.testsuite must align.  You can fix this by changing the second
testcase like this:

	@tbot.testcase
	def checkswupdate(
	    mach: typing.Optional[linux.LinuxMachine] = None,
	    **kwargs: typing.Any,
	) -> None:
	    ...

The **kwargs syntax just collects all remaining keyword-arguments into a
dict[1].  In that case, because you don't need them, they are just
ignored.

[1]: https://pythontips.com/2013/08/04/args-and-kwargs-in-python-explained/

> stefano at papero:~/Projects/internal/tbot-denx$ tbot -b boards/gtu.py
> gtuswupdate
> Warning: Duplicate testcase 'checkswupdate'
> Warning: Duplicate testcase 'swupdateweb'
> tbot starting ...
> ├─Calling gtuswupdate ...
> │   ├─POWERON (gtu)
> │   ├─UBOOT (gtu-uboot)
> │   ├─LINUX (gtu-linux)
> │   ├─Calling testsuite ...
> │   │   ├─Calling checkswupdate ...
> │   │   │   └─Fail. (0.000s)
> │   │   ├─Calling swupdateweb ...
> │   │   │   ├─Start uploading image...
> │   │   │   ├─Software image uploaded successfully. Wait for
> installation to be finished...
> │   │   │   │
> │   │   │   ├─Wait update finished
> │   │   │   ├─[network_initializer] : Valid image found: copying to FLASH
> │   │   │   ├─Installation in progress
> │   │   │   ├─SWUPDATE successful !
> │   │   │   └─Done. (50.870s)
> │   │   ├─────────────────────────────────────────
> │   │   │ Failure: 1/2 tests failed
> │   │   │
> │   │   ├─checkswupdate:
> │   │   │ Traceback (most recent call last):
> │   │   │   File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/tc/__init__.py",
> line 12, in testsuite
> │   │   │     test(**kwargs)
> │   │   │   File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/__init__.py",
> line 55, in wrapped
> │   │   │     result = tc(*args, **kwargs)
> │   │   │ TypeError: checkswupdate() got an unexpected keyword argument
> 'path'
> │   │   │
> │   │   └─Fail. (50.872s)
> │   ├─POWEROFF (gtu)
> │   └─Fail. (79.512s)
> ├─Exception:
> │   Traceback (most recent call last):
> │     File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/main.py",
> line 220, in main
> │       testcases[tc]()
> │     File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/__init__.py",
> line 55, in wrapped
> │       result = tc(*args, **kwargs)
> │     File
> "/home/stefano/Projects/internal/tbot-denx/tc/gtuswupdate.py", line 32,
> in gtuswupdate
> │       ip=ip
> │     File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/__init__.py",
> line 55, in wrapped
> │       result = tc(*args, **kwargs)
> │     File
> "/home/stefano/.local/lib/python3.6/site-packages/tbot-0.6.3-py3.6.egg/tbot/tc/__init__.py",
> line 31, in testsuite
> │       raise Exception(f"{len(errors)}/{len(args)} tests failed")
> │   Exception: 1/2 tests failed
> ├─────────────────────────────────────────
> ├─Log written to
> '/home/stefano/Projects/internal/tbot-denx/log/none-gtu-0039.json'
> └─FAILURE (79.646s)
> 
> It is clear why it fials, but it seems to me quite normal that testcases
> have different set of parameters. Which is the solution ?
> 
> Best regards,
> Stefano
> 

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws at denx.de