[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Simon Glass]

Hi Stephen,

On 14 November 2015 at 23:53, Stephen Warren <swarren at wwwdotorg.org> wrote:
> This tool aims to test U-Boot by executing U-Boot shell commands using the
> console interface. A single top-level script exists to execute or attach
> to the U-Boot console, run the entire script of tests against it, and
> summarize the results. Advantages of this approach are:
>
> - Testing is performed in the same way a user or script would interact
>   with U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
>   It is asserted that writing test-related code in Python is simpler and
>   more flexible that writing it all in C.
> - It is reasonably simple to interact with U-Boot in this way.
>
> A few simple tests are provided as examples. Soon, we should convert as
> many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role
for unit tests and things like test/dm. But if we can arrange to call
all U-Boot tests (unit and functional) from one 'test.py' command that
would be a win.

I'll look more when I can get it to work - see below.

>
> In the future, I hope to publish (out-of-tree) the hook scripts, relay
> control utilities, and udev rules I will use for my own HW setup.
>
> See README.md for more details!
>
> Signed-off-by: Stephen Warren <swarren at wwwdotorg.org>
> ---
>  .gitignore                           |   1 +
>  test/py/README.md                    | 287 +++++++++++++++++++++++++++++++++++
>  test/py/board_jetson_tk1.py          |   1 +
>  test/py/board_sandbox.py             |   1 +
>  test/py/board_seaboard.py            |   1 +
>  test/py/conftest.py                  | 225 +++++++++++++++++++++++++++
>  test/py/multiplexed_log.css          |  70 +++++++++
>  test/py/multiplexed_log.py           | 172 +++++++++++++++++++++
>  test/py/pytest.ini                   |   5 +
>  test/py/soc_tegra124.py              |   1 +
>  test/py/soc_tegra20.py               |   1 +
>  test/py/test.py                      |  12 ++
>  test/py/test_000_version.py          |   9 ++
>  test/py/test_env.py                  |  96 ++++++++++++
>  test/py/test_help.py                 |   2 +
>  test/py/test_md.py                   |  12 ++
>  test/py/test_sandbox_exit.py         |  15 ++
>  test/py/test_unknown_cmd.py          |   4 +
>  test/py/uboot_console_base.py        | 143 +++++++++++++++++
>  test/py/uboot_console_exec_attach.py |  28 ++++
>  test/py/uboot_console_sandbox.py     |  22 +++
>  21 files changed, 1108 insertions(+)
>  create mode 100644 test/py/README.md
>  create mode 100644 test/py/board_jetson_tk1.py
>  create mode 100644 test/py/board_sandbox.py
>  create mode 100644 test/py/board_seaboard.py
>  create mode 100644 test/py/conftest.py
>  create mode 100644 test/py/multiplexed_log.css
>  create mode 100644 test/py/multiplexed_log.py
>  create mode 100644 test/py/pytest.ini
>  create mode 100644 test/py/soc_tegra124.py
>  create mode 100644 test/py/soc_tegra20.py
>  create mode 100755 test/py/test.py
>  create mode 100644 test/py/test_000_version.py
>  create mode 100644 test/py/test_env.py
>  create mode 100644 test/py/test_help.py
>  create mode 100644 test/py/test_md.py
>  create mode 100644 test/py/test_sandbox_exit.py
>  create mode 100644 test/py/test_unknown_cmd.py
>  create mode 100644 test/py/uboot_console_base.py
>  create mode 100644 test/py/uboot_console_exec_attach.py
>  create mode 100644 test/py/uboot_console_sandbox.py

I get this on my Ubuntu 64-bit machine (14.04.3)

$ ./test/py/test.py --bd sandbox --buildTraceback (most recent call last):
  File "./test/py/test.py", line 12, in <module>
    os.execvp("py.test", args)
  File "/usr/lib/python2.7/os.py", line 344, in execvp
    _execvpe(file, args)
  File "/usr/lib/python2.7/os.py", line 380, in _execvpe
    func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Regards,
Simon