[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Heiko Schocher]

Hello Stephen,

Am 02.12.2015 um 23:18 schrieb Stephen Warren:
> This tool aims to test U-Boot by executing U-Boot shell commands using the
> console interface. A single top-level script exists to execute or attach
> to the U-Boot console, run the entire script of tests against it, and
> summarize the results. Advantages of this approach are:
>
> - Testing is performed in the same way a user or script would interact
>    with U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
>    It is asserted that writing test-related code in Python is simpler and
>    more flexible that writing it all in C.
> - It is reasonably simple to interact with U-Boot in this way.
>
> A few simple tests are provided as examples. Soon, we should convert as
> many as possible of the other tests in test/* and test/cmd_ut.c too.
>
> In the future, I hope to publish (out-of-tree) the hook scripts, relay
> control utilities, and udev rules I will use for my own HW setup.
>
> See README.md for more details!
>
> Signed-off-by: Stephen Warren <swarren at wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren at nvidia.com>
> ---

Nice work!

I am working on another python approach, not only good for testing
u-boot, also works with linux, or other console based tests, see:

[1] tbot
https://github.com/hsdenx/tbot

With tbot it is possible to connect to a lab PC over ssh. So no need
to have the hw physically accessible. Currently there is only one lab,
the denx lab in munich (I live in hungary and tbot runs at my home),

It should be really easy to add other labs, if others want to join.
see:
https://github.com/hsdenx/tbot/blob/master/doc/howto_add_new_lab.txt

I set up nightly builds using buildbot
[2] http://buildbot.net/
on my rasspyberry pi at my home (!low dsl speed, especially if my kids
play games, so if you cannot see the webpage press F5 again ;-)

[3] http://xeidos.ddns.net/buildbot/waterfall

For every build there is a logfile, also shell log. The logfile is
generated from tbot. It is possible to define different loglevel,
as tbot uses "logging" python module. Each testcase can add logfile
messages.


Testcases can call other testcases, so you can do very small testcases,
and combinate them to bigger ones ...

As tbot defines board states, it is possible to switch between
u-boot and linux tests, and tbot will switch automatically to the board
state the testcase is defined for ... a testcase has only to call
https://github.com/hsdenx/tbot/blob/master/src/common/tbotlib.py#L677

So it is for example easy to set a date in u-boot, switch to board state
linux, check there the date, set another date in linux, and switch
to u-boot and check there again the time ... (had such a testcase,
but not for current version of tbot)

It would be great to have a lot of labs accessible with board(s) in it
for u-boot/linux tests and make there automated nightly builds ...

Currently there are only basic tbot testcases, but some good examples,
what is possible:
- smartweb dfu test:
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_smartweb.py
   - compile u-boot source from current mainline
   - install it on the board
   - make dfu tests with dfu-util installed on the PC in the Lab

- ari_ubi
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_aristainetos2.py
   - compile current mainline source for the aristainetos2 board
   - install it on the board
   - make u-boot ubi/ubifs test on the nand and spi nor

- tqm5200s
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_tqm5200s_try_cur_ub.py
   - compile current mainline source for the tqm5200s board
   - install it on the board with BDI
   - boot u-boot (currently fail, bugfix is posted on the ML)

   - interesting testcase here:
     https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
     This testcase checks out a git tree, and start a "git bisect"
     with starting a testcase on the board for detecting if current
     version is good or bad. At the end, you get the info, which
     commit breaks ... so detected the fail for the tqm5200s, see:
     http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
     If I find time I want to start this tc automatically in the nightly build
     if a testcase for a board fails. So we get for free every night
     breaking commits tested on real boards ...

- there are also linux test on [3] ...

   - register check (used on the mcx board for pinmux on the ccu1 board
     for pinmux and gpmc register)
     https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lx_check_reg_file.py
     checks the registers defined in a register file with current settings
     (use devmem2 for this task)
     register file example:
     https://github.com/hsdenx/tbot/blob/master/src/files/ccu1_pinmux_scm.reg
     create this file with testcase:
     https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lx_create_reg_file.py

   - https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_sirius_dds.py
     The sirius dds board had problems with ubi/ubifs and power cuts.
     This testcase exeutes 50 power cuts, if no error detected
     it succeeds, else fail.
     The above testcase does:
     - go into state u-boot
     - start linux with ubifs as rootfs
       line 40 (as it is an old testcase, this should be better
       "tb.set_board_state("linux")"
     - wait until Userspace APP SiriusApplicat is started
     - wait random seconds (3 -10)
     - power off the board
     - wait 3 seconds for powering really of the board
     - loop this 50 times
     if we have an error in this steps, testcase ends with error

     For this testcase 56 lines needed (more as the half are comments)

Basic framework is running, time for adding more testcases is needed...

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany