[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Heiko Schocher]

Hello Stephen,

Am 16.12.2015 um 17:27 schrieb Stephen Warren:
> On 12/16/2015 08:11 AM, Michal Simek wrote:
>> On 9.12.2015 17:32, Stephen Warren wrote:
>>> On 12/02/2015 03:18 PM, Stephen Warren wrote:
>>>> This tool aims to test U-Boot by executing U-Boot shell commands using
>>>> the
>>>> console interface. A single top-level script exists to execute or attach
>>>> to the U-Boot console, run the entire script of tests against it, and
>>>> summarize the results. Advantages of this approach are:
>>>>
>>>> - Testing is performed in the same way a user or script would interact
>>>>     with U-Boot; there can be no disconnect.
>>>> - There is no need to write or embed test-related code into U-Boot
>>>> itself.
>>>>     It is asserted that writing test-related code in Python is simpler and
>>>>     more flexible that writing it all in C.
>>>> - It is reasonably simple to interact with U-Boot in this way.
>>>>
>>>> A few simple tests are provided as examples. Soon, we should convert as
>>>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>>>
>>>> In the future, I hope to publish (out-of-tree) the hook scripts, relay
>>>> control utilities, and udev rules I will use for my own HW setup.
>>>
>>> I finally got permission to publish these. Examples are at:
>>> https://github.com/swarren/uboot-test-hooks
>>
>> Interesting. What's the normal setup which you have for the board?
>> I see from your description that you use numato usb relay - I expect one
>> with more channels for reset.
>> Then you are able to control boot mode. Is it also via the same relay?
>> How do you power up the board?
>
> In my current setup I leave the board on all the time (or rather, manually turn on the power when
> I'm about to run the tests). Automating control of the power source is a step I'll take later.

Maybe you give tbot (I mentioned it in this thread) a chance?

There, this things are automated, also you can do linux (and other console based)
tests ... Currently added a testcase [1] which adds patches from patchwork,
which are in a users ToDo list, to a git tree! In this case it is a u-boot git
tree ... checks them with checkpatch, compiles it, and tries the new image on
the board and calls testcases ... fully automated in a now weekly build [2] ...
(only weekly, but thats a setup parameter in buildbot, as I have tbot and buildbot
running on a raspberry pi)
and don;t forget, I have the board not where I run tbot, the boards are ~1000km
from me .. So, it is possible to add a U-Boot Testsetup on a server,
and test boards all over the world ...

> For Tegra, there are two important signals: reset and "force recovery". Each of these has a separate
> relay, so the system currently uses 2 relays per target board. The numato relay board I own has 8
> relays, although there are a number of different models.
>
> On Tegra, when reset is pulsed:
>
> - If force-recovery is connected, the SoC enters USB recovery mode. In this state, SW can be
> downloaded over USB into RAM and executed.
>
> - If force-recovery is not connected, the SoC boots normally, from SW stored in flash (eMMC, SPI, ...)
>
> The example scripts always use recovery mode to download U-Boot into RAM rather than writing it to
> flash first and then resetting. This saves wear cycles on the flash, but does mean the download
> happens in the "reset" rather than "flash" script, which may make the examples a bit different than
> for some other SoCs.
>
> Finally, the example scripts support two boards; my home/laptop dev setup that uses a Numato relay
> board to control the signals to the board I use there, and my work desktop dev setup that uses our
> "PM342" debug board to controll the signals. The latter works logically the same as the numato relay
> board, except it contains electronic switches driven by an FTDI chip.

I seperated such functions into a seperate python class, so such setup
specific things should be easy to add for other (in tbot called lab)
setups ...

Currently I have only a bdi testcase, which flashes a new image into
the board, when it is broken ... but such relay things can be added
of course ...

bye,
Heiko
[1] get patchlist from patchwork
     https://github.com/hsdenx/tbot/commit/0bcaf4877e7aad4df2039913dcb6e85303a0b15f
     apply them to git tree:
     https://github.com/hsdenx/tbot/commit/b7e2de3731252b518754cc3f71dc782559b0cad6
     and use this on the smartweb board:
     https://github.com/hsdenx/tbot/commit/56a1ac18e5730ae9ffa7686acfc52877272be91d

[2] weekly started testcase for the smartweb board:
     http://xeidos.ddns.net/buildbot/builders/smartweb_dfu
     log with the testcases from [1]
     http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/32/steps/shell/logs/tbotlog

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany