[U-Boot] U-Boot: using tbot for U-Boot tests was: [PATCH V2 1/7] test/py: Implement pytest infrastructure

Tue Jan 19 04:42:06 CET 2016

Hi Heiko,

On 15 January 2016 at 23:29, Heiko Schocher <hs at denx.de> wrote:
> Hello Simon,
>
> Am 15.01.2016 um 00:12 schrieb Simon Glass:
>>
>> Hi Heiko,
>>
>> On 16 December 2015 at 22:45, Heiko Schocher <hs at denx.de> wrote:
>>>
>>> Hello Stephen,
>>>
>>>
>>> Am 16.12.2015 um 17:27 schrieb Stephen Warren:
>>>>
>>>>
>>>> On 12/16/2015 08:11 AM, Michal Simek wrote:
>>>>>
>>>>>
>>>>> On 9.12.2015 17:32, Stephen Warren wrote:
>>>>>>
>>>>>>
>>>>>> On 12/02/2015 03:18 PM, Stephen Warren wrote:
>
> [...]
>>>>
>>>> In my current setup I leave the board on all the time (or rather,
>>>> manually
>>>> turn on the power when
>>>> I'm about to run the tests). Automating control of the power source is a
>>>> step I'll take later.
>>>
>>>
>>>
>>> Maybe you give tbot (I mentioned it in this thread) a chance?
>>>
>>> There, this things are automated, also you can do linux (and other
>>> console
>>> based)
>>> tests ... Currently added a testcase [1] which adds patches from
>>> patchwork,
>>> which are in a users ToDo list, to a git tree! In this case it is a
>>> u-boot
>>> git
>>> tree ... checks them with checkpatch, compiles it, and tries the new
>>> image
>>> on
>>> the board and calls testcases ... fully automated in a now weekly build
>>> [2]
>>> ...
>>> (only weekly, but thats a setup parameter in buildbot, as I have tbot and
>>> buildbot
>>> running on a raspberry pi)
>>> and don;t forget, I have the board not where I run tbot, the boards are
>>> ~1000km
>>> from me .. So, it is possible to add a U-Boot Testsetup on a server,
>>> and test boards all over the world ...
>>
>>
>> This sounds like a great development.
>
>
> Thanks!
>
>> How can we get this so that it can be used by U-Boot people? Do you
>> think you could add a README to the mainline, or some scripts to aid
>> setting it up? I would be interested in setting up a few boards that
>> run continuous testing, and I suspect others would also if it were
>> easier.
>
>
> Yes, good idea!
>
> I think about preparing ASAP a patch for U-Boot, creating:
> u-boot:/tools/tbot/README -> common infos about tbot
> u-boot:/tools/tbot/README.install -> HowTo install / using it
> u-boot:/tools/tbot/README.create_a_testcase
> u-boot:/tools/tbot/README-ToDo -> my current list of ToDo
>
> Is this Ok?

Sounds great.

>
> As a motivation for using tbot ;-)  I just created a video from tbot
> running the smartweb testcase:
> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_smartweb.py
>
> https://www.youtube.com/watch?v=ZwUA0QNDnP4

OK that helps explain it. It looks like it uses DFU to write new
images to the boards - is that right?

>
> Keep in mind, that I run tbot for the video here on my laptop at my home
> in hungary, the smartweb board is in munich/germany.
>
> I use this testcase also in my weekly buildbot setup on my raspberry pi [2].
> Tbot logs (not only U-Boot also linux tests) can be found there for
> interested people
> (Remark: Wolfgang said, the logs are unreadable, because filled with a
> lot of unnecessary developer output, Yes, he is correct!! I have on my
> ToDo list, to add a new logging level, where only board input/output is
> printed ... the loglevel tbot uses is definded in the board config file
> https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L13 )

Yes that's my main comment.

>
> What is done in the smartweb testcase:
> - rm old u-boot code, checkout current U-Boot master
>   @ 00:14
> - set a toolchain
>   @ 00:18
> - get all patchwork patches from my patchwork ToDo list
>   calling testcase:
>
> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_get_patchwork_number_list.py
>   @ 00:18
> - adding some patchwork patches, I have in a python list
>   (This list is setup in the board config file:
>    https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L34
>   (Heiko speculating: it would be nice if tbot removes from
>    this list a patchworkpatchnumber, if it detects, that this patch
>    is already now in mainline ... )
> - apply local patches (If there are)
> - apply the patchwork patches
>
> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_apply_patchwork_patches.py
>   currently not failing, when checkpatch detects errors/warnings
>   @ 00:59
> - compile U-Boot
>   @ 02:35
> - update SPL on the board
>   @ 03:26
> - update U-Boot on the board
>   @ 03:55
> - start DFU testcase on the board
>   @ 04:13
>   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_ub_dfu.py
>
>   This testcase starts the "dfu" U-Boot command, which waits until Ctrl-C
>   is pressed
>   Then I start on the lab PC the Userspace tool "dfu-util" which
> communicates
>   over USB with the smartweb board ... and do some dfu up and downloads.
>
> - at the end save the SPL and U-Boot bins, so I always have the lastest
>   working bins [1]
>
> - power off the board
>
> bye,
> Heiko

Well I'll await your patches to the list and then see if I can get it
working. What sort of hardware do I need for the power / reset
control?

>
> [1] saving the latest working bins is interesting, because if a current
>   U-Boot does not work on he board, I :
>   - can restore the board with a debugger through a testcase, using the
>     latest working bins.
>   - and/or I can start a testcase which starts "git bisect" testcase:
>     https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
>
>     This testcase needs 3 variables:
>     tb.board_git_bisect_get_source_tc: Name of testcase which get/switches
>       into the source tree you want to start a git bisect session
>     tb.board_git_bisect_call_tc: Name of testcase which gets started,
>       when "git bisect" waits for good or bad ... This testcase must
>       find out if current source is good or bad.
>     tb.board_git_bisect_good_commit: last working bins (Therefore I save
>       the bins -> so I have the commit ;-)
>
>     This testcase is independent from U-Boot ... you can also use it
>     in a linux tree or other source code ...
>
>   I used this testcase for example here:
>
> http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
>   at the end it calls "git bisect log" (search for this string), and
>   it found the first bad commit, and I did nothing else as starting
>   this testcase :-D
>
> [2]http://xeidos.ddns.net/buildbot/tgrid
>    http://xeidos.ddns.net/buildbot/builders/smartweb_dfu
>    (If you do not see a webpage, reload it ... my DSL upload speed
>     is not the fastest, also if my kids play ps4 games, it is busy)
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Regards,
Simon