[discussion]: reporting U-Boot test results with tbot

[Heiko Schocher]

Hello Tom, Simon,

motivation for reading:

Get test results from U-Boot users fully automated from git checkout
until reporting testresults to us back with one (or two) commands.

(and one more reason why it makes sense to use tbot)

As I see in some posts from Simon on u-boot and tbot ML, that Simon uses
now tbot (which makes me really happy). Now as also U-Boots test/py is
integrated into tbot [1], it is possible with one tbot command/testcase
to checkout and patch U-Boot source, configure and build it [2], install
resulting binaries on target and call test/py. See a full log [7]

So it makes sense to me to go a step further...

Sometime ago, I had running cyclic u-boot tests with (my old) tbot
and showed the results on a webpage [3]. As you see, not really running
anymore ... :-(

I want to see this reactivated, but slightly modified, so others
can also report testresults ... hopefully with tbot...

What do we need to reactivate this again:

- Try tbot !

   Installation:
   http://tbot.tools/installation.html

   Quick start:
   http://tbot.tools/quickstart.html

   Configuration:
   http://tbot.tools/configuration.html

   Harald rewrote the docs and we hope this is much more clearer
   now, but try it out and report your problems, hints, ideas back
   to us on tbot ml [4]

   Also Simon is working on a tool called labman, with which he
   creates complete tbot config files.

   So there is progress here, but we need users and feedback

<advertisement>
   tbot is designed to help developers for their daily work, if not
   it is a bug and needs a fix. And of course you can use tbot also
   for linux, yocto development ...
</advertisement>

- write and run a server, where we collect testresults

   proof of concept see:

   https://github.com/EmbLux-Kft/uboot_results

   running demo

   http://xeidos.ddns.net/ubtestresults/home

   May you want to click the links ... the link in the column ID
   shows you all infos for one build.

   ID 51 : build with linaro toolchain 2019.12
   ID 50 : build with linaro toolchain 2018.05

   ID 51 saved some bytes in U-Boot image!

   Important topic here is to discuss which informations we want to collect

   https://github.com/EmbLux-Kft/uboot_results#todo

   here a list what I collect currently from an U-Boot build:

   - build date
   - architecture
   - CPU
   - SoC
   - base commit
   - used toolchain
   - board name
   - used defconfig
   - size of resulting images
   - test py results (ToDo)
   - tbot json log file, first step done, see result:
     http://xeidos.ddns.net/ubtestresults/result/53
     and click the links tbot.log or tbot.json

   - ...

   Read the Readme to see whats missing, basic features working,
   how to run and post testresults by hand...

   but we want to automate this ... so, yes, again ... tbot!

- write a tbot generator [5], which collects all the stuff we want to
   push to our server and push it

   proof of concept:

   https://github.com/EmbLux-Kft/tbot/blob/devel/generators/push-testresult.py

   Without any real error checking yet, nor what to report when tbot fails,
   but a first step...


Puuh... now a real life example for the wandboard:

Ok, I need some patches on top of current mainline tbot, see:

https://github.com/EmbLux-Kft/tbot/commits/devel

also my tbot config and testcases:

https://github.com/EmbLux-Kft/tbot-tbot2go/commits/wandboard-devel-messe

(Not all is needed, I should prepare a minimal setup...)

With this, I have 2 commands which do all the steps from U-Boot mainline
checkout until reporting test result to server...

tbot Command:
$ tbot @argswandboardlab1 wandboard_ub_build_install_test -q -q --log $TBOT_LOGFILE > 
$TBOT_STDIO_LOGFILE

Command for executing tbot generator
$ ./push-testresult.py -p /home/hs/data/Entwicklung/tbot -f $TBOT_LOGFILE

full log see [7]

And current mainline U-Boot is tested on the wandboard and result
is reported to my running demo server!

You can now execute the 2 commands from jenkins, buildbot, gitlab CI or
as I did for the demo, in a simple cron job ... and you have a full
CI for U-Boot. It is really worth to try!

The cron job now runs each morning at ~4pm for the wandboard... we will
see how it works ...

That is another reason why it makes sense for me to use tbot, as if
you have finished your daily developer work, you have already testcases
for integrating into a CI ... and if you get a bug report, simply write
a python function, which triggers your bug on the board, so only once
to type the commands by hand... and if bug is fixed ... you already
have a new testcase for your CI ...

There are more things I want to reactivate from my old tbot like

- download patches from patchwork, run checkpatch and add the patches to
   the build.

- automated git bisect if things go wrong, we already have git bisect
   integrated in tbot see:

   http://tbot.tools/modules/tc.html#tbot.tc.git.GitRepository.bisect

- generate a U-Boot documentation with logs from commands
   executed on the board, as our old DULG [6] did.

- make nice graphs from the data we collect
   - spl/u-boot image size for a specific board over the time
   - spl/u-boot image size for a specific board with using
     different toolchains

- ...

But this is just for the future ...

What do you think? Comments/ideas welcome.

by,
Heiko

[1] http://tbot.tools/modules/tc.html#u-boot-test-py
[2] http://tbot.tools/modules/tc.html#u-boot
[3] old tbot test results webpage
     http://xeidos.ddns.net/tests/test_db_auslesen.php
[4] tbot at lists.denx.de
[5] http://tbot.tools/logging.html#generators
[6] http://www.denx.de/wiki/DULG/WebHome
[7] https://github.com/EmbLux-Kft/uboot_results/blob/master/LOG.md
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de