[discussion]: reporting U-Boot test results with tbot
hs at denx.de
Thu Apr 30 16:58:20 CEST 2020
Hello Tom, Simon,
motivation for reading:
Get test results from U-Boot users fully automated from git checkout
until reporting testresults to us back with one (or two) commands.
(and one more reason why it makes sense to use tbot)
As I see in some posts from Simon on u-boot and tbot ML, that Simon uses
now tbot (which makes me really happy). Now as also U-Boots test/py is
integrated into tbot , it is possible with one tbot command/testcase
to checkout and patch U-Boot source, configure and build it , install
resulting binaries on target and call test/py. See a full log 
So it makes sense to me to go a step further...
Sometime ago, I had running cyclic u-boot tests with (my old) tbot
and showed the results on a webpage . As you see, not really running
anymore ... :-(
I want to see this reactivated, but slightly modified, so others
can also report testresults ... hopefully with tbot...
What do we need to reactivate this again:
- Try tbot !
Harald rewrote the docs and we hope this is much more clearer
now, but try it out and report your problems, hints, ideas back
to us on tbot ml 
Also Simon is working on a tool called labman, with which he
creates complete tbot config files.
So there is progress here, but we need users and feedback
tbot is designed to help developers for their daily work, if not
it is a bug and needs a fix. And of course you can use tbot also
for linux, yocto development ...
- write and run a server, where we collect testresults
proof of concept see:
May you want to click the links ... the link in the column ID
shows you all infos for one build.
ID 51 : build with linaro toolchain 2019.12
ID 50 : build with linaro toolchain 2018.05
ID 51 saved some bytes in U-Boot image!
Important topic here is to discuss which informations we want to collect
here a list what I collect currently from an U-Boot build:
- build date
- base commit
- used toolchain
- board name
- used defconfig
- size of resulting images
- test py results (ToDo)
- tbot json log file, first step done, see result:
and click the links tbot.log or tbot.json
Read the Readme to see whats missing, basic features working,
how to run and post testresults by hand...
but we want to automate this ... so, yes, again ... tbot!
- write a tbot generator , which collects all the stuff we want to
push to our server and push it
proof of concept:
Without any real error checking yet, nor what to report when tbot fails,
but a first step...
Puuh... now a real life example for the wandboard:
Ok, I need some patches on top of current mainline tbot, see:
also my tbot config and testcases:
(Not all is needed, I should prepare a minimal setup...)
With this, I have 2 commands which do all the steps from U-Boot mainline
checkout until reporting test result to server...
$ tbot @argswandboardlab1 wandboard_ub_build_install_test -q -q --log $TBOT_LOGFILE >
Command for executing tbot generator
$ ./push-testresult.py -p /home/hs/data/Entwicklung/tbot -f $TBOT_LOGFILE
full log see 
And current mainline U-Boot is tested on the wandboard and result
is reported to my running demo server!
You can now execute the 2 commands from jenkins, buildbot, gitlab CI or
as I did for the demo, in a simple cron job ... and you have a full
CI for U-Boot. It is really worth to try!
The cron job now runs each morning at ~4pm for the wandboard... we will
see how it works ...
That is another reason why it makes sense for me to use tbot, as if
you have finished your daily developer work, you have already testcases
for integrating into a CI ... and if you get a bug report, simply write
a python function, which triggers your bug on the board, so only once
to type the commands by hand... and if bug is fixed ... you already
have a new testcase for your CI ...
There are more things I want to reactivate from my old tbot like
- download patches from patchwork, run checkpatch and add the patches to
- automated git bisect if things go wrong, we already have git bisect
integrated in tbot see:
- generate a U-Boot documentation with logs from commands
executed on the board, as our old DULG  did.
- make nice graphs from the data we collect
- spl/u-boot image size for a specific board over the time
- spl/u-boot image size for a specific board with using
But this is just for the future ...
What do you think? Comments/ideas welcome.
 old tbot test results webpage
 tbot at lists.denx.de
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52 Fax: +49-8142-66989-80 Email: hs at denx.de
More information about the U-Boot-Board-Maintainers