[discussion]: reporting U-Boot test results with tbot

Fri May 1 07:21:27 CEST 2020

Hello Tom,

just playing to get a first graph out of my DB...:

https://github.com/EmbLux-Kft/uboot_results/commit/6dc79a119e9b8937645284f30b37dd4cf9a95d48

result:

http://xeidos.ddns.net/ubtestresults/stats/wandboard_defconfig/6
                                                                ^
                                            ^       get the last n=6 entries
                                            |
                                            name of defconfig

and you see commits from last night are increasing code size
of u-boot image!

values from last night:
http://xeidos.ddns.net/ubtestresults/result/54

the night before:
http://xeidos.ddns.net/ubtestresults/result/53

Same toolchain as before... so I won some bytes from value
2 -> 3 as I used newer linaro toolchain, and lost this last night...

This is not a problem for the wandboard, but it shows why it makes
sense to collect testresult ;-)

I currently have only one defconfig running, so only wandboard_defconfig
works, but you can play with n ... but the first ~40 entries in
my DB are not real values, they are hand edited and I  used them for
testing...

bye,
Heiko

Am 30.04.2020 um 16:58 schrieb Heiko Schocher:
> Hello Tom, Simon,
> 
> motivation for reading:
> 
> Get test results from U-Boot users fully automated from git checkout
> until reporting testresults to us back with one (or two) commands.
> 
> (and one more reason why it makes sense to use tbot)
> 
> As I see in some posts from Simon on u-boot and tbot ML, that Simon uses
> now tbot (which makes me really happy). Now as also U-Boots test/py is
> integrated into tbot [1], it is possible with one tbot command/testcase
> to checkout and patch U-Boot source, configure and build it [2], install
> resulting binaries on target and call test/py. See a full log [7]
> 
> So it makes sense to me to go a step further...
> 
> Sometime ago, I had running cyclic u-boot tests with (my old) tbot
> and showed the results on a webpage [3]. As you see, not really running
> anymore ... :-(
> 
> I want to see this reactivated, but slightly modified, so others
> can also report testresults ... hopefully with tbot...
> 
> What do we need to reactivate this again:
> 
> - Try tbot !
> 
>    Installation:
>    http://tbot.tools/installation.html
> 
>    Quick start:
>    http://tbot.tools/quickstart.html
> 
>    Configuration:
>    http://tbot.tools/configuration.html
> 
>    Harald rewrote the docs and we hope this is much more clearer
>    now, but try it out and report your problems, hints, ideas back
>    to us on tbot ml [4]
> 
>    Also Simon is working on a tool called labman, with which he
>    creates complete tbot config files.
> 
>    So there is progress here, but we need users and feedback
> 
> <advertisement>
>    tbot is designed to help developers for their daily work, if not
>    it is a bug and needs a fix. And of course you can use tbot also
>    for linux, yocto development ...
> </advertisement>
> 
> - write and run a server, where we collect testresults
> 
>    proof of concept see:
> 
>    https://github.com/EmbLux-Kft/uboot_results
> 
>    running demo
> 
>    http://xeidos.ddns.net/ubtestresults/home
> 
>    May you want to click the links ... the link in the column ID
>    shows you all infos for one build.
> 
>    ID 51 : build with linaro toolchain 2019.12
>    ID 50 : build with linaro toolchain 2018.05
> 
>    ID 51 saved some bytes in U-Boot image!
> 
>    Important topic here is to discuss which informations we want to collect
> 
>    https://github.com/EmbLux-Kft/uboot_results#todo
> 
>    here a list what I collect currently from an U-Boot build:
> 
>    - build date
>    - architecture
>    - CPU
>    - SoC
>    - base commit
>    - used toolchain
>    - board name
>    - used defconfig
>    - size of resulting images
>    - test py results (ToDo)
>    - tbot json log file, first step done, see result:
>      http://xeidos.ddns.net/ubtestresults/result/53
>      and click the links tbot.log or tbot.json
> 
>    - ...
> 
>    Read the Readme to see whats missing, basic features working,
>    how to run and post testresults by hand...
> 
>    but we want to automate this ... so, yes, again ... tbot!
> 
> - write a tbot generator [5], which collects all the stuff we want to
>    push to our server and push it
> 
>    proof of concept:
> 
>    https://github.com/EmbLux-Kft/tbot/blob/devel/generators/push-testresult.py
> 
>    Without any real error checking yet, nor what to report when tbot fails,
>    but a first step...
> 
> 
> Puuh... now a real life example for the wandboard:
> 
> Ok, I need some patches on top of current mainline tbot, see:
> 
> https://github.com/EmbLux-Kft/tbot/commits/devel
> 
> also my tbot config and testcases:
> 
> https://github.com/EmbLux-Kft/tbot-tbot2go/commits/wandboard-devel-messe
> 
> (Not all is needed, I should prepare a minimal setup...)
> 
> With this, I have 2 commands which do all the steps from U-Boot mainline
> checkout until reporting test result to server...
> 
> tbot Command:
> $ tbot @argswandboardlab1 wandboard_ub_build_install_test -q -q --log $TBOT_LOGFILE > 
> $TBOT_STDIO_LOGFILE
> 
> Command for executing tbot generator
> $ ./push-testresult.py -p /home/hs/data/Entwicklung/tbot -f $TBOT_LOGFILE
> 
> full log see [7]
> 
> And current mainline U-Boot is tested on the wandboard and result
> is reported to my running demo server!
> 
> You can now execute the 2 commands from jenkins, buildbot, gitlab CI or
> as I did for the demo, in a simple cron job ... and you have a full
> CI for U-Boot. It is really worth to try!
> 
> The cron job now runs each morning at ~4pm for the wandboard... we will
> see how it works ...
> 
> That is another reason why it makes sense for me to use tbot, as if
> you have finished your daily developer work, you have already testcases
> for integrating into a CI ... and if you get a bug report, simply write
> a python function, which triggers your bug on the board, so only once
> to type the commands by hand... and if bug is fixed ... you already
> have a new testcase for your CI ...
> 
> There are more things I want to reactivate from my old tbot like
> 
> - download patches from patchwork, run checkpatch and add the patches to
>    the build.
> 
> - automated git bisect if things go wrong, we already have git bisect
>    integrated in tbot see:
> 
>    http://tbot.tools/modules/tc.html#tbot.tc.git.GitRepository.bisect
> 
> - generate a U-Boot documentation with logs from commands
>    executed on the board, as our old DULG [6] did.
> 
> - make nice graphs from the data we collect
>    - spl/u-boot image size for a specific board over the time
>    - spl/u-boot image size for a specific board with using
>      different toolchains
> 
> - ...
> 
> But this is just for the future ...
> 
> What do you think? Comments/ideas welcome.
> 
> by,
> Heiko
> 
> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
> [2] http://tbot.tools/modules/tc.html#u-boot
> [3] old tbot test results webpage
>      http://xeidos.ddns.net/tests/test_db_auslesen.php
> [4] tbot at lists.denx.de
> [5] http://tbot.tools/logging.html#generators
> [6] http://www.denx.de/wiki/DULG/WebHome
> [7] https://github.com/EmbLux-Kft/uboot_results/blob/master/LOG.md

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de