[U-Boot] [RFC] Continuos integration with DUTS v2

[Jerry Van Baren]

Hi Niklaus,

Niklaus Giger wrote:
> Hi
> 
> As I consider testing as an important part to ensure high code quality for
> any product. It should form part of the global development process.
> 
> 1) When adding a new board or feature to U-Boot running tests to ensure that 
> it works as advertised should be mandatory but not time consuming.

Yes.

> 2) If one has a board up and running, I consider it adviseable to ensure that
> it always compiles the actual state of the U-Boot code and passes all the 
> tests.

Yes!

> 3) It would be nice to have somewhere a central repository where everybody
> (specially the maintainers) could see whether the different boards pass all
> the tests or not.

Yes.  Something to consider: Rockbox has a distributed build/report 
framework that we might be able to use or at least learn from (Daniel 
Stenberg is a u-boot user/contributor).

<http://www.rockbox.org/wiki/BuildClient>
-8<-------------------------------------------------------------------->8-
Why not distcc

distcc is a great and fun project, but unsuitable for use by us. Since 
distcc still preprocesses each file locally, hand over the file to the 
remote servers to get build and then gets the object file back, it 
requires slow compiles and fast networks to be really efficient.

In contrast we have relatively fast compiles and a slow network, to the 
extent that transferring the file over and getting an object file back 
is slower than simply compiling it locally. This is of course also due 
to the somewhat slow network between our build servers.
-8<--------------------------------------------------------------------->8-

> To achieve the points 1) and 2) I am working on a solution which should 
> achieve
> these goals if you meet the following requirements
> - you have a spare board to run the tests.
> - you have some HW to switch on/off the power (There are solutions that
>   switch 4 or 8 outlet for 100 or 200 Euros).

Jon Diekema has the NP-0201D and likes it (the temperature probe isn't 
useful).  Looks like the HP-02H (220vAC) would be the European version.

<http://www.synaccess-net.com/remote_power_products.php/1/8> - $200 
NP-02 (simpler)

<http://www.synaccess-net.com/remote_power_products.php/1/36> - $245 
NP-0201D (adds current, etc)

> - you have system running a GNU/Linux. (My system runs Debian Lenny) which has
>   some background processing power.
> - you have (at least weekly) some time to look after the results and report/fix
>   the problems.
> - have 1 to 4 hours to setup a testing environment.
> 
> I would be interested what you think about the points raised above and whether
> following the instructions given under
> http://ngiger.dyndns.org/duts_v2/doc/index.html and
> http://ngiger.dyndns.org/hudson.doc/doc/index.html
> allows you to achieve above mentioned goals 1) and 2).
> 
> 
> Background:
> ----------
> 
> In 2004 - 2006 I run for a while a buildbot for the Xenomai project.
> 
> In the last two months I used my pre-existing Ruby scripts and the
> DENX Universal Test Suite (http://www.denx.de/wiki/DUTS/DUTSDocs) to
> rewrite into DUTS-v2 (http://ngiger.dyndns.org/duts_v2/doc/index.html).
> Quite some effort did go into writing unit tests for test suite.
> Easy adaption to various environments was a important design factor.

Ruby - yes!  I didn't realize how much you've already done.
TCL - meh, I have used it but never warmed up to it.

> My previous experiences with the continuos integrations tools
> http://buildbot.net/trac and http://cruisecontrol.sourceforge.net/
> made me have a look at https://hudson.dev.java.net/.
> 
> Hudson proved to be really simple to setup and administer. My experiences
> are documented at http://ngiger.dyndns.org/hudson.doc/doc/index.html.

I've heard good things about hudson, have not done anything in it or 
buildbot.  I've looked at cruisecontrol XML files (minor hacking) - 
writing programs (build scripts) in a markup language is evil (hudson 
apparently fixes this by hiding the XML).

> Future extensions:
> ------------------
> 
> a) Use the same infrastructure not only for U-Boot but also for related 
> projects like xenomai, RT-preempt, ltp.
> 
> b) To achieve goal 3) I could not find any project around the linux kernel 
> which publishes in an automated way test results.

See above <http://www.rockbox.org/wiki/BuildClient> (perl :-/ since I'm 
being a language snob tonight).

> If I don't receive other suggestions I plan the following steps:
> - Create a XML file with test test results containings
>   - version of the test suite/scripts
>   - information about the target like
>     - HW configuration (CPU, RAM, devices)
>     - SW version/configuration
>   - name and result of each teststep
>   - performance information (e.g. min/max/avg, histogram) for parameters like
>     latency, throughput
>   - ???
> - Send XML file at end of test-run to a special mailing list
> - Process mailing lists to feed all the XML into a database
> - Web-Frontend to this database using ??? (Ruby On Rails comes to my mind)
> 
> Any comments to these points?

I've been wanting to do this for my own use (regression testing for the 
FDT command handling, board support regression testing) but have not 
made time to do it.  :-(  I appreciate you blazing a path.

> Next steps:
> -----------
> 
> - Listen to your suggestions
> - Fix bugs reported
> - Move code from my SVN-repository to git.denx.de ?
> - Open wiki entries for points 1 - 3 ?

Yes to all four points.

> Best regards
> 
> Niklaus

Thanks!
gvb