[U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Stephen Warren]

On 10/22/2016 06:56 AM, Tom Rini wrote:
> On Fri, Oct 21, 2016 at 12:24:07PM -0600, Stephen Warren wrote:
>> On 10/20/2016 01:40 PM, Tom Rini wrote:
>>> - Add a PPA for a more recent qemu (required for PowerPC to work)
>>> - Add tests to run test.py for various QEMU platforms.  This relies on
>>>  swarren's uboot-test-hooks repository to provide the abstractions.

>>> + # Clone uboot-test-hooks
>>> + - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks
>>
>> Does it make sense to rely on my copy of the repo being around all
>> the time? I have no intention of removing it, but when I created it
>> I'd assumed others would create their own clones for their own
>> infra-structure. I wonder if all the dependencies for U-Boot's
>> testing should be hosted on U-Boot infra-structure?
>
> A good point.  The way I've been thinking of it is that we also need a
> repo with as many and varied configs as possible, which has become yours
> :).  That said, we could move a copy of this over to the u-boot account
> on github if you'd like.

I think it's fine for now.

>> That all said, this should be fine; just something to ponder.
>>
>>> + - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
>>
>> It may be useful to add the following too:
>>
>> - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
>>
>> That way, if we ever need per-board test configuration files, the
>> path to access them will already be available.
>
> A good point.  I think we could get the network tests going, maybe with
> a little re-work to dynamically hash the file from the host side?  But
> that's an aside.

Yes, assuming qemu networking works, I expect that would be possible. 
Yes, I think it'd be pretty easy to dynamically create a test download 
file and calculate the CRC from the per-board Python config files.

>>> +    - env:
>>> +        - CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-"
>>> +      script:
>>> +        - ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build;
>>> +          ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build;
>>> +          ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build;
>>
>> Does it make sense to make that 3 separate scripts? I assume that
>> would (a) allow the individual status of the 3 test.py invocations
>> to be seen separately (b) might allow/cause travis-ci to run them in
>> parallel; I'm not sure if this Travis config file is a sequential
>> list of commands, or a list of potentially parallel actions.
>
> So, the way travis-ci works is that you can (and we do) split tasks up
> to run in parallel.  The hunk above is a series to run in sequence on
> the same host. The downside to moving this to parallel is the amount of
> duplication you need in each stanza.

The duplication doesn't seem like a terribly big deal given it will 
allow finer-grained status on the main per-build page.

 > We could fix that (at least in the
> case of non-buildman-fetched toolchains) if test.py used buildman to
> build, as that will determine CROSS_COMPILE correctly from the default
> .buildman file we create earlier on.  Thanks!

I'll look into whether that's possible.

BTW, does/can Travis save the test-log.html from the test.py 
invocations? That would make debugging any failures a little easier.