[u-boot-test-hooks PATCH v4 3/3] Provide some basic scripts for Labgrid integration

[neil.armstrong at linaro.org]

On 29/08/2024 14:17, Simon Glass wrote:
> Hi Peter,
> 
> On Thu, 29 Aug 2024 at 04:43, Peter Robinson <pbrobinson at gmail.com> wrote:
>>
>> On Wed, 28 Aug 2024 at 22:25, Simon Glass <sjg at chromium.org> wrote:
>>>
>>> Hi Peter,
>>>
>>> On Wed, 28 Aug 2024 at 12:14, Peter Robinson <pbrobinson at gmail.com> wrote:
>>>>
>>>> Hi Simon,
>>>>
>>>>> With Labgrid we don't need to specify the various methods, except for
>>>>> the console, which simply calls labgrid-client.
>>>>>
>>>>> This allows supporting any boards in your lab, without adding per-board
>>>>> configuration to these hooks.
>>>>>
>>>>> Provide ellesmere files as an example.
>>>>
>>>> What's ellesmere?
>>>
>>> It is a lake but also the name of a computer.
>>>
>>>>
>>>>> Signed-off-by: Simon Glass <sjg at chromium.org>
>>>>> ---
>>>>>
>>>>> Changes in v4:
>>>>> - Support pytest fully with dual-build boards like Beagleplay
>>>>>
>>>>> Changes in v3:
>>>>> - Update scripts for latest version of Labgrid integration
>>>>> - Add poweroff.none and poweron.none
>>>>> - Provide -n flag when querying board info
>>>>> - Target the grpc version of Labgrid which is now in -master
>>>>> - Update README to cover the changes
>>>>>
>>>>> Changes in v2:
>>>>> - Make use of the common script (only) to set bin_dir
>>>>>
>>>>>   README.md                    | 50 ++++++++++++++++++++++++++++++++++++
>>>>
>>>> Maybe that should be in a separate labsgrid readme?
>>>
>>> My hope is that Labgrid becomes the normal way of running these tests,
>>
>> Generally I agree with automated testing platforms, I think <insert
>> you're preferred platform here> makes sense, there's a bunch like
>> Linaro and a bunch in the arm ecosystem use LAVA [1] and  there's a
>> bunch more that I'm aware of.
> 
> I am somewhat familiar with LAVA and I believe it can be used to test
> U-Boot, although I need to learn how. Looking at a test run [2] for
> beaglebone black I see that it is using a recent kernel but the U-Boot
> seems to be older.
> 
>>
>> Does it make sense, of course at some point in the future post this
>> being merged, make sense to look at a general way of making it easier
>> to plugin these sort of HW test platforms using this as a basis? I ask
>> mostly because putting a bunch of my devices into some sort of
>> platform can auto test things and of course everyone has an opinion to
>> which is the best one :-P
> 
> Yes. I had heard from Tom that Labgrid is the new hotness for now.
> Having dug into it I believe it is a good solution, although it can
> certainly be improved to handle scale better.
> 
> Anyway, IMO the current test hooks are not a great solution, just
> because the configuration is spread all over the place and it relies
> on lots of little shell scripts. So I believe that the Labgrid
> integration is a closer to where we want to be with others that come
> along.

I'd say all those scripts are actually here to ease integration with any
system, booting U-Boot and Linux are two different beasts.
Most of the time you need special jumpers enabled to setup vendor-specific
bootrom mode to reflash or ram-boot U-Boot.
So forcing us people to switch to Labgrid because it matches your boards
behavior is a little adventurous, and while we can add some enhancement
to the actual pytest and hooks, I'm against saying Labgrid should be the
golden CI system.

Neil

> 
> I would love for you to add a board into Labgrid and see how you go.
> 
>>
>> [1] https://validation.linaro.org/
>>
>>> so having it in the main place (and perhaps eventually removing the
>>> old way) is my goal.
>>>
>>>>
>>>>>   bin/console.labgrid          | 42 ++++++++++++++++++++++++++++++
>>>>>   bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++
>>>>>   bin/ellesmere/conf.all       | 24 +++++++++++++++++
>>>>>   bin/getrole.labgrid          | 25 ++++++++++++++++++
>>>>>   bin/release.labgrid          | 22 ++++++++++++++++
>>>>>   bin/release.none             | 22 ++++++++++++++++
>>>>>   bin/u-boot-test-getrole      | 38 +++++++++++++++++++++++++++
>>>>>   bin/u-boot-test-release      | 26 +++++++++++++++++++
>>>>>   9 files changed, 295 insertions(+)
>>>>>   create mode 100644 bin/console.labgrid
>>>>>   create mode 100755 bin/ellesmere/common-labgrid
>>>>>   create mode 100644 bin/ellesmere/conf.all
>>>>>   create mode 100644 bin/getrole.labgrid
>>>>>   create mode 100644 bin/release.labgrid
>>>>>   create mode 100644 bin/release.none
>>>>>   create mode 100755 bin/u-boot-test-getrole
>>>>>   create mode 100755 bin/u-boot-test-release
>>>>>
>>>
>>> [..]
>>>
>>> Regards,
>>> Simon
> 
> Regards,
> Simon
> 
> [2] https://validation.linaro.org/scheduler/job/4089871#results_482796468