[u-boot-test-hooks PATCH v4 3/3] Provide some basic scripts for Labgrid integration

[Simon Glass]

Hi Neil,

On Thu, 29 Aug 2024 at 08:44, <neil.armstrong at linaro.org> wrote:
>
> On 29/08/2024 14:17, Simon Glass wrote:
> > Hi Peter,
> >
> > On Thu, 29 Aug 2024 at 04:43, Peter Robinson <pbrobinson at gmail.com> wrote:
> >>
> >> On Wed, 28 Aug 2024 at 22:25, Simon Glass <sjg at chromium.org> wrote:
> >>>
> >>> Hi Peter,
> >>>
> >>> On Wed, 28 Aug 2024 at 12:14, Peter Robinson <pbrobinson at gmail.com> wrote:
> >>>>
> >>>> Hi Simon,
> >>>>
> >>>>> With Labgrid we don't need to specify the various methods, except for
> >>>>> the console, which simply calls labgrid-client.
> >>>>>
> >>>>> This allows supporting any boards in your lab, without adding per-board
> >>>>> configuration to these hooks.
> >>>>>
> >>>>> Provide ellesmere files as an example.
> >>>>
> >>>> What's ellesmere?
> >>>
> >>> It is a lake but also the name of a computer.
> >>>
> >>>>
> >>>>> Signed-off-by: Simon Glass <sjg at chromium.org>
> >>>>> ---
> >>>>>
> >>>>> Changes in v4:
> >>>>> - Support pytest fully with dual-build boards like Beagleplay
> >>>>>
> >>>>> Changes in v3:
> >>>>> - Update scripts for latest version of Labgrid integration
> >>>>> - Add poweroff.none and poweron.none
> >>>>> - Provide -n flag when querying board info
> >>>>> - Target the grpc version of Labgrid which is now in -master
> >>>>> - Update README to cover the changes
> >>>>>
> >>>>> Changes in v2:
> >>>>> - Make use of the common script (only) to set bin_dir
> >>>>>
> >>>>>   README.md                    | 50 ++++++++++++++++++++++++++++++++++++
> >>>>
> >>>> Maybe that should be in a separate labsgrid readme?
> >>>
> >>> My hope is that Labgrid becomes the normal way of running these tests,
> >>
> >> Generally I agree with automated testing platforms, I think <insert
> >> you're preferred platform here> makes sense, there's a bunch like
> >> Linaro and a bunch in the arm ecosystem use LAVA [1] and  there's a
> >> bunch more that I'm aware of.
> >
> > I am somewhat familiar with LAVA and I believe it can be used to test
> > U-Boot, although I need to learn how. Looking at a test run [2] for
> > beaglebone black I see that it is using a recent kernel but the U-Boot
> > seems to be older.
> >
> >>
> >> Does it make sense, of course at some point in the future post this
> >> being merged, make sense to look at a general way of making it easier
> >> to plugin these sort of HW test platforms using this as a basis? I ask
> >> mostly because putting a bunch of my devices into some sort of
> >> platform can auto test things and of course everyone has an opinion to
> >> which is the best one :-P
> >
> > Yes. I had heard from Tom that Labgrid is the new hotness for now.
> > Having dug into it I believe it is a good solution, although it can
> > certainly be improved to handle scale better.
> >
> > Anyway, IMO the current test hooks are not a great solution, just
> > because the configuration is spread all over the place and it relies
> > on lots of little shell scripts. So I believe that the Labgrid
> > integration is a closer to where we want to be with others that come
> > along.
>
> I'd say all those scripts are actually here to ease integration with any
> system, booting U-Boot and Linux are two different beasts.

That's fine, go ahead and use the scripts. My point is that Labgrid
doesn't need them and in fact it makes everything pretty painful if we
try to use all of them.

> Most of the time you need special jumpers enabled to setup vendor-specific
> bootrom mode to reflash or ram-boot U-Boot.

I deal with this by soldering onto the recovery switch. For example
Beagleplay needs it, otherwise it boots from internal eMMC or
something.

Perhaps Linaro could create a standard here, where boards have a
'recovery' line which labs can twiddle?

> So forcing us people to switch to Labgrid because it matches your boards
> behavior is a little adventurous, and while we can add some enhancement
> to the actual pytest and hooks, I'm against saying Labgrid should be the
> golden CI system.

Who is forcing anyone to use Labgrid?

I invented my own system (Labman [3]) several years ago. Let me know
if you are interested and I can share some slides. I was thinking of
trying to tidy it up, but Tom suggested that Labgrid might be worth a
look. I had never even heard of it before that.

I don't mind about choosing golden CI systems. The more the merrier. I
want to have my boards connected to gitlab so things like bloblist
changes don't break my boards. At this point I have convinced myself
that Labgrid (as amended) is a good solution for this.

Regards,
Simon

>
> Neil
>
> >
> > I would love for you to add a board into Labgrid and see how you go.
> >
> >>
> >> [1] https://validation.linaro.org/
> >>
> >>> so having it in the main place (and perhaps eventually removing the
> >>> old way) is my goal.
> >>>
> >>>>
> >>>>>   bin/console.labgrid          | 42 ++++++++++++++++++++++++++++++
> >>>>>   bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++
> >>>>>   bin/ellesmere/conf.all       | 24 +++++++++++++++++
> >>>>>   bin/getrole.labgrid          | 25 ++++++++++++++++++
> >>>>>   bin/release.labgrid          | 22 ++++++++++++++++
> >>>>>   bin/release.none             | 22 ++++++++++++++++
> >>>>>   bin/u-boot-test-getrole      | 38 +++++++++++++++++++++++++++
> >>>>>   bin/u-boot-test-release      | 26 +++++++++++++++++++
> >>>>>   9 files changed, 295 insertions(+)
> >>>>>   create mode 100644 bin/console.labgrid
> >>>>>   create mode 100755 bin/ellesmere/common-labgrid
> >>>>>   create mode 100644 bin/ellesmere/conf.all
> >>>>>   create mode 100644 bin/getrole.labgrid
> >>>>>   create mode 100644 bin/release.labgrid
> >>>>>   create mode 100644 bin/release.none
> >>>>>   create mode 100755 bin/u-boot-test-getrole
> >>>>>   create mode 100755 bin/u-boot-test-release
> >>>>>
> >>>
> >>> [..]
> >>>
> >>> Regards,
> >>> Simon
> >
> > Regards,
> > Simon
> >
> > [2] https://validation.linaro.org/scheduler/job/4089871#results_482796468
>

[3] https://github.com/sjg20/u-boot/tree/lab6a/tools/labman