[PATCH v5 00/20] labgrid: Provide an integration with Labgrid

[Tom Rini]

On Thu, Aug 29, 2024 at 09:00:25AM -0600, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 29 Aug 2024 at 08:13, Tom Rini <trini at konsulko.com> wrote:
> >
> > On Wed, Aug 28, 2024 at 04:08:26PM -0600, Simon Glass wrote:
> >
> > > Labgrid provides access to a hardware lab in an automated way. It is
> > > possible to boot U-Boot on boards in the lab without physically touching
> > > them. It relies on relays, USB UARTs and SD muxes, among other things.
> > >
> > > By way of background, about 4 years ago I wrong a thing called Labman[1]
> > > which allowed my lab of about 30 devices to be operated remotely, using
> > > tbot for the console and build integration. While it worked OK and I
> > > used it for many bisects, I didn't take it any further.
> > >
> > > It turns out that there was already an existing program, called Labgrid,
> > > which I did not know about at time (thank you Tom for telling me). It is
> > > more rounded than Labman and has a number of advantages:
> > >
> > > - does not need udev rules, mostly
> > > - has several existing users who rely on it
> > > - supports multiple machines exporting their devices
> > >
> > > It lacks a 'lab check' feature and a few other things, but these can be
> > > remedied.
> > >
> > > On and off over the past several weeks I have been experimenting with
> > > Labgrid. I have managed to create an initial U-Boot integration (this
> > > series) by adding various features to Labgrid[2] and the U-Boot test
> > > hooks.
> > >
> > > I hope that this might inspire others to set up boards and run tests
> > > automatically, rather than relying on infrequent, manual test. Perhaps
> > > it may even be possible to have a number of labs available.
> > >
> > > Included in the integration are a number of simple scripts which make it
> > > easy to connect to boards and run tests:
> > >
> > > ub-int <target>
> > >     Build and boot on a target, starting an interactive session
> > >
> > > ub-cli <target>
> > >     Build and boot on a target, ensure U-Boot starts and provide an interactive
> > >     session from there
> > >
> > > ub-smoke <target>
> > >     Smoke test U-Boot to check that it boots to a prompt on a target
> > >
> > > ub-bisect <target>
> > >     Bisect a git tree to locate a failure on a particular target
> > >
> > > ub-pyt <target> <testspec>
> > >     Run U-Boot pytests on a target
> > >
> > > Some of these help to provide the same tbot[4] workflow which I have
> > > relied on for several years, albeit much simpler versions.
> > >
> > > The goal here is to create some sort of script which can collect
> > > patches from the mailing list, apply them and test them on a selection
> > > of boards. I suspect that script already exists, so please let me know
> > > what you suggest.
> > >
> > > I hope you find this interesting and take a look!
> > >
> > > [1] https://github.com/sjg20/u-boot/tree/lab6a
> > > [2] https://github.com/labgrid-project/labgrid/pull/1411
> > > [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid
> > > [4] https://tbot.tools/index.html
> >
> > Part of my concern here is still that this series contains:
> >
> > - Generic test improvements and fixes:
> >
> > >   test: Use a constant for the test timeout
> > >   test: Pass stderr to stdout
> > >   test: Avoid failing skipped tests
> > >   test: Move the receive code into a function
> > >   test: Separate out the exception handling
> > >   test: Detect dead connections
> > >   test: Tidy up remaining exceptions
> > >   test: Improve handling of sending commands
> > >   test: Fix mulptiplex_log typo
> > >   test: Avoid double echo when starting up
> 
> So should I send those as a separate series?

Well, as some of the other comments have been getting to, I don't know
what's really useful / needed, or not here.

> Note that when things land we will need to look at the gitlab output,
> to make sure it is what we want.

In general getting our pytest output more understandable is always nice
and useful, but this shouldn't be any different from all the platforms
we already test in CI.

> > - Labgrid specific listed as generic changes:
> >
> > >   test: Release board after tests complete
> > >   test: Try to shut down the lab console gracefully
> >
> > - Your labgrid+builman implementation specific changes:
> >
> > >   test: Allow signaling that U-Boot is ready
> > >   test: Allow connecting to a running board
> > >   test: Create a common function to get the config
> > >   test: Introduce the concept of a role
> > >   test: Introduce lab mode
> > >   test: Add a section for closing the connection
> > >   test: Support testing with two board-builds
> >
> > Which makes this harder to review and clearly merge. For example, we
> > already have hooks for power on / power off, and we should just call
> > them as wow that would make everyones lab easier (I used to have a
> > pre-script to turn on everything).
> 
> Power on/off is handled by the strategy[1] in Labgrid. Some boards
> have power control, some don't. Some use reset, some don't. Some need
> a recovery button to be pressed when resetting. The fundamental point
> here is that Labgrid knows how to manage the lab, so any external
> scripts are always going to be fiddly. Also, they are such a pain to
> maintain.
> 
> I see Labgrid as a lab manager, something which looks after the lab
> and knows how to operate it.

I think the biggest take away here is that we need to make sure whatever
goes in to the hooks repository can be easily customized to fit how
someone uses their lab, and that's going to be different from what I
have and what you have.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20240830/79c5a07a/attachment.sig>