[PATCH v5 00/20] labgrid: Provide an integration with Labgrid

[Simon Glass]

Hi Tom,

On Fri, 30 Aug 2024 at 15:53, Tom Rini <trini at konsulko.com> wrote:
>
> On Thu, Aug 29, 2024 at 09:00:25AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Thu, 29 Aug 2024 at 08:13, Tom Rini <trini at konsulko.com> wrote:
> > >
> > > On Wed, Aug 28, 2024 at 04:08:26PM -0600, Simon Glass wrote:
> > >
> > > > Labgrid provides access to a hardware lab in an automated way. It is
> > > > possible to boot U-Boot on boards in the lab without physically touching
> > > > them. It relies on relays, USB UARTs and SD muxes, among other things.
> > > >
> > > > By way of background, about 4 years ago I wrong a thing called Labman[1]
> > > > which allowed my lab of about 30 devices to be operated remotely, using
> > > > tbot for the console and build integration. While it worked OK and I
> > > > used it for many bisects, I didn't take it any further.
> > > >
> > > > It turns out that there was already an existing program, called Labgrid,
> > > > which I did not know about at time (thank you Tom for telling me). It is
> > > > more rounded than Labman and has a number of advantages:
> > > >
> > > > - does not need udev rules, mostly
> > > > - has several existing users who rely on it
> > > > - supports multiple machines exporting their devices
> > > >
> > > > It lacks a 'lab check' feature and a few other things, but these can be
> > > > remedied.
> > > >
> > > > On and off over the past several weeks I have been experimenting with
> > > > Labgrid. I have managed to create an initial U-Boot integration (this
> > > > series) by adding various features to Labgrid[2] and the U-Boot test
> > > > hooks.
> > > >
> > > > I hope that this might inspire others to set up boards and run tests
> > > > automatically, rather than relying on infrequent, manual test. Perhaps
> > > > it may even be possible to have a number of labs available.
> > > >
> > > > Included in the integration are a number of simple scripts which make it
> > > > easy to connect to boards and run tests:
> > > >
> > > > ub-int <target>
> > > >     Build and boot on a target, starting an interactive session
> > > >
> > > > ub-cli <target>
> > > >     Build and boot on a target, ensure U-Boot starts and provide an interactive
> > > >     session from there
> > > >
> > > > ub-smoke <target>
> > > >     Smoke test U-Boot to check that it boots to a prompt on a target
> > > >
> > > > ub-bisect <target>
> > > >     Bisect a git tree to locate a failure on a particular target
> > > >
> > > > ub-pyt <target> <testspec>
> > > >     Run U-Boot pytests on a target
> > > >
> > > > Some of these help to provide the same tbot[4] workflow which I have
> > > > relied on for several years, albeit much simpler versions.
> > > >
> > > > The goal here is to create some sort of script which can collect
> > > > patches from the mailing list, apply them and test them on a selection
> > > > of boards. I suspect that script already exists, so please let me know
> > > > what you suggest.
> > > >
> > > > I hope you find this interesting and take a look!
> > > >
> > > > [1] https://github.com/sjg20/u-boot/tree/lab6a
> > > > [2] https://github.com/labgrid-project/labgrid/pull/1411
> > > > [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid
> > > > [4] https://tbot.tools/index.html
> > >
> > > Part of my concern here is still that this series contains:
> > >
> > > - Generic test improvements and fixes:
> > >
> > > >   test: Use a constant for the test timeout
> > > >   test: Pass stderr to stdout
> > > >   test: Avoid failing skipped tests
> > > >   test: Move the receive code into a function
> > > >   test: Separate out the exception handling
> > > >   test: Detect dead connections
> > > >   test: Tidy up remaining exceptions
> > > >   test: Improve handling of sending commands
> > > >   test: Fix mulptiplex_log typo
> > > >   test: Avoid double echo when starting up
> >
> > So should I send those as a separate series?
>
> Well, as some of the other comments have been getting to, I don't know
> what's really useful / needed, or not here.
>
> > Note that when things land we will need to look at the gitlab output,
> > to make sure it is what we want.
>
> In general getting our pytest output more understandable is always nice
> and useful, but this shouldn't be any different from all the platforms
> we already test in CI.

OK I can adjust that. For my testing I have it spitting out all the
output, since hunting through log files to see why a particular board
didn't boot is a right pain.

>
> > > - Labgrid specific listed as generic changes:
> > >
> > > >   test: Release board after tests complete
> > > >   test: Try to shut down the lab console gracefully
> > >
> > > - Your labgrid+builman implementation specific changes:
> > >
> > > >   test: Allow signaling that U-Boot is ready
> > > >   test: Allow connecting to a running board
> > > >   test: Create a common function to get the config
> > > >   test: Introduce the concept of a role
> > > >   test: Introduce lab mode
> > > >   test: Add a section for closing the connection
> > > >   test: Support testing with two board-builds
> > >
> > > Which makes this harder to review and clearly merge. For example, we
> > > already have hooks for power on / power off, and we should just call
> > > them as wow that would make everyones lab easier (I used to have a
> > > pre-script to turn on everything).
> >
> > Power on/off is handled by the strategy[1] in Labgrid. Some boards
> > have power control, some don't. Some use reset, some don't. Some need
> > a recovery button to be pressed when resetting. The fundamental point
> > here is that Labgrid knows how to manage the lab, so any external
> > scripts are always going to be fiddly. Also, they are such a pain to
> > maintain.
> >
> > I see Labgrid as a lab manager, something which looks after the lab
> > and knows how to operate it.
>
> I think the biggest take away here is that we need to make sure whatever
> goes in to the hooks repository can be easily customized to fit how
> someone uses their lab, and that's going to be different from what I
> have and what you have.

Yes and be sure that my Labgrid stuff doesn't stomp on some other use case.

Regards,
Simon