[U-Boot] Using kernelCI infrastructure for firmware testing

[Kevin Hilman]

"Patrick Rudolph" <patrick.rudolph at 9elements.com> writes:

> Hi folks,
> this is an attempt to improve firmware testing by using the
> infrastructure and knowledge of the kernelci community. If you think
> this is not the right place, please point me in the right direction.
>
> I'm a coreboot[1] developer trying to make sure that the master
> branch[2] doesn't regress. Currently there's no public firmware
> testing, only internal validation suites used by some companies that
> lack direct and automated feedback before a commit is actually merged.
> As this isn't a coreboot only topic, but applies to all open source
> "bios vendors", I added the u-boot project in CC as well.
>
> For me firmware testing looks pretty similar to kernel testing:
> * flash firmware to test
> * boot a known good linux kernel
> * run tests in userspace and verify hardware/software works as expected
>
> On the hardware side we have boards in our lab that allow remote power
> cycling and firmware flashing. It is attached to self hosted stock
> LAVA2018. But as we are firmware engineers, we don't want to deal with
> the administration of servers.
>
> Here are a few questions for you:
> * Would it make sense to also cover open source firmware tests on kernelci?

Yes.  We've talked about this a few times over the years, and all the
infra and tooling we have built for kernelCI would be well suited for
firmware testing.

The catch is mostly in time/resources invested.  For now, we are
primarily focused on kernel testing, but if there are contributors that
want to extend this to firmware, we are an open-source project and
welcome the contributions.

I would say the primary challenge here is that each SoC family, and
sometimes even each board has unique challenges to (re)flashing the
firmware as well as the ability to recover from non-working firmware.

This presents a problem not necessariuly for KernelCI but for the lab
admins for that hardware, and especially those writing the LAVA
device-types for those platforms.

I've added the main LAVA developer Remi Durrafort to Cc since he's been
doing a lot of work in LAVA to make firmware updates and recovery mode
usable from LAVA.  Recent versions of LAVA seem to be growing better
support for this:
https://docs.lavasoftware.org/lava/bootloaders.html

> * Do you build the linux images yourself?

We build all the linux kernel images, but rely on the lab admins to have
build/flashed working firmware and bootloaders.

> * Would you accept firmware images generated by a third party?

As long as the source is available and can be (re)built, that's
typically fine.

> * Can anybody get an account for the LAVA server to run firmware test?

That's up to each LAVA lab administrator.  Today, KernelCI uses a
distributed set of (mostly LAVA) labs.  Each lab admin has given
priveleges to KernelCI to send jobs.

> * What communication channels do you recommend?

This mailing list is a good place to start, but to be completely
honest, firmware testing is relatively low on our current list of
priorites.  But as I said above, it's feature we will welcome with open
arms, even if it's not a feature that the current set of developers will
be very active on.

I would say your first step is probably to get some support for the
devices you care about into upstream LAVA.  Once that is working, we
would work on how KernelCI could start generating jobs for those devices.

> * Will there be meetings or conferences to get in contact with the
> community to talk about this?

We just had an automated-testing summit connected to Embedded Linux
Conference Europe last month, where many of us were gathered.

Remi even gave a talk on bootloader testing with LAVA, maybe he can
share the slides for that.

Kevin