[RFC] make sandbox UT more generic

[Simon Glass]

Hi,

On Tue, 5 Sept 2023 at 21:00, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
>
> Hi Simon,
>
> On Thu, Aug 31, 2023 at 09:04:43AM -0600, Simon Glass wrote:
> > Hi,
> >
> > On Wed, 30 Aug 2023 at 23:28, AKASHI Takahiro
> > <takahiro.akashi at linaro.org> wrote:
> > >
> > > Hi Simon,
> > >
> > > On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> > > > Hi,
> > > >
> > > > On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> > > > <takahiro.akashi at linaro.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > > > > and want to re-use sandbox UT to test the code. However,
> > > > > It is somehow sandbox-specific (with additional DT nodes).
> > > > > How can/should we make it more generic for other targets/drivers
> > > > > rather than just by copying the test code?
> > > > > (I have already created a test for pinmux since there is only
> > > > > one existing scenario, but gpio test has many.)
> > > > >
> > > > > Even if I say 'generic', my case may be special since real
> > > > > hardware (device drivers) cannot always run all the test cases,
> > > > > while SCMI-based drivers potentially can with a dummy SCMI server
> > > > > for sandbox.
> > > > > See:
> > > > >     drivers/firmware/scmi/sandbox-scmi_agent.c
> > > >
> > > > We don't have a good way to test drivers that talk to hardware, in general.
> > > >
> > > > For I2C, SPI and some PCI devices you can sometimes write an emulator
> > > > for the chip and then your driver can talk to the emulator as if it
> > > > were talking to the hardware. Sandbox does actually support that with
> > > > memory-mapped I/O too, although it is fairly rarely used.
> > >
> > > Well, I don't want or need to emulate some *real* hardware.
> > > Instead, I would like to emulate what the current sandbox drivers
> > > (pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
> > > (some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).
> > >
> > > As you might know, SCMI protocol with associated drivers on U-Boot is
> > > so generic that it would be able to talk to any of real pinctrl/gpio
> > > drivers/firmware (say, run on OPTEE or SCP).
> > > By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
> > > SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
> > > similar to what sandbox does.
> >
> > I actually know almost nothing about SCMI.
> >
> > >
> > > I have already implemented pinmux test with some tweaks by copying
> > > test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
> > > in test.dts.
> > > But I'm looking for any other means without test code duplication.
> > >
> > > Did I clarify my question a bit?
> >
> > Well you should be able to factor out the test code into a function,
> > then call it from two places with the two different devices (or other
> > params) that are needed.
> >
> > For the DT, copying a few nodes is not the end of the world, IMO.
> >
> > BTW have you seen this talk? [2] It seems that you are moving pieces
> > into firmware which should be OS drivers?
> >
> > Anyway, if you place a sandbox pinmux device under the SCMI node in
> > the DT, then you should end up with a pinmux device you can use likely
> > normal. Then if that device uses the sandbox emulator, you can run the
> > existing tests on it with little modification, I suspect.
> >
> > But if I am still missing the point, a diagram or patch might help me
> > understand!
>
> I just posted my RFC for supporting SCMI pinctrl protocol[1],
> hoping it will help you understand what I'm planning to do regarding
> test methodology, in particular by looking at patch#5 and #6.

OK...I am mostly out this week but will take a look next week.


>
> [1] https://lists.denx.de/pipermail/u-boot/2023-September/529765.html
>
> Thanks,
> -Takahiro Akashi
>
> > Regards,
> > Simon
> >
> >
> > >
> > > -Takahiro Akashi
> > >
> > >
> > > > We have done this a lot with Zephyr, as well[1] and achieved 90% code
> > > > coverage on some boards.
> > > >
> > > > But I'm not quite sure I am answering the right question, so I will stop here.
> > > >
> > > > Regards,
> > > > Simon
> > > >
> > > > [1] https://www.youtube.com/watch?v=usXCAXR2G_c
> >
> > [2] https://www.usenix.org/conference/osdi21/presentation/fri-keynote

Regards,
Simon