[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi,

Thanks all for comments. It sounds like people are keen on the idea so
far as it goes. I will work on a patch set complete enough to bring up
a U-Boot prompt, allowing typing of 'help' and with a special segfault
feature for anything else.

Before I do this, and to avoid me redoing work later:

1. What should I call the architecture? I have so far called it 'native'.

2. What should I call the vendor (board/xxx)? 'test' or 'sandbox'?

3. What should I call the board? Is that 'sandbox'?

4. When I create a driver, like the serial test driver, should that be
serial_test.c, test_serial.c, sandbox_serial or something else?

5. I need an arch/xxx/lib/board.c. Rather than try to create a new
minimal one, I think I should copy an existing one and change it as
little as possible (in the light of Graeme's comments and to avoid yet
another version of this file). Thoughts?

Regarding the actual test framework, I think this can come later.
Clearly we can do unit tests at the file level, or more
integration-like tests such as I listed in my email. But I know very
little about Ruby (little girl up the road?)

Summary of suggestions so that I don't flood this thread with replies:

Andreas Bießmann: why don't use some unit testing framework like
cunit, or ceedling (which
can do HW mocks easily)?

- the test framework is TBD - as Mike commented this is something else
- another level that can be piled on later.

Marek Vasut: sounds just awesome. I was planning on doing a similar
thing, but "from the
other side". I'm sure you're aware of the POST test framework. I was planning to
either extend it or write new self-standing implementation that'd allow debuging
platform-dependent drivers.

- sounds great to me - perhaps we could have a standard test for SPI
flash, another for NAND, another for USB, etc.

Mike Frysinger: i certainly think it's a worthwhile exercise,
especially if it gets us to the
point where we can do `make check` and have confidence in our common code
changes not being completely f-ed up.

- that is my hope, particularly if 'make check' is fast.

Wolfgang Denk: In a later step it might even be possible to implement some "pass
through" mode to acces actual devices / drivers in some way.  Think of
using libusb to talk to USB devices.

- Yes - that would allow checking of more layers of the USB stack. But
I even see some value in just mocking out the whole USB stack and
providing a pretend USB memory stick as a block device..

Wolfgang Denk: I'm not sure what you mean by "a mocked remote host".
We should be
able to send and receive packets from a real network interface as
well.

- I mean that the tftp command will 'obtain' a file when it asks for
one, although the actual Ethernet layer is mocked and doesn't actually
go out on the wire. Imagine an Ethernet driver which has a half-baked
tftp server in it. Yes I also see value in actually using machine
interfaces since the testing can be more thorough.

Wolfgang Denk: Some parts of what you are looking for can be done by
running U-Boot
images in a QEMU based emulated envrionment.  So far we have only a
few targets in the tree that actually support this.  I wish we had
more of them.

...I am aware that this is a different thing from what you propose here,
but I would like to state that we should not forget this option -
anybody who wants to work in that area will be more than welcome, too.

-Yes, both are valuable. Maybe one day we will get a s/w model of an
SOC when it is released. For now, this sort of thing is quite a bit of
work. There are loads of automated ways to test things- ones I can
think of are:

1.  real hardware, with some sort of control board which can push
buttons, produce/access Ethernet/USB traffic, camera to watch the LCD,
LCD to driver the camera, etc.

2. test code which runs on real hardware either at the board level or
at the U-Boot command level. Requires a user to watch / driver things

3. test code which runs on faked hardware (this is what I am proposing
here). Basically the opposite of 2.

4. test code could run on real hardware, but is actually run on QEMU
etc., with special hooks in the simulator for injecting failure,
handling the other end of ethernet//USB, etc..A lot of work but really
cool.

5. unit tests for each file, where anything that it calls is mocked
and replaced with something which can inject failure, etc. We should
have more of this anyway. People often don't like writing it though
:-(

Graeme Russ: Indead - And I think it will serve as further impetus to
arch-neutralise
code outside /arch/

- Yes - let's hope so.

Regards,
Simon