[U-Boot] RFC: Testing U-Boot Part 1

[Simon Glass]

Hi,

Summary: I am quite keen on improving the test infrastructure in
U-Boot. I would like to have a test suite that can run in a minute or
two on a Linux PC and test all non-platform code.

Detail
======
We can break the U-Boot code base into two parts:

1. Platform code, which is SOC-specifc. At present this is the CPU
init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the
drivers directory). There is also a small amount of generic CPU code
in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not
within the SOC).

2. Portable/Generic U-Boot code, which is cross-platform. This
includes many drivers, the various commands, file system support,
things like the MMC, USB and network stacks, etc.

My focus in this email the the second part of the code base - all the
code which is not platform-specific, but can still have bugs.

Proposal
========
To a large extent, testing of this part of the code base could simply
be built on one or more of the available platforms. We could then run
the tests on that platform, and announce that the code base works fine
on that platform. Obviously the platform needs to support all possible
features of U-Boot.

However, this approach fails to take advantage of two useful
properties of this code:

- It is cross-platform, and even runs on x86
- It is standalone, requiring no OS libraries to run

For speed, debugging and convenience, it would be nice to run U-Boot
under a generic Linux environment on a workstation, and test all the
generic non-platform code. The basic problem with this is that the
non-platform code requires the platform code to operate. Even the x86
platform code is designed for standalone operation on a particular x86
board, and is not suitable for running under x86 Linux.

To get around this I propose that we create a new ‘native’
architecture. We write code in ‘arch/native’ which can run under
Linux. Since all the non-platform code will happily run under this new
‘architecture’, we can then write tests which run quickly under x86
Linux (or another Linux for that matter). This U-Boot 'architecture'
should build natively on any 32/64-bit Linux machine since it just
uses standard Linux system calls. Calls to Linux would be entirely
within this arch/native subdirectory.

Benefit
=======
What will this buy us? Well we can do things like:

- Create a test SPI flash device, which is file-backed. Use this to
write a test of the cmd_sf layer by issuing a series of commands and
making sure that the correct SPI flash contents is obtained

- Create a test MMC or IDE device, backed by a file. Use this to issue
ext2 and fat commands to manipulate the filesystem. Then loopback
mount the file and check from Linux that U-Boot did the right thing

- Create a test Ethernet device with a mocked remote host attached.
Use this to issue network commands like bootp and tftp and check that
the correct results are obtained.

Ultimately (one day) we might have a set of unit tests for U-Boot
which can run to very quickly check that a patch does not break the
core U-Boot code.

Comments
========
At this early stage I am looking for comments on the concept - how
useful it is and how best to implement it.

Regards,
Simon