[U-Boot] [PATCH 01/14] test: Add a README

[Simon Glass]

+Roger, for Travis-CI

Hi Teddy,

On 3 July 2016 at 13:17, Teddy Reed <teddy.reed at gmail.com> wrote:
> Hi Simon,
>
> On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass <sjg at chromium.org> wrote:
>> Add a few notes about how testing works in U-Boot.
>>
>> Signed-off-by: Simon Glass <sjg at chromium.org>
>
> Reviewed-by: Teddy Reed <teddy.reed at gmail.com>

Thanks for the comments. Will tidy up.

>
>> ---
>>
>>  test/README | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 82 insertions(+)
>>  create mode 100644 test/README
>>
>> diff --git a/test/README b/test/README
>> new file mode 100644
>> index 0000000..dfd83d6
>> --- /dev/null
>> +++ b/test/README
>> @@ -0,0 +1,82 @@
>> +Testing in U-Boot
>> +=================
>> +
>> +U-Boot has a large amount of code. This file describes how this code is
>> +tested and what tests you should write when adding a new feature.
>> +
>> +
>> +Sandbox
>> +-------
>> +U-Boot can be built as a user-space application (e.g. for Linux). This
>> +allows test to be executed without needing target hardware. The 'sandbox'
>> +target provides this feature and it is widely used in tests.
>> +
>> +
>> +Pytest Suite
>> +------------
>> +
>> +Many tests are available using the pytest suite, in test/py. This can run
>> +either on sandbox or on real hardware. It relies on the U-Boot console to
>> +inject test commands and check the result. It is slower to run than C code,
>> +but provides the ability to unify lots of test and summarise their results.
>
> lots of tests
>
>> +
>> +You can run the tests on sandbox with:
>> +
>> +       ./test/py/test.py --bd sandbox --build
>> +
>> +This will produce HTML output in build-sandbox/test-log.html
>> +
>> +See test/py/README.md for more information about the pytest suite.
>> +
>> +
>> +tbot
>> +----
>> +
>> +Tbot provides a way to execute tests on target hardware. It is intended for
>> +trying out both U-Boot and Linux (and potentially other software) on a
>> +number of boards automatically. It can be used to create a continuous test
>> +environment. See tools/tbot/README for more information.
>> +
>> +
>> +Ad-hoc tests
>> +------------
>> +
>> +There are several ad-hoc tests which run outside the pytest environment:
>> +
>> +   test/fs     - File system test (shell script)
>> +   test/image  - FIT and lagacy image tests (shell script and Python)
>
> s/lagacy/legacy/
>
>> +   test/stdint - A test that stdint.h can be used in U-Boot (shell script)
>> +   trace       - Test for the tracing feature (shell script)
>> +   vboot       - Test for verified boot (shell script)
>> +
>> +The above should be converted to run as part of the pytest suite.
>
> Is this a NOTE or a TODO?

TODO - I'll reword it.

>
>> +
>> +
>> +When to write tests
>> +-------------------
>> +
>> +If you add code to U-Boot without a test you are taking a risk. Even if you
>> +perform thorough manual testing at the time of submission, it may break when
>> +future changes are made to U-Boot. It may even break when applied to mainline,
>> +if other changes interact with it. A good mindset is that untested code
>> +probably doesn't work and should be deleted.
>> +
>> +You can assume that the Pytest suite will be run before patches are accepted
>> +to mainline, so this provides protection against future breakage.
>> +
>> +On the other hand there is quite a bit of code that is not covered with tests,
>> +or is covered sparingly. So here are some suggestions:
>> +
>> +- If you are adding a new uclass, add a sandbox driver and a test that uses it
>> +- If you are modifying code covered by an existing test, add a new test case
>> +  to cover your changes
>> +- If the code you are modifying has not tests, consider writing one. Even a
>> +  very basic test is useful, and may be picked up and enhanced by others. It
>> +  is much easier to add onto a test - writing a new large test can seem
>> +  daunting to most contributors.
>> +
>> +
>> +Future work
>> +-----------
>> +
>> +Converting existing shell scripts into pytest tests.
>> --
>> 2.8.0.rc3.226.g39d4020
>>
>> _______________________________________________
>> U-Boot mailing list
>> U-Boot at lists.denx.de
>> http://lists.denx.de/mailman/listinfo/u-boot
>
> Thank for these docs! I wonder if a clone/mirror of the repo on Github
> can be set up to run the sandbox/Pytest tests in TravisCI with minimum
> effort? :)

There is a .travis.yml file - I have not tried this but I believe
people are already doing it somewhere.

Regards.
Simon