[PATCH v4 42/42] doc: Explain briefly how to write new tests

Mon Mar 8 00:21:08 CET 2021

Hi Heinrich,

On Thu, 4 Mar 2021 at 13:14, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 04.03.21 14:51, Simon Glass wrote:
> > Add a second on writing tests, covering when to use Python and C, where
> > to put the tests, etc. Add a link to the existing Python test
> > documentation.
> >
> > Signed-off-by: Simon Glass <sjg at chromium.org>
> > ---
> >
> > Changes in v4:
> > - Add correct SPDX header
> >
> > Changes in v2:
> > - Add new patches to cover running an SPL test
> >
> >  doc/develop/index.rst         |   1 +
> >  doc/develop/py_testing.rst    |   3 +-
> >  doc/develop/testing.rst       |   2 +
> >  doc/develop/tests_sandbox.rst |   7 +
> >  doc/develop/tests_writing.rst | 339 ++++++++++++++++++++++++++++++++++
> >  5 files changed, 351 insertions(+), 1 deletion(-)
> >  create mode 100644 doc/develop/tests_writing.rst
> >
> > diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> > index 9208668a2d4..8f0a598b5f2 100644
> > --- a/doc/develop/index.rst
> > +++ b/doc/develop/index.rst
> > @@ -32,6 +32,7 @@ Testing
> >
> >     testing
> >     py_testing
> > +   tests_writing
> >     tests_sandbox
> >
> >  Refactoring
> > diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst
> > index 7f01858cfda..c4cecc0a01b 100644
> > --- a/doc/develop/py_testing.rst
> > +++ b/doc/develop/py_testing.rst
> > @@ -13,7 +13,8 @@ results. Advantages of this approach are:
> >    U-Boot; there can be no disconnect.
> >  - There is no need to write or embed test-related code into U-Boot itself.
> >    It is asserted that writing test-related code in Python is simpler and more
> > -  flexible than writing it all in C.
> > +  flexible than writing it all in C. But see :doc:`tests_writing` for caveats
> > +  and more discussion / analysis.
> >  - It is reasonably simple to interact with U-Boot in this way.
> >
> >  Requirements
> > diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
> > index b181c2e2e41..ced13ac8bb4 100644
> > --- a/doc/develop/testing.rst
> > +++ b/doc/develop/testing.rst
> > @@ -117,6 +117,8 @@ or is covered sparingly. So here are some suggestions:
> >    is much easier to add onto a test - writing a new large test can seem
> >    daunting to most contributors.
> >
> > +See doc:`tests_writing` for how to write tests.
> > +
> >
> >  Future work
> >  -----------
> > diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
> > index 8c2eac4c5f4..7e5cc84e43c 100644
> > --- a/doc/develop/tests_sandbox.rst
> > +++ b/doc/develop/tests_sandbox.rst
> > @@ -199,3 +199,10 @@ linker_list::
> >     000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent
> >     000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle
> >     000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props
> > +
> > +
> > +Writing tests
> > +-------------
> > +
> > +See :doc:`tests_writing` for how to write new tests.
> > +
> > diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst
> > new file mode 100644
> > index 00000000000..d926c00f6e1
> > --- /dev/null
> > +++ b/doc/develop/tests_writing.rst
> > @@ -0,0 +1,339 @@
> > +.. SPDX-License-Identifier: GPL-2.0+
> > +.. Copyright 2021 Google LLC
> > +.. sectionauthor:: Simon Glass <sjg at chromium.org>
> > +
> > +Writing Tests
> > +=============
> > +
> > +This describes how to write tests in U-Boot and describes the possible options.
> > +
> > +Test types
> > +----------
> > +
> > +There are two basic types of test in U-Boot:
> > +
> > +  - Python tests, in test/py/tests
> > +  - C tests, in test/ and its subdirectories
>
> You missed lib/efi_selftest/ for the UEFI tests.

OK I will mention that. But really it is something of a pain that they
are separate.

>
> > +
> > +Python tests talk to U-Boot via the command line. They support both sandbox and
> > +real hardware. They typically do not require building test code into U-Boot
> > +itself. They are fairly slow to run, due to the command-line interface and there
> > +being two separate processes. Python tests are fairly easy to write. They can
> > +be a little tricky to debug sometimes due to the voluminous output of pytest.
> > +
> > +C tests are written directly in U-Boot. While they can be used on boards, they
> > +are more commonly used with sandbox, as they obviously add to U-Boot code size.
> > +C tests are easy to write so long as the required facilities exist. Where they
> > +do not it can involve refactoring or adding new features to sandbox. They are
> > +fast to run and easy to debug.
> > +
> > +Regardless of which test type is used, all tests are collected and run by the
> > +pytest framework, so there is typically no need to run them separately. This
> > +means that C tests can be used when it makes sense, and Python tests when it
> > +doesn't.
> > +
> > +
> > +This table shows how to decide whether to write a C or Python test:
> > +
> > +=====================  ===========================  =============================
> > +Attribute              C test                       Python test
> > +=====================  ===========================  =============================
> > +Fast to run?           Yes                          No (two separate processes)
> > +Easy to write?         Yes, if required test        Yes
> > +                       features exist in sandbox
>
> "in sandbox"? Better "on the target system" which might be the sandbox.

OK.

>
> You cannot test architecture specific features on the sandbox, e.g.
>
> * hardware drivers
> * communication with firmware like TF-A and SBI
> * FPGA code
> * exception handling
>
> These can be tested on real hardware using C tests.

We have managed to test various arch-specific things on sandbox. We
could easily add FPGA tests just by implementing a suitable driver.

The bit that I think you are missing here is that sandbox is the
standard way of running tests. Of course we can run on real hardware,
but that requires having that hardware and having the out-of-tree test
harness stuff set up (Stephen Warren's work). The more we can do in
sandbox, the easier it is.

>
> We still have not succeeded in running file system tests in Gitlab.

I never run them as they are too slow.

>
> For both C and Python there are trivial cases and hard ones.
>
> > +Needs code in U-Boot?  Yes                          No, provided the test can be
> > +                                                    executed and the result
> > +                                                    determined using the command
> > +                                                    line
> > +Easy to debug?         Yes                          Yes, but the amount of output can
> > +                                                    sometimes require a bit of digging
>
> Debugging the Python tests is non-trivial. Just try to find out why some
> FAT tests fail. Though Python calls sync in the test preparation the
> Python test does not see the full generated file.
>
> The problem is not the output but the non-output.

OK well I'll try some different wording.

>
> > +Can use gdb?           Yes, directly                Yes, with --gdbserver
> > +Can run on boards?     Some can, but only if        Yes
> > +                       compiled in and not
> > +                       dependent on sandbox
>
> Many Python tests cannot be run on boards.

OK will update it to 'some'

>
> > +=====================  ===========================  =============================
> > +
> > +
> > +Python or C
> > +-----------
> > +
> > +Typically in U-Boot we encourage C test using sandbox for all features. This
> > +allows fast testing, easy development and allows contributors to make changes
> > +without needing dozens of boards to test with.
> > +
> > +When a test requires setup or interaction with the running host (such as to
> > +generate images and then running U-Boot to check that they can be loaded), or
> > +cannot be run on sandbox, Python tests should be used. These should typically
>
> Why should I use Python if the sandbox cannot run it? I could still
> write a C test invoked by the "ut" command.
>
> > +NOT rely on running with sandbox, but instead should function correctly on any
> > +board supported by U-Boot.
> > +
> > +
> > +How slow are Python tests?
> > +--------------------------
> > +
> > +Under the hood, Python tests work by starting a sandbox test and connecting to
>
> Above you said that they should run on any hardware. So why do you refer
> to the sandbox here?

Because I am talking about sandbox. I will update it to make that clear.

[..]

> > +Writing C tests
> > +---------------
> > +
> > +C tests are arranged into suites which are typically executed by the 'ut'
> > +command. Each suite is in its own file. This section describes how to accomplish
> > +some common test tasks.
>
> UEFI tests are written in C, started with 'bootefi selftest', and are in
> non of the suites you refer to.

OK will mention that again here.

Regards,
Simon