[PATCH v4 06/42] doc: Explain how to run tests without pytest

Sun Mar 7 23:49:14 CET 2021

Hi Heinrich,

On Thu, 4 Mar 2021 at 11:13, Heinrich Schuchardt <xypron.glpk at gmx.de> wrote:
>
> On 3/4/21 2:50 PM, Simon Glass wrote:
> > Add details about how to run a sandbox test directly, without using
> > pytest. This is more convenient for rapid development, since it is faster
> > and allows easier use of a debugger. Also mention sandbox_flattree as an
> > example of the different sandbox builds available.
> >
> > Signed-off-by: Simon Glass <sjg at chromium.org>
> > ---
> >
> > (no changes since v2)
> >
> > Changes in v2:
> > - Put the docs in tests_sandbox since it is more related to sandbox
> > - Put in a mention of tests_sandbox in the main testing docs
> >
> >   doc/develop/index.rst         |  1 +
> >   doc/develop/testing.rst       |  9 ++++
> >   doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++
>
> Thank you for adding this documentation.
>
> tests_sandbox.rst seems to be the wrong file name:

But if you look at it, it is all about running tests using sandbox.

>
> We have three types of tests:
>
> * Python tests
> * library tests started using the 'ut' shell commad
> * UEFI related tests started using the 'bootefi selftest' command
>
> All three tpyes of tests are run both on real hardware and on the sandbox.
>
> Unfortunately a few Python tests require the sandbox.
>
> Some tests are not run on the sandbox by default though they could, e.g.
> the UEFI tests run only on QEMU because you decided not to activate
> CONFIG_EFI_SELFTEST on sandbox_defconfig.

It was a long time ago, but I recall spending a year trying to
upstream a bit of refactoring to allow it and then giving up. If you
have a way to do it, please go ahead.

>
> Some tests can only be used interactively, e.g.
>
>      setenv efi_selftest block image transfer
>      bootefi selftest
>
> Some tests cannot be run on the sandbox at all, e.g.
> test/py/tests/test_fpga.py

That is a bug :-)

>
> >   3 files changed, 89 insertions(+)
> >   create mode 100644 doc/develop/tests_sandbox.rst
> >
> > diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> > index ac57fdb8f30..50b1de3bdff 100644
> > --- a/doc/develop/index.rst
> > +++ b/doc/develop/index.rst
> > @@ -33,3 +33,4 @@ Testing
> >      coccinelle
> >      testing
> >      py_testing
> > +   tests_sandbox
> > diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
> > index f01ca4dc408..87c90eee271 100644
> > --- a/doc/develop/testing.rst
> > +++ b/doc/develop/testing.rst
> > @@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This
> >   allows test to be executed without needing target hardware. The 'sandbox'
> >   target provides this feature and it is widely used in tests.
> >
> > +See :doc:`tests_sandbox` for more information.
> >
> >   Pytest Suite
> >   ------------
> > @@ -51,8 +52,16 @@ You can run the tests on sandbox with::
> >
> >   This will produce HTML output in build-sandbox/test-log.html
> >
> > +Some tests run with other versions of sandbox. For example sandbox_flattree
> > +runs the tests with livetree (the hierachical devicetree) disabled. You can
> > +also select particular tests with -k::
> > +
> > +   ./test/py/test.py --bd sandbox_flattree --build -k hello
> > +
> >   See test/py/README.md for more information about the pytest suite.
> >
> > +See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
> > +
> >
> >   tbot
> >   ----
> > diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
> > new file mode 100644
> > index 00000000000..85bbd4f6734
> > --- /dev/null
> > +++ b/doc/develop/tests_sandbox.rst
> > @@ -0,0 +1,79 @@
> > +.. SPDX-License-Identifier: GPL-2.0+
> > +
> > +Tests Under the Hood
> > +====================
> > +
> > +Running sandbox tests directly
> > +------------------------------
> > +
> > +Typically tests are run using the pytest suite. This is easy and always gets
> > +things right.
>
> No, in the contrary:
>
> It is difficult to set up Python testing for real hardware. Running
> non-Python tests from the console is trivial both on the sandbox and on
> real hardware.

OK I will reword it.

>
> > +
> > +But it is also possible to run some sandbox tests directly. For example, this
> > +runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
> > +
> > +   $ ./u-boot -T -c "ut dm gpio"
> This works only on the sandbox. To create examples assume that the shell
> is already started:

The whole point of this file is to describe using sandbox!

>
>      => ut dm gpio
>
> and
>
>      => bootefi selftest
>
> We should add a man-page for the ut command in /doc/usage/.

OK, please go ahead.

>
> > +
> > +
> > +   U-Boot 2021.01
> > +
> > +   Model: sandbox
> > +   DRAM:  128 MiB
> > +   WDT:   Started with servicing (60s timeout)
> > +   MMC:   mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
> > +   In:    serial
> > +   Out:   vidconsole
> > +   Err:   vidconsole
> > +   Model: sandbox
> > +   SCSI:
> > +   Net:   eth0: eth at 10002000, eth5: eth at 10003000, eth3: sbe5, eth6: eth at 10004000
> > +   Test: dm_test_gpio: gpio.c
> > +   Test: dm_test_gpio: gpio.c (flat tree)
> > +   Failures: 0
> > +
>
> Only the text below is sandbox specific.
>
> > +The -T option tells U-Boot to run with the 'test' devicetree (test.dts) instead
>
> %s/U-Boot/the U-Boot sandbox/g
>
> > +of -D which selects the normal sandbox.dts - this is necessary because many
> > +tests rely on things in the test devicetree. If you try running tests without
>
> %s/things/nodes/ ?

or properties

>
> > +-T then you may see failures, like::
> > +
> > +   $ ./u-boot -c "ut dm gpio"
> > +
> > +
> > +   U-Boot 2021.01
> > +
> > +   DRAM:  128 MiB
> > +   WDT:   Not found!
> > +   MMC:
> > +   In:    serial
> > +   Out:   serial
> > +   Err:   serial
> > +   SCSI:
> > +   Net:   No ethernet found.
> > +   Please run with test device tree:
> > +       ./u-boot -d arch/sandbox/dts/test.dtb
> > +   Test: dm_test_gpio: gpio.c
> > +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> > +   Test: dm_test_gpio: gpio.c (flat tree)
> > +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> > +   Failures: 2
> > +
> > +The message above should provide a hint if you forget. Even running with -D
>
> If I forget what? Please, complete the sentence.

OK.

Regards,
Simon