[PATCH v3 05/41] doc: Explain how to run tests without pytest

[Simon Glass]

Add details about how to run a sandbox test directly, without using
pytest. This is more convenient for rapid development, since it is faster
and allows easier use of a debugger. Also mention sandbox_flattree as an
example of the different sandbox builds available.

Signed-off-by: Simon Glass <sjg at chromium.org>
---

(no changes since v2)

Changes in v2:
- Put the docs in tests_sandbox since it is more related to sandbox
- Put in a mention of tests_sandbox in the main testing docs

 doc/develop/index.rst         |  1 +
 doc/develop/testing.rst       |  9 ++++
 doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++
 3 files changed, 89 insertions(+)
 create mode 100644 doc/develop/tests_sandbox.rst

diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index beaa64d8d90..6e59cc47164 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -32,3 +32,4 @@ Testing
    coccinelle
    testing
    py_testing
+   tests_sandbox
diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
index f01ca4dc408..87c90eee271 100644
--- a/doc/develop/testing.rst
+++ b/doc/develop/testing.rst
@@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This
 allows test to be executed without needing target hardware. The 'sandbox'
 target provides this feature and it is widely used in tests.
 
+See :doc:`tests_sandbox` for more information.
 
 Pytest Suite
 ------------
@@ -51,8 +52,16 @@ You can run the tests on sandbox with::
 
 This will produce HTML output in build-sandbox/test-log.html
 
+Some tests run with other versions of sandbox. For example sandbox_flattree
+runs the tests with livetree (the hierachical devicetree) disabled. You can
+also select particular tests with -k::
+
+   ./test/py/test.py --bd sandbox_flattree --build -k hello
+
 See test/py/README.md for more information about the pytest suite.
 
+See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
+
 
 tbot
 ----
diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
new file mode 100644
index 00000000000..85bbd4f6734
--- /dev/null
+++ b/doc/develop/tests_sandbox.rst
@@ -0,0 +1,79 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Tests Under the Hood
+====================
+
+Running sandbox tests directly
+------------------------------
+
+Typically tests are run using the pytest suite. This is easy and always gets
+things right.
+
+But it is also possible to run some sandbox tests directly. For example, this
+runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
+
+   $ ./u-boot -T -c "ut dm gpio"
+
+
+   U-Boot 2021.01
+
+   Model: sandbox
+   DRAM:  128 MiB
+   WDT:   Started with servicing (60s timeout)
+   MMC:   mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
+   In:    serial
+   Out:   vidconsole
+   Err:   vidconsole
+   Model: sandbox
+   SCSI:
+   Net:   eth0: eth at 10002000, eth5: eth at 10003000, eth3: sbe5, eth6: eth at 10004000
+   Test: dm_test_gpio: gpio.c
+   Test: dm_test_gpio: gpio.c (flat tree)
+   Failures: 0
+
+The -T option tells U-Boot to run with the 'test' devicetree (test.dts) instead
+of -D which selects the normal sandbox.dts - this is necessary because many
+tests rely on things in the test devicetree. If you try running tests without
+-T then you may see failures, like::
+
+   $ ./u-boot -c "ut dm gpio"
+
+
+   U-Boot 2021.01
+
+   DRAM:  128 MiB
+   WDT:   Not found!
+   MMC:
+   In:    serial
+   Out:   serial
+   Err:   serial
+   SCSI:
+   Net:   No ethernet found.
+   Please run with test device tree:
+       ./u-boot -d arch/sandbox/dts/test.dtb
+   Test: dm_test_gpio: gpio.c
+   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
+   Test: dm_test_gpio: gpio.c (flat tree)
+   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
+   Failures: 2
+
+The message above should provide a hint if you forget. Even running with -D
+will produce different results.
+
+You can easily use gdb on these tests, without needing --gdbserver::
+
+   $ gdb u-boot --args -T -c "ut dm gpio"
+   ...
+   (gdb) break dm_test_gpio
+   Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37.
+   (gdb) run -T -c "ut dm gpio"
+   Starting program: u-boot -T -c "ut dm gpio"
+   Test: dm_test_gpio: gpio.c
+
+   Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>)
+       at files/test/dm/gpio.c:37
+   37		ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio));
+   (gdb)
+
+You can then single-step and look at variables as needed.
+
-- 
2.30.0.365.g02bc693789-goog