[PATCH v5 06/41] doc: Explain how to run tests without pytest

[Simon Glass]

Add details about how to run a sandbox test directly, without using
pytest. This is more convenient for rapid development, since it is faster
and allows easier use of a debugger. Also mention sandbox_flattree as an
example of the different sandbox builds available.

Signed-off-by: Simon Glass <sjg at chromium.org>
---

Changes in v5:
- Update based on Heinrich's comments

Changes in v2:
- Put the docs in tests_sandbox since it is more related to sandbox
- Put in a mention of tests_sandbox in the main testing docs

 doc/develop/index.rst         |  1 +
 doc/develop/testing.rst       |  9 ++++
 doc/develop/tests_sandbox.rst | 80 +++++++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+)
 create mode 100644 doc/develop/tests_sandbox.rst

diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index ac57fdb8f30..50b1de3bdff 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -33,3 +33,4 @@ Testing
    coccinelle
    testing
    py_testing
+   tests_sandbox
diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
index f01ca4dc408..87c90eee271 100644
--- a/doc/develop/testing.rst
+++ b/doc/develop/testing.rst
@@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This
 allows test to be executed without needing target hardware. The 'sandbox'
 target provides this feature and it is widely used in tests.
 
+See :doc:`tests_sandbox` for more information.
 
 Pytest Suite
 ------------
@@ -51,8 +52,16 @@ You can run the tests on sandbox with::
 
 This will produce HTML output in build-sandbox/test-log.html
 
+Some tests run with other versions of sandbox. For example sandbox_flattree
+runs the tests with livetree (the hierachical devicetree) disabled. You can
+also select particular tests with -k::
+
+   ./test/py/test.py --bd sandbox_flattree --build -k hello
+
 See test/py/README.md for more information about the pytest suite.
 
+See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
+
 
 tbot
 ----
diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
new file mode 100644
index 00000000000..42b64882cd9
--- /dev/null
+++ b/doc/develop/tests_sandbox.rst
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Tests Under the Hood
+====================
+
+Running sandbox tests directly
+------------------------------
+
+Typically tests are run using the pytest suite. Running pytests on sandbox is
+easy and always gets things right. For example some tests require files to be
+set up before they can work.
+
+But it is also possible to run some sandbox tests directly. For example, this
+runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
+
+   $ ./u-boot -T -c "ut dm gpio"
+
+
+   U-Boot 2021.01
+
+   Model: sandbox
+   DRAM:  128 MiB
+   WDT:   Started with servicing (60s timeout)
+   MMC:   mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
+   In:    serial
+   Out:   vidconsole
+   Err:   vidconsole
+   Model: sandbox
+   SCSI:
+   Net:   eth0: eth at 10002000, eth5: eth at 10003000, eth3: sbe5, eth6: eth at 10004000
+   Test: dm_test_gpio: gpio.c
+   Test: dm_test_gpio: gpio.c (flat tree)
+   Failures: 0
+
+The -T option tells the U-Boot sandbox to run with the 'test' devicetree
+(test.dts) instead of -D which selects the normal sandbox.dts - this is
+necessary because many tests rely on nodes or properties in the test devicetree.
+If you try running tests without -T then you may see failures, like::
+
+   $ ./u-boot -c "ut dm gpio"
+
+
+   U-Boot 2021.01
+
+   DRAM:  128 MiB
+   WDT:   Not found!
+   MMC:
+   In:    serial
+   Out:   serial
+   Err:   serial
+   SCSI:
+   Net:   No ethernet found.
+   Please run with test device tree:
+       ./u-boot -d arch/sandbox/dts/test.dtb
+   Test: dm_test_gpio: gpio.c
+   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
+   Test: dm_test_gpio: gpio.c (flat tree)
+   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
+   Failures: 2
+
+The message above should provide a hint if you forget to use the -T flag. Even
+running with -D will produce different results.
+
+You can easily use gdb on these tests, without needing --gdbserver::
+
+   $ gdb u-boot --args -T -c "ut dm gpio"
+   ...
+   (gdb) break dm_test_gpio
+   Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37.
+   (gdb) run -T -c "ut dm gpio"
+   Starting program: u-boot -T -c "ut dm gpio"
+   Test: dm_test_gpio: gpio.c
+
+   Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>)
+       at files/test/dm/gpio.c:37
+   37		ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio));
+   (gdb)
+
+You can then single-step and look at variables as needed.
+
-- 
2.30.1.766.gb4fecdf3b7-goog