[PATCH RFC] doc: Include pytests directly in documentation
Heinrich Schuchardt
xypron.glpk at gmx.de
Thu Mar 6 17:03:02 CET 2025
On 25.02.25 21:50, Tom Rini wrote:
> As there is support within the Sphinx framework to take Python
> docstrings and generate documentation from it, we can make use of that
> to more clearly and consistently document our pytests and how to
> configure them. To do this we:
> - Add PYTHONPATH to DOC_TARGETS so that modules will be found, have CI
> install pytest module as well.
> - Add the autodoc extension for Python and then redirects so we can move
> py_testing.html (which is heavily referenced in archives, etc).
> - Create a doc/develop/pytest directory and move the existing py_testing
> file there, update references.
> - Update the test_000_version.py test to have a docstring
> - Move the imports in test_net_boot.py so the docstring is in the
> correct place.
> - Use code-block for all of the code blocks in test_net_boot.py so they
> are rendered correctly in documentation.
> - Adding these code-block lines makes the diff larger when read but a
> "git show -w" for example makes it clearer that the textual changes
> are removing '#' from lines so the output reads nicer.
>
> Signed-off-by: Tom Rini <trini at konsulko.com>
Some of the test functions seem to lack documentation:
test_net_boot.setup_networking()
test_net_boot.setup_tftpboot_boot()
Would you know what 000 in the following filename corresponds to?
test/py/tests/test_000_version.py
Maybe we should rename to test_version.py it before adding this patch.
Tested-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
Acked-by: Heinrich Schuchardt <xypron.glpk at gmx.de>
> ---
> Cc: Heinrich Schuchardt <xypron.glpk at gmx.de>
> Cc: Simon Glass <sjg at chromium.org>
>
> As this is a RFC, I didn't split it up as much as it needs to be as I'm
> looking for some feedback on the concept.
> ---
> .azure-pipelines.yml | 2 +-
> .gitlab-ci.yml | 2 +-
> Makefile | 3 +-
> doc/conf.py | 8 +-
> doc/develop/ci_testing.rst | 2 +-
> doc/develop/index.rst | 2 +-
> doc/develop/pytest/index.rst | 21 +++
> doc/develop/pytest/test_000_version.rst | 8 +
> doc/develop/pytest/test_net_boot.rst | 8 +
> .../{py_testing.rst => pytest/usage.rst} | 2 +-
> doc/develop/testing.rst | 4 +-
> doc/develop/tests_writing.rst | 2 +-
> doc/sphinx/requirements.txt | 1 +
> test/py/tests/test_000_version.py | 10 +-
> test/py/tests/test_net_boot.py | 163 ++++++++++--------
> 15 files changed, 150 insertions(+), 88 deletions(-)
> create mode 100644 doc/develop/pytest/index.rst
> create mode 100644 doc/develop/pytest/test_000_version.rst
> create mode 100644 doc/develop/pytest/test_net_boot.rst
> rename doc/develop/{py_testing.rst => pytest/usage.rst} (99%)
>
> diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
> index d5cfa59a8a61..e401e5cf99b4 100644
> --- a/.azure-pipelines.yml
> +++ b/.azure-pipelines.yml
> @@ -92,7 +92,7 @@ stages:
> set -e
> virtualenv -p /usr/bin/python3 /tmp/venvhtml
> . /tmp/venvhtml/bin/activate
> - pip install -r doc/sphinx/requirements.txt
> + pip install -r doc/sphinx/requirements.txt pytest
> make htmldocs KDOC_WERROR=1
> make infodocs
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 64f92f8b5b9d..083f67695081 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -156,7 +156,7 @@ docs:
> script:
> - virtualenv -p /usr/bin/python3 /tmp/venvhtml
> - . /tmp/venvhtml/bin/activate
> - - pip install -r doc/sphinx/requirements.txt
> + - pip install -r doc/sphinx/requirements.txt pytest
> - make htmldocs KDOC_WERROR=1
> - make infodocs
>
> diff --git a/Makefile b/Makefile
> index 93fa86a18705..e2e452ae4aa0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2445,7 +2445,8 @@ DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
> linkcheckdocs dochelp refcheckdocs texinfodocs infodocs
> PHONY += $(DOC_TARGETS)
> $(DOC_TARGETS): scripts_basic FORCE
> - $(Q)$(MAKE) $(build)=doc $@
> + $(Q)PYTHONPATH=$(srctree)/test/py/tests:$(srctree)/test/py \
> + $(MAKE) $(build)=doc $@
>
> PHONY += checkstack ubootrelease ubootversion
>
> diff --git a/doc/conf.py b/doc/conf.py
> index c50daf874a53..a935d674aae9 100644
> --- a/doc/conf.py
> +++ b/doc/conf.py
> @@ -47,7 +47,8 @@ needs_sphinx = '2.4.4'
> extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include',
> 'kfigure', 'sphinx.ext.ifconfig', # 'automarkup',
> 'maintainers_include', 'sphinx.ext.autosectionlabel',
> - 'kernel_abi', 'kernel_feat', 'sphinx-prompt']
> + 'kernel_abi', 'kernel_feat', 'sphinx-prompt',
> + 'sphinx.ext.autodoc', 'sphinx_reredirects' ]
>
> #
> # cdomain is badly broken in Sphinx 3+. Leaving it out generates *most*
> @@ -148,6 +149,11 @@ project = 'Das U-Boot'
> copyright = 'The U-Boot development community'
> author = 'The U-Boot development community'
>
> +# Pages we have moved after being heavily referenced externally
> +redirects = {
> + "develop/py_testing": "pytest/usage.html"
> +}
> +
> # The version info for the project you're documenting, acts as replacement for
> # |version| and |release|, also used in various other places throughout the
> # built documents.
> diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
> index ffaacedc3d88..2033e3447e9e 100644
> --- a/doc/develop/ci_testing.rst
> +++ b/doc/develop/ci_testing.rst
> @@ -18,7 +18,7 @@ be built with mingw to run on Windows.
> Each of the pipelines is written in such as way as to be a "world build" style
> test and as such we try and build all possible platforms. In addition, for all
> platforms that support being run in QEMU we run them in QEMU and use our pytest
> -suite. See :doc:`py_testing` for more information about those tests.
> +suite. See :doc:`pytest/usage` for more information about those tests.
>
> Azure Pipelines
> ---------------
> diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> index d9f2a8382077..131e5fb3188f 100644
> --- a/doc/develop/index.rst
> +++ b/doc/develop/index.rst
> @@ -81,7 +81,7 @@ Testing
> :maxdepth: 1
>
> testing
> - py_testing
> + pytest/index
> tests_writing
> tests_sandbox
> binman_tests
> diff --git a/doc/develop/pytest/index.rst b/doc/develop/pytest/index.rst
> new file mode 100644
> index 000000000000..36083882584f
> --- /dev/null
> +++ b/doc/develop/pytest/index.rst
> @@ -0,0 +1,21 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +pytest Framework within U-Boot
> +==============================
> +
> +General
> +-------
> +
> +.. toctree::
> + :maxdepth: 1
> +
> + usage
> +
> +Individual tests
> +----------------
> +
> +.. toctree::
> + :maxdepth: 1
> +
> + test_000_version
> + test_net_boot
> diff --git a/doc/develop/pytest/test_000_version.rst b/doc/develop/pytest/test_000_version.rst
> new file mode 100644
> index 000000000000..09e9a361729b
> --- /dev/null
> +++ b/doc/develop/pytest/test_000_version.rst
> @@ -0,0 +1,8 @@
> +test_000_version
> +================
> +
> +.. automodule:: test_000_version
> + :synopsis:
> + :member-order: bysource
> + :members:
> + :undoc-members:
> diff --git a/doc/develop/pytest/test_net_boot.rst b/doc/develop/pytest/test_net_boot.rst
> new file mode 100644
> index 000000000000..369bd3b809cf
> --- /dev/null
> +++ b/doc/develop/pytest/test_net_boot.rst
> @@ -0,0 +1,8 @@
> +test_net_boot
> +=============
> +
> +.. automodule:: test_net_boot
> + :synopsis:
> + :member-order: bysource
> + :members:
> + :undoc-members:
> diff --git a/doc/develop/py_testing.rst b/doc/develop/pytest/usage.rst
> similarity index 99%
> rename from doc/develop/py_testing.rst
> rename to doc/develop/pytest/usage.rst
> index b50473039be4..0b0038ba1899 100644
> --- a/doc/develop/py_testing.rst
> +++ b/doc/develop/pytest/usage.rst
> @@ -13,7 +13,7 @@ results. Advantages of this approach are:
> U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
> It is asserted that writing test-related code in Python is simpler and more
> - flexible than writing it all in C. But see :doc:`tests_writing` for caveats
> + flexible than writing it all in C. But see :doc:`../tests_writing` for caveats
> and more discussion / analysis.
> - It is reasonably simple to interact with U-Boot in this way.
>
> diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
> index 9114d11ad26d..aa7786c99fd5 100644
> --- a/doc/develop/testing.rst
> +++ b/doc/develop/testing.rst
> @@ -33,7 +33,7 @@ You can also run a selection tests in parallel with::
> make pcheck
>
> All of the above use the test/run script with a paremeter to select which tests
> -are run. See :doc:`py_testing` for more information.
> +are run. See :doc:`pytest/usage` for more information.
>
>
> Sandbox
> @@ -69,7 +69,7 @@ build::
>
> ./test/py/test.py --bd sandbox_spl --build -k test_spl
>
> -See :doc:`py_testing` for more information about the pytest suite.
> +See :doc:`pytest/usage` for more information about the pytest suite.
>
> See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
>
> diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst
> index 5f3c43d5da24..2adcfb072099 100644
> --- a/doc/develop/tests_writing.rst
> +++ b/doc/develop/tests_writing.rst
> @@ -376,5 +376,5 @@ An example SPL test is spl_test_load().
> Writing Python tests
> --------------------
>
> -See :doc:`py_testing` for brief notes how to write Python tests. You
> +See :doc:`pytest/usage` for brief notes how to write Python tests. You
> should be able to use the existing tests in test/py/tests as examples.
> diff --git a/doc/sphinx/requirements.txt b/doc/sphinx/requirements.txt
> index 13e0327c0f67..12c5772684dc 100644
> --- a/doc/sphinx/requirements.txt
> +++ b/doc/sphinx/requirements.txt
> @@ -15,6 +15,7 @@ six==1.16.0
> snowballstemmer==2.2.0
> Sphinx==8.1.3
> sphinx-prompt==1.9.0
> +sphinx-reredirects==0.1.5
> sphinx-rtd-theme==3.0.1
> sphinxcontrib-applehelp==2.0.0
> sphinxcontrib-devhelp==2.0.0
> diff --git a/test/py/tests/test_000_version.py b/test/py/tests/test_000_version.py
> index bd089ab5439d..77effbbc38f4 100644
> --- a/test/py/tests/test_000_version.py
> +++ b/test/py/tests/test_000_version.py
> @@ -2,10 +2,12 @@
> # Copyright (c) 2015 Stephen Warren
> # Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
>
> -# pytest runs tests the order of their module path, which is related to the
> -# filename containing the test. This file is named such that it is sorted
> -# first, simply as a very basic sanity check of the functionality of the U-Boot
> -# command prompt.
> +"""
> +pytest runs tests the order of their module path, which is related to the
> +filename containing the test. This file is named such that it is sorted
> +first, simply as a very basic sanity check of the functionality of the U-Boot
> +command prompt.
> +"""
>
> def test_version(u_boot_console):
> """Test that the "version" command prints the U-Boot version."""
> diff --git a/test/py/tests/test_net_boot.py b/test/py/tests/test_net_boot.py
> index d7d74356928e..68a91ef5b518 100644
> --- a/test/py/tests/test_net_boot.py
> +++ b/test/py/tests/test_net_boot.py
> @@ -1,11 +1,6 @@
> # SPDX-License-Identifier: GPL-2.0
> # (C) Copyright 2023, Advanced Micro Devices, Inc.
>
> -import pytest
> -import u_boot_utils
> -import test_net
> -import re
> -
> """
> Note: This test relies on boardenv_* containing configuration values to define
> which the network environment available for testing. Without this, this test
> @@ -13,77 +8,88 @@ will be automatically skipped.
>
> For example:
>
> -# Details regarding a boot image file that may be read from a TFTP server. This
> -# variable may be omitted or set to None if TFTP boot testing is not possible
> -# or desired.
> -env__net_tftp_bootable_file = {
> - 'fn': 'image.ub',
> - 'addr': 0x10000000,
> - 'size': 5058624,
> - 'crc32': 'c2244b26',
> - 'pattern': 'Linux',
> - 'config': 'config at 2',
> - 'timeout': 50000,
> - 'check_type': 'boot_error',
> - 'check_pattern': 'ERROR',
> -}
> -
> -# False or omitted if a TFTP boot test should be tested.
> -# If TFTP boot testing is not possible or desired, set this variable to True.
> -# For example: If FIT image is not proper to boot
> -env__tftp_boot_test_skip = False
> -
> -# Here is the example of FIT image configurations:
> -configurations {
> - default = "config at 1";
> - config at 1 {
> - description = "Boot Linux kernel with config at 1";
> - kernel = "kernel at 0";
> - fdt = "fdt at 0";
> - ramdisk = "ramdisk at 0";
> - hash at 1 {
> - algo = "sha1";
> - };
> - };
> - config at 2 {
> - description = "Boot Linux kernel with config at 2";
> - kernel = "kernel at 1";
> - fdt = "fdt at 1";
> - ramdisk = "ramdisk at 1";
> - hash at 1 {
> - algo = "sha1";
> - };
> - };
> -};
> -
> -# Details regarding a file that may be read from a TFTP server. This variable
> -# may be omitted or set to None if PXE testing is not possible or desired.
> -env__net_pxe_bootable_file = {
> - 'fn': 'default',
> - 'addr': 0x10000000,
> - 'size': 74,
> - 'timeout': 50000,
> - 'pattern': 'Linux',
> - 'valid_label': '1',
> - 'invalid_label': '2',
> - 'exp_str_invalid': 'Skipping install for failure retrieving',
> - 'local_label': '3',
> - 'exp_str_local': 'missing environment variable: localcmd',
> - 'empty_label': '4',
> - 'exp_str_empty': 'No kernel given, skipping boot',
> - 'check_type': 'boot_error',
> - 'check_pattern': 'ERROR',
> -}
> -
> -# False if a PXE boot test should be tested.
> -# If PXE boot testing is not possible or desired, set this variable to True.
> -# For example: If pxe configuration file is not proper to boot
> -env__pxe_boot_test_skip = False
> -
> -# Here is the example of pxe configuration file ordered based on the execution
> -# flow:
> +.. code-block:: python
> +
> + # Details regarding a boot image file that may be read from a TFTP server. This
> + # variable may be omitted or set to None if TFTP boot testing is not possible
> + # or desired.
> + env__net_tftp_bootable_file = {
> + 'fn': 'image.ub',
> + 'addr': 0x10000000,
> + 'size': 5058624,
> + 'crc32': 'c2244b26',
> + 'pattern': 'Linux',
> + 'config': 'config at 2',
> + 'timeout': 50000,
> + 'check_type': 'boot_error',
> + 'check_pattern': 'ERROR',
> + }
> +
> + # False or omitted if a TFTP boot test should be tested.
> + # If TFTP boot testing is not possible or desired, set this variable to True.
> + # For example: If FIT image is not proper to boot
> + env__tftp_boot_test_skip = False
> +
> +
> +Here is the example of FIT image configurations:
> +
> +.. code-block:: devicetree
> +
> + configurations {
> + default = "config at 1";
> + config at 1 {
> + description = "Boot Linux kernel with config at 1";
> + kernel = "kernel at 0";
> + fdt = "fdt at 0";
> + ramdisk = "ramdisk at 0";
> + hash at 1 {
> + algo = "sha1";
> + };
> + };
> + config at 2 {
> + description = "Boot Linux kernel with config at 2";
> + kernel = "kernel at 1";
> + fdt = "fdt at 1";
> + ramdisk = "ramdisk at 1";
> + hash at 1 {
> + algo = "sha1";
> + };
> + };
> + };
> +
> +.. code-block:: python
> +
> + # Details regarding a file that may be read from a TFTP server. This variable
> + # may be omitted or set to None if PXE testing is not possible or desired.
> + env__net_pxe_bootable_file = {
> + 'fn': 'default',
> + 'addr': 0x10000000,
> + 'size': 74,
> + 'timeout': 50000,
> + 'pattern': 'Linux',
> + 'valid_label': '1',
> + 'invalid_label': '2',
> + 'exp_str_invalid': 'Skipping install for failure retrieving',
> + 'local_label': '3',
> + 'exp_str_local': 'missing environment variable: localcmd',
> + 'empty_label': '4',
> + 'exp_str_empty': 'No kernel given, skipping boot',
> + 'check_type': 'boot_error',
> + 'check_pattern': 'ERROR',
> + }
> +
> + # False if a PXE boot test should be tested.
> + # If PXE boot testing is not possible or desired, set this variable to True.
> + # For example: If pxe configuration file is not proper to boot
> + env__pxe_boot_test_skip = False
> +
> +Here is the example of pxe configuration file ordered based on the execution
> +flow:
> +
> 1) /tftpboot/pxelinux.cfg/default-arm-zynqmp
>
> +.. code-block::
> +
> menu include pxelinux.cfg/default-arm
> timeout 50
>
> @@ -91,6 +97,8 @@ env__pxe_boot_test_skip = False
>
> 2) /tftpboot/pxelinux.cfg/default-arm
>
> +.. code-block::
> +
> menu title Linux boot selections
> menu include pxelinux.cfg/default
>
> @@ -110,6 +118,8 @@ env__pxe_boot_test_skip = False
>
> 3) /tftpboot/pxelinux.cfg/default
>
> +.. code-block::
> +
> label Linux
> menu label Boot kernel
> kernel Image
> @@ -117,6 +127,11 @@ env__pxe_boot_test_skip = False
> initrd rootfs.cpio.gz.u-boot
> """
>
> +import pytest
> +import u_boot_utils
> +import test_net
> +import re
> +
> def setup_networking(u_boot_console):
> test_net.test_net_dhcp(u_boot_console)
> if not test_net.net_set_up:
More information about the U-Boot
mailing list