[PATCH v10 3/9] env: Allow U-Boot scripts to be placed in a .env file
Daniel Golle
daniel at makrotopia.org
Fri Nov 12 19:12:47 CET 2021
On Thu, Oct 21, 2021 at 09:08:46PM -0600, Simon Glass wrote:
> At present U-Boot environment variables, and thus scripts, are defined
> by CONFIG_EXTRA_ENV_SETTINGS. It is painful to add large amounts of text
> to this file and dealing with quoting and newlines is harder than it
> should be. It would be better if we could just type the script into a
> text file and have it included by U-Boot.
>
> Add a feature that brings in a .env file associated with the board
> config, if present. To use it, create a file in a board/<vendor>
> directory, typically called <board>.env and controlled by the
> CONFIG_ENV_SOURCE_FILE option.
>
> The environment variables should be of the form "var=value". Values can
> extend to multiple lines. See the README under 'Environment Variables:'
> for more information and an example.
>
> In many cases environment variables need access to the U-Boot CONFIG
> variables to select different options. Enable this so that the environment
> scripts can be as useful as the ones currently in the board config files.
> This uses the C preprocessor, means that comments can be included in the
> environment using /* ... */
>
> Also support += to allow variables to be appended to. This is needed when
> using the preprocessor.
I hope to see this change moving forward!
It would be of great value for use in OpenWrt, as right now a per board
default environment is often included using CONFIG_ENV_SOURCE_FILE and
I was about to convert everything to C precompiler #defines to be more
flexible...
The suggested .env files made possible by this commit would provide an
ideal solution.
>
> Signed-off-by: Simon Glass <sjg at chromium.org>
> Reviewed-by: Marek Behún <marek.behun at nic.cz>
> Tested-by: Marek Behún <marek.behun at nic.cz>
> ---
>
> Changes in v10:
> - Use backslash to allow assignment to a variable ending in +
> - Add rST file into the index
> - Minor tweaks to the script's pattern matching
>
> Changes in v9:
> - Drop mention of other strange characters
> - Clarify that the + restriction is on the variable name not its value
> - Add some tests for the script
> - Deal with leading tabs
> - Squash indentation down to one space
> - Convert newlines within strings to spaces, which seems more consistent
> - Handle appending an empty string to an empty var
>
> Changes in v8:
> - Update commit message to avoid mentioning the 'env' subdirectory
> - Update commit message to mention the + restriction, etc.
> - Overwrite the env file each time, to avoid incremental-build problems
>
> Changes in v7:
> - Use 'env' basename instead of 'environment' for intermediate output files
> - Show a message indicating the source text file being used
> - Give an error if CONFIG_EXTRA_ENV_SETTINGS is also defined
> - Use CONFIG_ENV_SOURCE_FILE instead of rules to specify the text-file name
> - Make board.env the default name if CONFIG_ENV_SOURCE_FILE is empty
> - Rewrite the documentation
> - Drop the use of common.env
> - Update awk script to output the whole CONFIG string, or just a comment
>
> Changes in v6:
> - Combine the two env2string.awk patches into one
>
> Changes in v5:
> - Explain how to include the common.env file
> - Explain why variables starting with _ , and / are not supported
> - Expand the definition of how to declare an environment variable
> - Explain what happens to empty variables
> - Update maintainer
> - Move use of += to this patch
> - Explain that environment variables may not end in +
>
> Changes in v4:
> - Move this from being part of configuring U-Boot to part of building it
> - Don't put the environment in autoconf.mk as it is not needed
> - Add documentation in rST format instead of README
> - Drop mention of import/export
> - Update awk script to ignore blank lines, as generated by clang
> - Add documentation in rST format instead of README
>
> Changes in v3:
> - Adjust Makefile to generate the .inc and .h files in separate fules
> - Add more detail in the README about the format of .env files
> - Improve the comment about " in the awk script
> - Correctly terminate environment files with \n
> - Define __UBOOT_CONFIG__ when collecting environment files
>
> Changes in v2:
> - Move .env file from include/configs to board/
> - Use awk script to process environment since it is much easier on the brain
> - Add information and updated example script to README
> - Add dependency rule so that the environment is rebuilt when it changes
> - Add separate patch to enable C preprocessor for environment files
> - Enable var+=value form to simplify composing variables in multiple steps
>
> MAINTAINERS | 7 +++
> Makefile | 66 ++++++++++++++++++++++-
> config.mk | 2 +
> doc/usage/environment.rst | 81 ++++++++++++++++++++++++++++-
> doc/usage/index.rst | 1 +
> env/Kconfig | 18 +++++++
> env/embedded.c | 1 +
> include/env_default.h | 11 ++++
> scripts/env2string.awk | 80 ++++++++++++++++++++++++++++
> test/py/tests/test_env.py | 107 ++++++++++++++++++++++++++++++++++++++
> 10 files changed, 372 insertions(+), 2 deletions(-)
> create mode 100644 scripts/env2string.awk
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8845c6fd750..8820a0f895e 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -738,6 +738,13 @@ F: test/env/
> F: tools/env*
> F: tools/mkenvimage.c
>
> +ENVIRONMENT AS TEXT
> +M: Simon Glass <sjg at chromium.org>
> +R: Wolfgang Denk <wd at denx.de>
> +S: Maintained
> +F: doc/usage/environment.rst
> +F: scripts/env2string.awk
> +
> FPGA
> M: Michal Simek <michal.simek at xilinx.com>
> S: Maintained
> diff --git a/Makefile b/Makefile
> index 5194e4dc782..a80be71c78a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -517,6 +517,7 @@ version_h := include/generated/version_autogenerated.h
> timestamp_h := include/generated/timestamp_autogenerated.h
> defaultenv_h := include/generated/defaultenv_autogenerated.h
> dt_h := include/generated/dt.h
> +env_h := include/generated/environment.h
>
> no-dot-config-targets := clean clobber mrproper distclean \
> help %docs check% coccicheck \
> @@ -1786,6 +1787,69 @@ quiet_cmd_sym ?= SYM $@
> u-boot.sym: u-boot FORCE
> $(call if_changed,sym)
>
> +# Environment processing
> +# ---------------------------------------------------------------------------
> +
> +# Directory where we expect the .env file, if it exists
> +ENV_DIR := $(srctree)/board/$(BOARDDIR)
> +
> +# Basename of .env file, stripping quotes
> +ENV_SOURCE_FILE := $(CONFIG_ENV_SOURCE_FILE:"%"=%)
> +
> +# Filename of .env file
> +ENV_FILE_CFG := $(ENV_DIR)/$(ENV_SOURCE_FILE).env
> +
> +# Default filename, if CONFIG_ENV_SOURCE_FILE is empty
> +ENV_FILE_BOARD := $(ENV_DIR)/$(CONFIG_SYS_BOARD:"%"=%).env
> +
> +# Select between the CONFIG_ENV_SOURCE_FILE and the default one
> +ENV_FILE := $(if $(ENV_SOURCE_FILE),$(ENV_FILE_CFG),$(wildcard $(ENV_FILE_BOARD)))
> +
> +# Run the environment text file through the preprocessor, but only if it is
> +# non-empty, to save time and possible build errors if something is wonky with
> +# the board
> +quiet_cmd_gen_envp = ENVP $@
> + cmd_gen_envp = \
> + if [ -s "$(ENV_FILE)" ]; then \
> + $(CPP) -P $(CFLAGS) -x assembler-with-cpp -D__ASSEMBLY__ \
> + -D__UBOOT_CONFIG__ \
> + -I . -I include -I $(srctree)/include \
> + -include linux/kconfig.h -include include/config.h \
> + -I$(srctree)/arch/$(ARCH)/include \
> + $< -o $@; \
> + else \
> + echo -n >$@ ; \
> + fi
> +include/generated/env.in: include/generated/env.txt FORCE
> + $(call cmd,gen_envp)
> +
> +# Regenerate the environment if it changes
> +# We use 'wildcard' since the file is not required to exist (at present), in
> +# which case we don't want this dependency, but instead should create an empty
> +# file
> +# This rule is useful since it shows the source file for the environment
> +quiet_cmd_envc = ENVC $@
> + cmd_envc = \
> + if [ -f "$<" ]; then \
> + cat $< > $@; \
> + elif [ -n "$(ENV_SOURCE_FILE)" ]; then \
> + echo "Missing file $(ENV_FILE_CFG)"; \
> + else \
> + echo -n >$@ ; \
> + fi
> +
> +include/generated/env.txt: $(wildcard $(ENV_FILE)) FORCE
> + $(call cmd,envc)
> +
> +# Write out the resulting environment, converted to a C string
> +quiet_cmd_gen_envt = ENVT $@
> + cmd_gen_envt = \
> + awk -f $(srctree)/scripts/env2string.awk $< >$@
> +$(env_h): include/generated/env.in
> + $(call cmd,gen_envt)
> +
> +# ---------------------------------------------------------------------------
> +
> # The actual objects are generated when descending,
> # make sure no implicit rule kicks in
> $(sort $(u-boot-init) $(u-boot-main)): $(u-boot-dirs) ;
> @@ -1841,7 +1905,7 @@ endif
> # prepare2 creates a makefile if using a separate output directory
> prepare2: prepare3 outputmakefile cfg
>
> -prepare1: prepare2 $(version_h) $(timestamp_h) $(dt_h) \
> +prepare1: prepare2 $(version_h) $(timestamp_h) $(dt_h) $(env_h) \
> include/config/auto.conf
> ifeq ($(wildcard $(LDSCRIPT)),)
> @echo >&2 " Could not find linker script."
> diff --git a/config.mk b/config.mk
> index 7bb1fd4ed1b..2595aed218b 100644
> --- a/config.mk
> +++ b/config.mk
> @@ -50,8 +50,10 @@ endif
> ifneq ($(BOARD),)
> ifdef VENDOR
> BOARDDIR = $(VENDOR)/$(BOARD)
> +ENVDIR=${vendor}/env
> else
> BOARDDIR = $(BOARD)
> +ENVDIR=${board}/env
> endif
> endif
> ifdef BOARD
> diff --git a/doc/usage/environment.rst b/doc/usage/environment.rst
> index 7a733b44556..043c02d9a94 100644
> --- a/doc/usage/environment.rst
> +++ b/doc/usage/environment.rst
> @@ -15,7 +15,86 @@ environment is erased by accident, a default environment is provided.
>
> Some configuration options can be set using Environment Variables.
>
> -List of environment variables (most likely not complete):
> +Text-based Environment
> +----------------------
> +
> +The default environment for a board is created using a `.env` environment file
> +using a simple text format. The base filename for this is defined by
> +`CONFIG_ENV_SOURCE_FILE`, or `CONFIG_SYS_BOARD` if that is empty.
> +
> +The file must be in the board directory and have a .env extension, so
> +assuming that there is a board vendor, the resulting filename is therefore::
> +
> + board/<vendor>/<board>/<CONFIG_ENV_SOURCE_FILE>.env
> +
> +or::
> +
> + board/<vendor>/<board>/<CONFIG_SYS_BOARD>.env
> +
> +This is a plain text file where you can type your environment variables in
> +the form `var=value`. Blank lines and multi-line variables are supported.
> +The conversion script looks for a line that starts in column 1 with a string
> +and has an equals sign immediately afterwards. Spaces before the = are not
> +permitted. It is a good idea to indent your scripts so that only the 'var='
> +appears at the start of a line.
> +
> +To add additional text to a variable you can use `var+=value`. This text is
> +merged into the variable during the make process and made available as a
> +single value to U-Boot. Variables can contain `+` characters but in the unlikely
> +event that you want to have a variable name ending in plus, put a backslash
> +before the `+` so that the script knows you are not adding to an existing
> +variable but assigning to a new one::
> +
> + maximum\+=value
> +
> +This file can include C-style comments. Blank lines and multi-line
> +variables are supported, and you can use normal C preprocessor directives
> +and CONFIG defines from your board config also.
> +
> +For example, for snapper9260 you would create a text file called
> +`board/bluewater/snapper9260.env` containing the environment text.
> +
> +Example::
> +
> + stdout=serial
> + #ifdef CONFIG_LCD
> + stdout+=,lcd
> + #endif
> + bootcmd=
> + /* U-Boot script for booting */
> +
> + if [ -z ${tftpserverip} ]; then
> + echo "Use 'setenv tftpserverip a.b.c.d' to set IP address."
> + fi
> +
> + usb start; setenv autoload n; bootp;
> + tftpboot ${tftpserverip}:
> + bootm
> + failed=
> + /* Print a message when boot fails */
> + echo CONFIG_SYS_BOARD boot failed - please check your image
> + echo Load address is CONFIG_SYS_LOAD_ADDR
> +
> +If CONFIG_ENV_SOURCE_FILE is empty and the default filename is not present, then
> +the old-style C environment is used instead. See below.
> +
> +Old-style C environment
> +-----------------------
> +
> +Traditionally, the default environment is created in `include/env_default.h`,
> +and can be augmented by various `CONFIG` defines. See that file for details. In
> +particular you can define `CONFIG_EXTRA_ENV_SETTINGS` in your board file
> +to add environment variables.
> +
> +Board maintainers are encouraged to migrate to the text-based environment as it
> +is easier to maintain. The distro-board script still requires the old-style
> +environment but work is underway to address this.
> +
> +
> +List of environment variables
> +-----------------------------
> +
> +This is most-likely not complete:
>
> baudrate
> see CONFIG_BAUDRATE
> diff --git a/doc/usage/index.rst b/doc/usage/index.rst
> index 356f2a56181..4314112ff34 100644
> --- a/doc/usage/index.rst
> +++ b/doc/usage/index.rst
> @@ -10,6 +10,7 @@ Use U-Boot
> netconsole
> partitions
> cmdline
> + environment
>
> Shell commands
> --------------
> diff --git a/env/Kconfig b/env/Kconfig
> index f75f2b13536..b93ad5c8ee0 100644
> --- a/env/Kconfig
> +++ b/env/Kconfig
> @@ -3,6 +3,24 @@ menu "Environment"
> config ENV_SUPPORT
> def_bool y
>
> +config ENV_SOURCE_FILE
> + string "Environment file to use"
> + default ""
> + help
> + This sets the basename to use to generate the default environment.
> + This a text file as described in doc/usage/environment.rst
> +
> + The file must be in the board directory and have a .env extension, so
> + the resulting filename is typically
> + board/<vendor>/<board>/<CONFIG_ENV_SOURCE_FILE>.env
> +
> + If the file is not present, an error is produced.
> +
> + If this CONFIG is empty, U-Boot uses CONFIG SYS_BOARD as a default, if
> + the file board/<vendor>/<board>/<SYS_BOARD>.env exists. Otherwise the
> + environment is assumed to come from the ad-hoc
> + CONFIG_EXTRA_ENV_SETTINGS #define
> +
> config SAVEENV
> def_bool y if CMD_SAVEENV
>
> diff --git a/env/embedded.c b/env/embedded.c
> index 208553e6af1..9f26e6cad9c 100644
> --- a/env/embedded.c
> +++ b/env/embedded.c
> @@ -66,6 +66,7 @@
> #endif
>
> #define DEFAULT_ENV_INSTANCE_EMBEDDED
> +#include <config.h>
> #include <env_default.h>
>
> #ifdef CONFIG_ENV_ADDR_REDUND
> diff --git a/include/env_default.h b/include/env_default.h
> index 66e203eb6e4..c06506313e5 100644
> --- a/include/env_default.h
> +++ b/include/env_default.h
> @@ -10,6 +10,10 @@
> #include <env_callback.h>
> #include <linux/stringify.h>
>
> +#ifndef USE_HOSTCC
> +#include <generated/environment.h>
> +#endif
> +
> #ifdef DEFAULT_ENV_INSTANCE_EMBEDDED
> env_t embedded_environment __UBOOT_ENV_SECTION__(environment) = {
> ENV_CRC, /* CRC Sum */
> @@ -110,6 +114,13 @@ const uchar default_environment[] = {
> #if defined(CONFIG_BOOTCOUNT_BOOTLIMIT) && (CONFIG_BOOTCOUNT_BOOTLIMIT > 0)
> "bootlimit=" __stringify(CONFIG_BOOTCOUNT_BOOTLIMIT)"\0"
> #endif
> +#ifdef CONFIG_EXTRA_ENV_TEXT
> +# ifdef CONFIG_EXTRA_ENV_SETTINGS
> +# error "Your board uses a text-file environment, so must not define CONFIG_EXTRA_ENV_SETTINGS"
> +# endif
> + /* This is created in the Makefile */
> + CONFIG_EXTRA_ENV_TEXT
> +#endif
> #ifdef CONFIG_EXTRA_ENV_SETTINGS
> CONFIG_EXTRA_ENV_SETTINGS
> #endif
> diff --git a/scripts/env2string.awk b/scripts/env2string.awk
> new file mode 100644
> index 00000000000..57d0fc8f3ba
> --- /dev/null
> +++ b/scripts/env2string.awk
> @@ -0,0 +1,80 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright 2021 Google, Inc
> +#
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Awk script to parse a text file containing an environment and convert it
> +# to a C string which can be compiled into U-Boot.
> +
> +# The resulting output is:
> +#
> +# #define CONFIG_EXTRA_ENV_TEXT "<environment here>"
> +#
> +# If the input is empty, this script outputs a comment instead.
> +
> +BEGIN {
> + # env holds the env variable we are currently processing
> + env = "";
> + ORS = ""
> +}
> +
> +# Skip empty lines, as these are generated by the clang preprocessor
> +NF {
> + # Quote quotes
> + gsub("\"", "\\\"")
> +
> + # Is this the start of a new environment variable?
> + if (match($0, "^([^ \t=][^ =]*)=(.*)$", arr)) {
> + if (length(env) != 0) {
> + # Record the value of the variable now completed
> + vars[var] = env
> + }
> + var = arr[1]
> + env = arr[2]
> +
> + # Deal with += which concatenates the new string to the existing
> + # variable
> + if (length(env) != 0 && match(var, "^(.*)[+]$", var_arr))
> + {
> + # Allow var\+=val to indicate that the variable name is
> + # var+ and this is not actually a concatenation
> + if (substr(var_arr[1], length(var_arr[1])) == "\\") {
> + # Drop the backslash
> + sub(/\\[+]$/, "+", var)
> + } else {
> + var = var_arr[1]
> + env = vars[var] env
> + }
> + }
> + } else {
> + # Change newline to space
> + gsub(/^[ \t]+/, "")
> +
> + # Don't keep leading spaces generated by the previous blank line
> + if (length(env) == 0) {
> + env = $0
> + } else {
> + env = env " " $0
> + }
> + }
> +}
> +
> +END {
> + # Record the value of the variable now completed. If the variable is
> + # empty it is not set.
> + if (length(env) != 0) {
> + vars[var] = env
> + }
> +
> + if (length(vars) != 0) {
> + printf("%s", "#define CONFIG_EXTRA_ENV_TEXT \"")
> +
> + # Print out all the variables
> + for (var in vars) {
> + env = vars[var]
> + print var "=" vars[var] "\\0"
> + }
> + print "\"\n"
> + }
> +}
> diff --git a/test/py/tests/test_env.py b/test/py/tests/test_env.py
> index 9bed2f48d77..f85cb031382 100644
> --- a/test/py/tests/test_env.py
> +++ b/test/py/tests/test_env.py
> @@ -7,6 +7,7 @@
> import os
> import os.path
> from subprocess import call, check_call, CalledProcessError
> +import tempfile
>
> import pytest
> import u_boot_utils
> @@ -515,3 +516,109 @@ def test_env_ext4(state_test_env):
> finally:
> if fs_img:
> call('rm -f %s' % fs_img, shell=True)
> +
> +def test_env_text(u_boot_console):
> + """Test the script that converts the environment to a text file"""
> +
> + def check_script(intext, expect_val):
> + """Check a test case
> +
> + Args:
> + intext: Text to pass to the script
> + expect_val: Expected value of the CONFIG_EXTRA_ENV_TEXT string, or
> + None if we expect it not to be defined
> + """
> + with tempfile.TemporaryDirectory() as path:
> + fname = os.path.join(path, 'infile')
> + with open(fname, 'w') as inf:
> + print(intext, file=inf)
> + result = u_boot_utils.run_and_log(cons, ['awk', '-f', script, fname])
> + if expect_val is not None:
> + expect = '#define CONFIG_EXTRA_ENV_TEXT "%s"\n' % expect_val
> + assert result == expect
> + else:
> + assert result == ''
> +
> + cons = u_boot_console
> + script = os.path.join(cons.config.source_dir, 'scripts', 'env2string.awk')
> +
> + # simple script with a single var
> + check_script('fred=123', 'fred=123\\0')
> +
> + # no vars
> + check_script('', None)
> +
> + # two vars
> + check_script('''fred=123
> +ernie=456''', 'fred=123\\0ernie=456\\0')
> +
> + # blank lines
> + check_script('''fred=123
> +
> +
> +ernie=456
> +
> +''', 'fred=123\\0ernie=456\\0')
> +
> + # append
> + check_script('''fred=123
> +ernie=456
> +fred+= 456''', 'fred=123 456\\0ernie=456\\0')
> +
> + # append from empty
> + check_script('''fred=
> +ernie=456
> +fred+= 456''', 'fred= 456\\0ernie=456\\0')
> +
> + # variable with + in it
> + check_script('fred+ernie=123', 'fred+ernie=123\\0')
> +
> + # ignores variables that are empty
> + check_script('''fred=
> +fred+=
> +ernie=456''', 'ernie=456\\0')
> +
> + # single-character env name
> + check_script('''f=123
> +e=456
> +f+= 456''', 'e=456\\0f=123 456\\0')
> +
> + # contains quotes
> + check_script('''fred="my var"
> +ernie=another"''', 'fred=\\"my var\\"\\0ernie=another\\"\\0')
> +
> + # variable name ending in +
> + check_script('''fred\\+=my var
> +fred++= again''', 'fred+=my var again\\0')
> +
> + # variable name containing +
> + check_script('''fred+jane=both
> +fred+jane+=again
> +ernie=456''', 'fred+jane=bothagain\\0ernie=456\\0')
> +
> + # multi-line vars - new vars always start at column 1
> + check_script('''fred=first
> + second
> +\tthird with tab
> +
> + after blank
> + confusing=oops
> +ernie=another"''', 'fred=first second third with tab after blank confusing=oops\\0ernie=another\\"\\0')
> +
> + # real-world example
> + check_script('''ubifs_boot=
> + env exists bootubipart ||
> + env set bootubipart UBI;
> + env exists bootubivol ||
> + env set bootubivol boot;
> + if ubi part ${bootubipart} &&
> + ubifsmount ubi${devnum}:${bootubivol};
> + then
> + devtype=ubi;
> + run scan_dev_for_boot;
> + fi
> +''',
> + 'ubifs_boot=env exists bootubipart || env set bootubipart UBI; '
> + 'env exists bootubivol || env set bootubivol boot; '
> + 'if ubi part ${bootubipart} && ubifsmount ubi${devnum}:${bootubivol}; '
> + 'then devtype=ubi; run scan_dev_for_boot; fi\\0')
> --
> 2.33.0.1079.g6e70778dc9-goog
>
More information about the U-Boot-Custodians
mailing list