[PATCH v10 3/9] env: Allow U-Boot scripts to be placed in a .env file

Daniel Golle daniel at makrotopia.org
Fri Nov 12 19:12:47 CET 2021


On Thu, Oct 21, 2021 at 09:08:46PM -0600, Simon Glass wrote:
> At present U-Boot environment variables, and thus scripts, are defined
> by CONFIG_EXTRA_ENV_SETTINGS. It is painful to add large amounts of text
> to this file and dealing with quoting and newlines is harder than it
> should be. It would be better if we could just type the script into a
> text file and have it included by U-Boot.
> 
> Add a feature that brings in a .env file associated with the board
> config, if present. To use it, create a file in a board/<vendor>
> directory, typically called <board>.env and controlled by the
> CONFIG_ENV_SOURCE_FILE option.
> 
> The environment variables should be of the form "var=value". Values can
> extend to multiple lines. See the README under 'Environment Variables:'
> for more information and an example.
> 
> In many cases environment variables need access to the U-Boot CONFIG
> variables to select different options. Enable this so that the environment
> scripts can be as useful as the ones currently in the board config files.
> This uses the C preprocessor, means that comments can be included in the
> environment using /* ... */
> 
> Also support += to allow variables to be appended to. This is needed when
> using the preprocessor.

I hope to see this change moving forward!
It would be of great value for use in OpenWrt, as right now a per board
default environment is often included using CONFIG_ENV_SOURCE_FILE and
I was about to convert everything to C precompiler #defines to be more
flexible...
The suggested .env files made possible by this commit would provide an
ideal solution.


> 
> Signed-off-by: Simon Glass <sjg at chromium.org>
> Reviewed-by: Marek Behún <marek.behun at nic.cz>
> Tested-by: Marek Behún <marek.behun at nic.cz>
> ---
> 
> Changes in v10:
> - Use backslash to allow assignment to a variable ending in +
> - Add rST file into the index
> - Minor tweaks to the script's pattern matching
> 
> Changes in v9:
> - Drop mention of other strange characters
> - Clarify that the + restriction is on the variable name not its value
> - Add some tests for the script
> - Deal with leading tabs
> - Squash indentation down to one space
> - Convert newlines within strings to spaces, which seems more consistent
> - Handle appending an empty string to an empty var
> 
> Changes in v8:
> - Update commit message to avoid mentioning the 'env' subdirectory
> - Update commit message to mention the + restriction, etc.
> - Overwrite the env file each time, to avoid incremental-build problems
> 
> Changes in v7:
> - Use 'env' basename instead of 'environment' for intermediate output files
> - Show a message indicating the source text file being used
> - Give an error if CONFIG_EXTRA_ENV_SETTINGS is also defined
> - Use CONFIG_ENV_SOURCE_FILE instead of rules to specify the text-file name
> - Make board.env the default name if CONFIG_ENV_SOURCE_FILE is empty
> - Rewrite the documentation
> - Drop the use of common.env
> - Update awk script to output the whole CONFIG string, or just a comment
> 
> Changes in v6:
> - Combine the two env2string.awk patches into one
> 
> Changes in v5:
> - Explain how to include the common.env file
> - Explain why variables starting with _ , and / are not supported
> - Expand the definition of how to declare an environment variable
> - Explain what happens to empty variables
> - Update maintainer
> - Move use of += to this patch
> - Explain that environment variables may not end in +
> 
> Changes in v4:
> - Move this from being part of configuring U-Boot to part of building it
> - Don't put the environment in autoconf.mk as it is not needed
> - Add documentation in rST format instead of README
> - Drop mention of import/export
> - Update awk script to ignore blank lines, as generated by clang
> - Add documentation in rST format instead of README
> 
> Changes in v3:
> - Adjust Makefile to generate the .inc and .h files in separate fules
> - Add more detail in the README about the format of .env files
> - Improve the comment about " in the awk script
> - Correctly terminate environment files with \n
> - Define __UBOOT_CONFIG__ when collecting environment files
> 
> Changes in v2:
> - Move .env file from include/configs to board/
> - Use awk script to process environment since it is much easier on the brain
> - Add information and updated example script to README
> - Add dependency rule so that the environment is rebuilt when it changes
> - Add separate patch to enable C preprocessor for environment files
> - Enable var+=value form to simplify composing variables in multiple steps
> 
>  MAINTAINERS               |   7 +++
>  Makefile                  |  66 ++++++++++++++++++++++-
>  config.mk                 |   2 +
>  doc/usage/environment.rst |  81 ++++++++++++++++++++++++++++-
>  doc/usage/index.rst       |   1 +
>  env/Kconfig               |  18 +++++++
>  env/embedded.c            |   1 +
>  include/env_default.h     |  11 ++++
>  scripts/env2string.awk    |  80 ++++++++++++++++++++++++++++
>  test/py/tests/test_env.py | 107 ++++++++++++++++++++++++++++++++++++++
>  10 files changed, 372 insertions(+), 2 deletions(-)
>  create mode 100644 scripts/env2string.awk
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8845c6fd750..8820a0f895e 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -738,6 +738,13 @@ F:	test/env/
>  F:	tools/env*
>  F:	tools/mkenvimage.c
>  
> +ENVIRONMENT AS TEXT
> +M:	Simon Glass <sjg at chromium.org>
> +R:	Wolfgang Denk <wd at denx.de>
> +S:	Maintained
> +F:	doc/usage/environment.rst
> +F:	scripts/env2string.awk
> +
>  FPGA
>  M:	Michal Simek <michal.simek at xilinx.com>
>  S:	Maintained
> diff --git a/Makefile b/Makefile
> index 5194e4dc782..a80be71c78a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -517,6 +517,7 @@ version_h := include/generated/version_autogenerated.h
>  timestamp_h := include/generated/timestamp_autogenerated.h
>  defaultenv_h := include/generated/defaultenv_autogenerated.h
>  dt_h := include/generated/dt.h
> +env_h := include/generated/environment.h
>  
>  no-dot-config-targets := clean clobber mrproper distclean \
>  			 help %docs check% coccicheck \
> @@ -1786,6 +1787,69 @@ quiet_cmd_sym ?= SYM     $@
>  u-boot.sym: u-boot FORCE
>  	$(call if_changed,sym)
>  
> +# Environment processing
> +# ---------------------------------------------------------------------------
> +
> +# Directory where we expect the .env file, if it exists
> +ENV_DIR := $(srctree)/board/$(BOARDDIR)
> +
> +# Basename of .env file, stripping quotes
> +ENV_SOURCE_FILE := $(CONFIG_ENV_SOURCE_FILE:"%"=%)
> +
> +# Filename of .env file
> +ENV_FILE_CFG := $(ENV_DIR)/$(ENV_SOURCE_FILE).env
> +
> +# Default filename, if CONFIG_ENV_SOURCE_FILE is empty
> +ENV_FILE_BOARD := $(ENV_DIR)/$(CONFIG_SYS_BOARD:"%"=%).env
> +
> +# Select between the CONFIG_ENV_SOURCE_FILE and the default one
> +ENV_FILE := $(if $(ENV_SOURCE_FILE),$(ENV_FILE_CFG),$(wildcard $(ENV_FILE_BOARD)))
> +
> +# Run the environment text file through the preprocessor, but only if it is
> +# non-empty, to save time and possible build errors if something is wonky with
> +# the board
> +quiet_cmd_gen_envp = ENVP    $@
> +      cmd_gen_envp = \
> +	if [ -s "$(ENV_FILE)" ]; then \
> +		$(CPP) -P $(CFLAGS) -x assembler-with-cpp -D__ASSEMBLY__ \
> +			-D__UBOOT_CONFIG__ \
> +			-I . -I include -I $(srctree)/include \
> +			-include linux/kconfig.h -include include/config.h \
> +			-I$(srctree)/arch/$(ARCH)/include \
> +			$< -o $@; \
> +	else \
> +		echo -n >$@ ; \
> +	fi
> +include/generated/env.in: include/generated/env.txt FORCE
> +	$(call cmd,gen_envp)
> +
> +# Regenerate the environment if it changes
> +# We use 'wildcard' since the file is not required to exist (at present), in
> +# which case we don't want this dependency, but instead should create an empty
> +# file
> +# This rule is useful since it shows the source file for the environment
> +quiet_cmd_envc = ENVC    $@
> +      cmd_envc = \
> +	if [ -f "$<" ]; then \
> +		cat $< > $@; \
> +	elif [ -n "$(ENV_SOURCE_FILE)" ]; then \
> +		echo "Missing file $(ENV_FILE_CFG)"; \
> +	else \
> +		echo -n >$@ ; \
> +	fi
> +
> +include/generated/env.txt: $(wildcard $(ENV_FILE)) FORCE
> +	$(call cmd,envc)
> +
> +# Write out the resulting environment, converted to a C string
> +quiet_cmd_gen_envt = ENVT    $@
> +      cmd_gen_envt = \
> +	awk -f $(srctree)/scripts/env2string.awk $< >$@
> +$(env_h): include/generated/env.in
> +	$(call cmd,gen_envt)
> +
> +# ---------------------------------------------------------------------------
> +
>  # The actual objects are generated when descending,
>  # make sure no implicit rule kicks in
>  $(sort $(u-boot-init) $(u-boot-main)): $(u-boot-dirs) ;
> @@ -1841,7 +1905,7 @@ endif
>  # prepare2 creates a makefile if using a separate output directory
>  prepare2: prepare3 outputmakefile cfg
>  
> -prepare1: prepare2 $(version_h) $(timestamp_h) $(dt_h) \
> +prepare1: prepare2 $(version_h) $(timestamp_h) $(dt_h) $(env_h) \
>                     include/config/auto.conf
>  ifeq ($(wildcard $(LDSCRIPT)),)
>  	@echo >&2 "  Could not find linker script."
> diff --git a/config.mk b/config.mk
> index 7bb1fd4ed1b..2595aed218b 100644
> --- a/config.mk
> +++ b/config.mk
> @@ -50,8 +50,10 @@ endif
>  ifneq ($(BOARD),)
>  ifdef	VENDOR
>  BOARDDIR = $(VENDOR)/$(BOARD)
> +ENVDIR=${vendor}/env
>  else
>  BOARDDIR = $(BOARD)
> +ENVDIR=${board}/env
>  endif
>  endif
>  ifdef	BOARD
> diff --git a/doc/usage/environment.rst b/doc/usage/environment.rst
> index 7a733b44556..043c02d9a94 100644
> --- a/doc/usage/environment.rst
> +++ b/doc/usage/environment.rst
> @@ -15,7 +15,86 @@ environment is erased by accident, a default environment is provided.
>  
>  Some configuration options can be set using Environment Variables.
>  
> -List of environment variables (most likely not complete):
> +Text-based Environment
> +----------------------
> +
> +The default environment for a board is created using a `.env` environment file
> +using a simple text format. The base filename for this is defined by
> +`CONFIG_ENV_SOURCE_FILE`, or `CONFIG_SYS_BOARD` if that is empty.
> +
> +The file must be in the board directory and have a .env extension, so
> +assuming that there is a board vendor, the resulting filename is therefore::
> +
> +   board/<vendor>/<board>/<CONFIG_ENV_SOURCE_FILE>.env
> +
> +or::
> +
> +   board/<vendor>/<board>/<CONFIG_SYS_BOARD>.env
> +
> +This is a plain text file where you can type your environment variables in
> +the form `var=value`. Blank lines and multi-line variables are supported.
> +The conversion script looks for a line that starts in column 1 with a string
> +and has an equals sign immediately afterwards. Spaces before the = are not
> +permitted. It is a good idea to indent your scripts so that only the 'var='
> +appears at the start of a line.
> +
> +To add additional text to a variable you can use `var+=value`. This text is
> +merged into the variable during the make process and made available as a
> +single value to U-Boot. Variables can contain `+` characters but in the unlikely
> +event that you want to have a variable name ending in plus, put a backslash
> +before the `+` so that the script knows you are not adding to an existing
> +variable but assigning to a new one::
> +
> +    maximum\+=value
> +
> +This file can include C-style comments. Blank lines and multi-line
> +variables are supported, and you can use normal C preprocessor directives
> +and CONFIG defines from your board config also.
> +
> +For example, for snapper9260 you would create a text file called
> +`board/bluewater/snapper9260.env` containing the environment text.
> +
> +Example::
> +
> +    stdout=serial
> +    #ifdef CONFIG_LCD
> +    stdout+=,lcd
> +    #endif
> +    bootcmd=
> +        /* U-Boot script for booting */
> +
> +        if [ -z ${tftpserverip} ]; then
> +            echo "Use 'setenv tftpserverip a.b.c.d' to set IP address."
> +        fi
> +
> +        usb start; setenv autoload n; bootp;
> +        tftpboot ${tftpserverip}:
> +        bootm
> +    failed=
> +        /* Print a message when boot fails */
> +        echo CONFIG_SYS_BOARD boot failed - please check your image
> +        echo Load address is CONFIG_SYS_LOAD_ADDR
> +
> +If CONFIG_ENV_SOURCE_FILE is empty and the default filename is not present, then
> +the old-style C environment is used instead. See below.
> +
> +Old-style C environment
> +-----------------------
> +
> +Traditionally, the default environment is created in `include/env_default.h`,
> +and can be augmented by various `CONFIG` defines. See that file for details. In
> +particular you can define `CONFIG_EXTRA_ENV_SETTINGS` in your board file
> +to add environment variables.
> +
> +Board maintainers are encouraged to migrate to the text-based environment as it
> +is easier to maintain. The distro-board script still requires the old-style
> +environment but work is underway to address this.
> +
> +
> +List of environment variables
> +-----------------------------
> +
> +This is most-likely not complete:
>  
>  baudrate
>      see CONFIG_BAUDRATE
> diff --git a/doc/usage/index.rst b/doc/usage/index.rst
> index 356f2a56181..4314112ff34 100644
> --- a/doc/usage/index.rst
> +++ b/doc/usage/index.rst
> @@ -10,6 +10,7 @@ Use U-Boot
>     netconsole
>     partitions
>     cmdline
> +   environment
>  
>  Shell commands
>  --------------
> diff --git a/env/Kconfig b/env/Kconfig
> index f75f2b13536..b93ad5c8ee0 100644
> --- a/env/Kconfig
> +++ b/env/Kconfig
> @@ -3,6 +3,24 @@ menu "Environment"
>  config ENV_SUPPORT
>  	def_bool y
>  
> +config ENV_SOURCE_FILE
> +	string "Environment file to use"
> +	default ""
> +	help
> +	  This sets the basename to use to generate the default environment.
> +	  This a text file as described in doc/usage/environment.rst
> +
> +	  The file must be in the board directory and have a .env extension, so
> +	  the resulting filename is typically
> +	  board/<vendor>/<board>/<CONFIG_ENV_SOURCE_FILE>.env
> +
> +	  If the file is not present, an error is produced.
> +
> +	  If this CONFIG is empty, U-Boot uses CONFIG SYS_BOARD as a default, if
> +	  the file board/<vendor>/<board>/<SYS_BOARD>.env exists. Otherwise the
> +	  environment is assumed to come from the ad-hoc
> +	  CONFIG_EXTRA_ENV_SETTINGS #define
> +
>  config SAVEENV
>  	def_bool y if CMD_SAVEENV
>  
> diff --git a/env/embedded.c b/env/embedded.c
> index 208553e6af1..9f26e6cad9c 100644
> --- a/env/embedded.c
> +++ b/env/embedded.c
> @@ -66,6 +66,7 @@
>  #endif
>  
>  #define DEFAULT_ENV_INSTANCE_EMBEDDED
> +#include <config.h>
>  #include <env_default.h>
>  
>  #ifdef CONFIG_ENV_ADDR_REDUND
> diff --git a/include/env_default.h b/include/env_default.h
> index 66e203eb6e4..c06506313e5 100644
> --- a/include/env_default.h
> +++ b/include/env_default.h
> @@ -10,6 +10,10 @@
>  #include <env_callback.h>
>  #include <linux/stringify.h>
>  
> +#ifndef USE_HOSTCC
> +#include <generated/environment.h>
> +#endif
> +
>  #ifdef DEFAULT_ENV_INSTANCE_EMBEDDED
>  env_t embedded_environment __UBOOT_ENV_SECTION__(environment) = {
>  	ENV_CRC,	/* CRC Sum */
> @@ -110,6 +114,13 @@ const uchar default_environment[] = {
>  #if defined(CONFIG_BOOTCOUNT_BOOTLIMIT) && (CONFIG_BOOTCOUNT_BOOTLIMIT > 0)
>  	"bootlimit="	__stringify(CONFIG_BOOTCOUNT_BOOTLIMIT)"\0"
>  #endif
> +#ifdef CONFIG_EXTRA_ENV_TEXT
> +# ifdef CONFIG_EXTRA_ENV_SETTINGS
> +# error "Your board uses a text-file environment, so must not define CONFIG_EXTRA_ENV_SETTINGS"
> +# endif
> +	/* This is created in the Makefile */
> +	CONFIG_EXTRA_ENV_TEXT
> +#endif
>  #ifdef	CONFIG_EXTRA_ENV_SETTINGS
>  	CONFIG_EXTRA_ENV_SETTINGS
>  #endif
> diff --git a/scripts/env2string.awk b/scripts/env2string.awk
> new file mode 100644
> index 00000000000..57d0fc8f3ba
> --- /dev/null
> +++ b/scripts/env2string.awk
> @@ -0,0 +1,80 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright 2021 Google, Inc
> +#
> +# SPDX-License-Identifier:	GPL-2.0+
> +#
> +# Awk script to parse a text file containing an environment and convert it
> +# to a C string which can be compiled into U-Boot.
> +
> +# The resulting output is:
> +#
> +#   #define CONFIG_EXTRA_ENV_TEXT "<environment here>"
> +#
> +# If the input is empty, this script outputs a comment instead.
> +
> +BEGIN {
> +	# env holds the env variable we are currently processing
> +	env = "";
> +	ORS = ""
> +}
> +
> +# Skip empty lines, as these are generated by the clang preprocessor
> +NF {
> +	# Quote quotes
> +	gsub("\"", "\\\"")
> +
> +	# Is this the start of a new environment variable?
> +	if (match($0, "^([^ \t=][^ =]*)=(.*)$", arr)) {
> +		if (length(env) != 0) {
> +			# Record the value of the variable now completed
> +			vars[var] = env
> +		}
> +		var = arr[1]
> +		env = arr[2]
> +
> +		# Deal with += which concatenates the new string to the existing
> +		# variable
> +		if (length(env) != 0 && match(var, "^(.*)[+]$", var_arr))
> +		{
> +			# Allow var\+=val to indicate that the variable name is
> +			# var+ and this is not actually a concatenation
> +			if (substr(var_arr[1], length(var_arr[1])) == "\\") {
> +				# Drop the backslash
> +				sub(/\\[+]$/, "+", var)
> +			} else {
> +				var = var_arr[1]
> +				env = vars[var] env
> +			}
> +		}
> +	} else {
> +		# Change newline to space
> +		gsub(/^[ \t]+/, "")
> +
> +		# Don't keep leading spaces generated by the previous blank line
> +		if (length(env) == 0) {
> +			env = $0
> +		} else {
> +			env = env " " $0
> +		}
> +	}
> +}
> +
> +END {
> +	# Record the value of the variable now completed. If the variable is
> +	# empty it is not set.
> +	if (length(env) != 0) {
> +		vars[var] = env
> +	}
> +
> +	if (length(vars) != 0) {
> +		printf("%s", "#define CONFIG_EXTRA_ENV_TEXT \"")
> +
> +		# Print out all the variables
> +		for (var in vars) {
> +			env = vars[var]
> +			print var "=" vars[var] "\\0"
> +		}
> +		print "\"\n"
> +	}
> +}
> diff --git a/test/py/tests/test_env.py b/test/py/tests/test_env.py
> index 9bed2f48d77..f85cb031382 100644
> --- a/test/py/tests/test_env.py
> +++ b/test/py/tests/test_env.py
> @@ -7,6 +7,7 @@
>  import os
>  import os.path
>  from subprocess import call, check_call, CalledProcessError
> +import tempfile
>  
>  import pytest
>  import u_boot_utils
> @@ -515,3 +516,109 @@ def test_env_ext4(state_test_env):
>      finally:
>          if fs_img:
>              call('rm -f %s' % fs_img, shell=True)
> +
> +def test_env_text(u_boot_console):
> +    """Test the script that converts the environment to a text file"""
> +
> +    def check_script(intext, expect_val):
> +        """Check a test case
> +
> +        Args:
> +            intext: Text to pass to the script
> +            expect_val: Expected value of the CONFIG_EXTRA_ENV_TEXT string, or
> +                None if we expect it not to be defined
> +        """
> +        with tempfile.TemporaryDirectory() as path:
> +            fname = os.path.join(path, 'infile')
> +            with open(fname, 'w') as inf:
> +                print(intext, file=inf)
> +            result = u_boot_utils.run_and_log(cons, ['awk', '-f', script, fname])
> +            if expect_val is not None:
> +                expect = '#define CONFIG_EXTRA_ENV_TEXT "%s"\n' % expect_val
> +                assert result == expect
> +            else:
> +                assert result == ''
> +
> +    cons = u_boot_console
> +    script = os.path.join(cons.config.source_dir, 'scripts', 'env2string.awk')
> +
> +    # simple script with a single var
> +    check_script('fred=123', 'fred=123\\0')
> +
> +    # no vars
> +    check_script('', None)
> +
> +    # two vars
> +    check_script('''fred=123
> +ernie=456''', 'fred=123\\0ernie=456\\0')
> +
> +    # blank lines
> +    check_script('''fred=123
> +
> +
> +ernie=456
> +
> +''', 'fred=123\\0ernie=456\\0')
> +
> +    # append
> +    check_script('''fred=123
> +ernie=456
> +fred+= 456''', 'fred=123 456\\0ernie=456\\0')
> +
> +    # append from empty
> +    check_script('''fred=
> +ernie=456
> +fred+= 456''', 'fred= 456\\0ernie=456\\0')
> +
> +    # variable with + in it
> +    check_script('fred+ernie=123', 'fred+ernie=123\\0')
> +
> +    # ignores variables that are empty
> +    check_script('''fred=
> +fred+=
> +ernie=456''', 'ernie=456\\0')
> +
> +    # single-character env name
> +    check_script('''f=123
> +e=456
> +f+= 456''', 'e=456\\0f=123 456\\0')
> +
> +    # contains quotes
> +    check_script('''fred="my var"
> +ernie=another"''', 'fred=\\"my var\\"\\0ernie=another\\"\\0')
> +
> +    # variable name ending in +
> +    check_script('''fred\\+=my var
> +fred++= again''', 'fred+=my var again\\0')
> +
> +    # variable name containing +
> +    check_script('''fred+jane=both
> +fred+jane+=again
> +ernie=456''', 'fred+jane=bothagain\\0ernie=456\\0')
> +
> +    # multi-line vars - new vars always start at column 1
> +    check_script('''fred=first
> + second
> +\tthird with tab
> +
> +   after blank
> + confusing=oops
> +ernie=another"''', 'fred=first second third with tab after blank confusing=oops\\0ernie=another\\"\\0')
> +
> +    # real-world example
> +    check_script('''ubifs_boot=
> +	env exists bootubipart ||
> +		env set bootubipart UBI;
> +	env exists bootubivol ||
> +		env set bootubivol boot;
> +	if ubi part ${bootubipart} &&
> +		ubifsmount ubi${devnum}:${bootubivol};
> +	then
> +		devtype=ubi;
> +		run scan_dev_for_boot;
> +	fi
> +''',
> +        'ubifs_boot=env exists bootubipart || env set bootubipart UBI; '
> +        'env exists bootubivol || env set bootubivol boot; '
> +        'if ubi part ${bootubipart} && ubifsmount ubi${devnum}:${bootubivol}; '
> +        'then devtype=ubi; run scan_dev_for_boot; fi\\0')
> -- 
> 2.33.0.1079.g6e70778dc9-goog
> 


More information about the U-Boot mailing list