[U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options

Simon Glass sjg at chromium.org
Tue Oct 9 16:20:25 UTC 2018


Hi Jean-Jacques,

On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot at ti.com> wrote:
> configs2csv.py is tool that allow to check how some options are used for a
> particular subset of platforms.
> The purpose is to identify the targets that are actually using one or more
> options of interest.
> For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
> It relies on the config database produced by tools/moveconfig.py.
> If the database doesn't exist, it will build it for the restricted set of
> the selected platforms. Once the database is built, it is much faster than
> greping the configs directory and more accurate as it relies on the
> information found in u-boot.cfg instead of defconfigs.
> It possible to look for options in the u-boot, the SPL or the TPL
> configurations. It can also perform diffs between those configurations.
>
> usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
>                       [--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
>                       [--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
>                       [--arch ARCH] [--cpu CPU] [--board BOARD]
>                       [--target TARGET]
>                       OPTION [OPTION ...]
>
> all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
> ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
> CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
> using the omap2, omap3, omap4, omap5 or k3 SOCs.
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot at ti.com>
>
> ---
>
> Changes in v2:
> - basically rewrote the whole thing
> - use tools/moveconfig.py to generate the database of configs
> - use tools/find_defconfigs.py to get the list of defconfigs off interest
> - removed diff with .config. tools/moveconfig.py does a better job
>
>  tools/configs2csv.py | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 387 insertions(+)
>  create mode 100755 tools/configs2csv.py

This looks like a useful tool and a big improvement on the movconfig
starting point. Style comments below.

>
> diff --git a/tools/configs2csv.py b/tools/configs2csv.py
> new file mode 100755
> index 0000000..70b6602
> --- /dev/null
> +++ b/tools/configs2csv.py
> @@ -0,0 +1,387 @@
> +#!/usr/bin/env python
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Author: JJ Hiblot <jjhiblot at ti.com>
> +#
> +
> +"""
> +scan the configuration of specified targets (ie defconfigs) and outputs a
> +summary in a csv file.
> +Useful tool to check what platform is using a particular set of options.
> +
> +
> +How does it work?
> +-----------------
> +
> +This tools uses the config database produced by tools/moveconfig.py (called
> +with option -B to get all the configs: SPl, TPL and u-boot). If the database
> +is not present, it will build it. A rebuild can be forced with the option
> +'--rebuild-db'.
> +
> +The list of the targets of interest can be specified by a set of filter (soc,
> +vendor, defconfig name, ..). All those filters are actually regexp, allowing
> +for complex selection. The selection process is done by
> +tools/find_defconfigs.py
> +ex: --soc omap[23] --vendor 'ti|compulab' will inspect the omap2 and omap3
> +platforms from TI and compulab
> +
> +
> +examples:
> +---------
> +
> +
> +1) Get an overview of the usage of CONFIG_DM, CONFIG_SPL_DM, and DM/I2C related
> +   options for platforms with omap5 or k3 SOC in u-boot and in SPL
> +
> +$ tools/configs2csv.py CONFIG_SPL_DM CONFIG_DM CONFIG_DM_I2C.* --vendor ti \
> +       --soc 'omap5|k3' -X --u-boot --spl  -o dummy.csv
> +

Is this output below showing the contents of the ,csv file in a non-CVS format?

> +vendor   soc            defconfig            type     CONFIG_DM  CONFIG_DM_I2C  CONFIG_DM_I2C_COMPAT  CONFIG_SPL_DM
> +ti      omap5     am57xx_evm_defconfig        SPL         X                                               X
> +ti      omap5     am57xx_evm_defconfig        u-boot      X         X                    X                X
> +ti      omap5     am57xx_hs_evm_defconfig     SPL         X                                               X
> +ti      omap5     am57xx_hs_evm_defconfig     u-boot      X         X                    X                X
> +ti      k3        am65x_evm_a53_defconfig     SPL         X                                               X
> +ti      k3        am65x_evm_a53_defconfig     u-boot      X                                               X
> +ti      omap5     dra7xx_evm_defconfig        SPL         X                                               X
> +ti      omap5     dra7xx_evm_defconfig        u-boot      X         X                    X                X
> +ti      omap5     dra7xx_hs_evm_defconfig     SPL         X                                               X
> +ti      omap5     dra7xx_hs_evm_defconfig     u-boot      X         X                    X                X
> +ti      omap5     omap5_uevm_defconfig        SPL
> +ti      omap5     omap5_uevm_defconfig        u-boot
> +
> +
> +This shows quickly that DM is not supported at all for omap5_uevm, that
> +only am65x_evm_a53 in not using DM_I2C in u-boot, and finally that DM_I2C is
> +not enabled in the SPL for any platform although SPL_DM is.
> +Also all the other platforms that enabled DM_I2C, also enabled
> +CONFIG_DM_I2C_COMPAT.
> +
> +
> +2) Check differences in config between SPL, TPL and u-boot (--diff option)
> +
> +Some platforms may disable/enable stuff in the configuration header files if
> +in SPl. This makes it hard to know the usage of a variable by just looking at
> +the .config. This is specially true for DM stuff.
> +
> +$ tools/configs2csv.py CONFIG\(_SPL\)?_DM_.*  --vendor ti \
> +  --soc 'omap5|k3' --diff --spl --u-boot > dummy.csv
> +
> +vendor    soc        defconfig             CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT  CONFIG_DM_STDIO  CONFIG_DM_WARN
> +ti        omap5    am57xx_evm_defconfig       u-boot         u-boot                 u-boot           u-boot
> +ti        omap5    am57xx_hs_evm_defconfig    u-boot         u-boot                 u-boot           u-boot
> +ti        k3       am65x_evm_a53_defconfig                                          u-boot           u-boot
> +ti        omap5    dra7xx_evm_defconfig       u-boot         u-boot                 u-boot           u-boot
> +ti        omap5    dra7xx_hs_evm_defconfig    u-boot         u-boot                 u-boot           u-boot
> +
> +This shows that k3 has no real config diff between SPl and u-boot. whereas am57
> +and dra7 have different settings for DM_I2C and DM_I2C_COMPAT
> +
> +"""
> +
> +import argparse
> +import csv
> +import os
> +import re
> +import sys
> +from collections import namedtuple
> +from itertools import combinations
> +
> +import find_defconfigs
> +
> +CONFIG_DATABASE = 'moveconfig.db'
> +target = namedtuple("target", ["defconfig", "binary_type"])
> +
> +
> +class db:

Please capitalise the class name. Also how about ConfigDb or something
a little longer than db? Below you actually use db as a variable name.

> +
> +    """ db is an object that store a collection of targets
> +    a target is identified buy its defconfig and its binary type. ex:
> +    (omap3_evm_defconfig,SPL)
> +    The main purpose of this object is to output a CSV file that describes all
> +    the targets.
> +    There is also the possibility to create a "diff" db from a db. This new db
> +    contains a summary of the differences between target of same defconfig.
> +    """
> +
> +    def __init__(self):
> +        self.targets = dict()
> +
> +    def add_target(self, target):
> +        self.targets[target] = dict()
> +
> +    def add_option(self, target, option, value):
> +        self.targets[target][option] = value
> +
> +    def add_options(self, target, dic):
> +        self.targets[target].update(dic)
> +
> +    def output_csv(
> +            self, output, show_X=False, header=True, left_columns=None, discard_empty_rows=False):

Please add function comments with Args and Returns (if you have
argument and return value) along with what the function does.

> +        all_options = set()
> +        if len(self.targets) == 0:
> +            return
> +
> +        if discard_empty_rows:
> +            dic = {k: self.targets[k] for k in self.targets if self.targets[k]}
> +        else:
> +            dic = self.targets.copy()
> +        for target in dic.keys():
> +            for option in dic[target].keys():
> +                all_options.add(option)
> +                if show_X:
> +                    dic[target][option] = "X"
> +            if left_columns:
> +                left_columns(target, dic, header=False)
> +
> +        columns = []
> +        if left_columns:
> +            columns.extend(left_columns(None, header=True))
> +        columns.extend(sorted(all_options))
> +
> +        writer = csv.DictWriter(output, fieldnames=columns,
> +                                lineterminator='\n')
> +        if header:
> +            writer.writeheader()
> +        for target in sorted(dic.keys()):
> +            writer.writerow(dic[target])
> +
> +    def diff_one_defconfig(self, defconfig):
> +        """ This function creates a dictionary of the differences between the
> +        binaries os a single target.
> +        For example, for "dra7xx_evm_defconfig" it will compute the diffence
> +        between the options used to compile u-boot and the SPL (not the TPL
> +        because this platform doesn't have it).
> +        The return value looks as follow: { 'CONFIG_DM_I2C: "u-boot",
> +        CONFIG_SPL_BUILD:"SPL", CONFIG_DUMMY_SPI_FREQ: "diff" }.
> +
> +        The algorithm can probably be optimized, but I didn't care enough.
> +        algo is:
> +        - return immediately is there is only one binary type (u-boot)
> +        - create a dic that is merge of the dic for all the binary types
> +        - for each binary type, compare its dic to the merged_dic. If it is
> +        different then break. It means that at least one option is different.
> +        - if no difference has been found, then return
> +        - at this point, we know that there is at least one diff. For each
> +        binary types and for all options used for this binary type, check if it
> +        is in the merged dic and, if so, if its value is the same. update our
> +        return dic with the proper description.
> +

Drop blank line before """

Returns:
   dict:
      key: ...
      value: ...

See buildman,, etc. for examples.

> +        """
> +
> +        diffs = dict()
> +        diff_found = False
> +
> +        # get all binary types (spl, TPL, u-boot) generated by this defconfig
> +        all_binary_types = sorted(
> +            set([t.binary_type for t in self.targets.keys() if t.defconfig == defconfig]))
> +
> +        # If there is only one type of binary, no need to do a diff
> +        if len(all_binary_types) <= 1:
> +            return None
> +
> +        # create a dict with all options:values used by all binaries
> +        merged_dic = dict()
> +        for bin_type in all_binary_types:
> +            merged_dic.update(self.targets[target(defconfig, bin_type)])
> +
> +        # check if all binaries have the same options (should be the case for
> +        # most of the defconfigs)
> +        for bin_type in all_binary_types:
> +            if self.targets[target(defconfig, bin_type)] != merged_dic:
> +                diff_found = True
> +                break
> +        if not diff_found:
> +            return None
> +
> +        # at this point, we know that there are some options that differ
> +        # between binaries (either not present or different)
> +
> +        # Get a list (actually a set) of the options that are different
> +        differing_keys = set()
> +        for bin_type in all_binary_types:
> +            dic = self.targets[target(defconfig, bin_type)]
> +            for opt, value in merged_dic.items():
> +                if dic.get(opt, None) != value:
> +                    differing_keys.add(opt)
> +
> +        # create a dictionary that summarize the differences
> +        for bin_type in all_binary_types:
> +            dic = self.targets[target(defconfig, bin_type)]
> +            for opt in differing_keys:
> +                dic_value = dic.get(opt, None)
> +                merged_value = merged_dic.get(opt, None)
> +                previous = diffs.get(opt, None)
> +                if dic_value:
> +                    if dic_value != merged_value:
> +                        diffs[opt] = "diff"
> +                    elif previous != "diff":
> +                        diffs[opt] = ' / '.join(
> +                            [previous, bin_type]) if previous else bin_type
> +
> +        return diffs
> +
> +    def diff(self):
> +        """ create a new db that contains the differences between the binaries
> +        for all the targets in the db """

""" goes on its own line. Looks like this needs Returns comment.

> +        diff_db = db()
> +        # get a list of all the targets
> +        all_defconfigs = set([t.defconfig for t in self.targets.keys()])
> +        # for every target of the list, get a dictionary of the difference.
> +        # if the dictionary is not empty, add it the new db
> +        for defconfig in all_defconfigs:
> +            diff_dic = self.diff_one_defconfig(defconfig)
> +            if diff_dic:
> +                diff_db.add_target(target(defconfig, None))
> +                diff_db.add_options(target(defconfig, None), diff_dic)
> +        return diff_db
> +
> +
> +def read_db(boards, option_filter, binary_types):

Needs function comment again.

> +    defconfig = ""
> +    _db = db()
> +
> +    # Read in the database
> +    with open(CONFIG_DATABASE) as fd:
> +        for line in fd.readlines():
> +            line = line.rstrip()
> +            if not line:  # Separator between defconfigs.
> +                # We do not really care. We detect a new config by the absence
> +                # of ' 'at the beginning of the line
> +                pass
> +            elif line[0] == ' ':  # CONFIG_xxx line
> +                if t and option_filter(line):
> +                    config, value = line.strip().split('=', 1)
> +                    _db.add_option(t, config, value)
> +            else:  # New defconfig
> +                infos = line.split()
> +                defconfig = infos[0]
> +                try:
> +                    binary_type = infos[1]
> +                except:
> +                    binary_type = "u-boot"
> +                if binary_type in binary_types and defconfig in boards:
> +                    t = target(defconfig, binary_type)
> +                    _db.add_target(t)
> +                else:
> +                    t = None
> +    return _db
> +
> +
> +def main():
> +    parser = argparse.ArgumentParser(description="Show CONFIG options usage")
> +    parser.add_argument("options", metavar='OPTION', type=str, nargs='+',
> +                        help="regexp to filter on options.\
> +        ex: CONFIG_DM_I2C_COMPAT or '.*DM_MMC.*'")
> +    parser.add_argument(
> +        "-X", help="show a X instead of the value of the option",

Please capitalise the help, e.g. 'Show a X'

> +                        action="store_true")
> +    parser.add_argument("--u-boot", help="parse the u-boot configs",
> +                        action="store_true")
> +    parser.add_argument("--spl", help="parse the SPL configs",
> +                        action="store_true")
> +    parser.add_argument("--tpl", help="parse the TPL configs",
> +                        action="store_true")
> +    parser.add_argument("--diff",
> +                        help="show only the options that differs between the selected configs (SPL, TPL, u-boot)",
> +                        action="store_true")
> +    parser.add_argument("--rebuild-db",
> +                        help="Force a rebuild of the config database",
> +                        action="store_true")
> +    parser.add_argument('-j', '--jobs',
> +                        help='the number of jobs to run simultaneously')
> +    parser.add_argument('-o', '--output',
> +                        help='The output CSV filename. uses stdout if not specified')
> +    parser.add_argument('--no-header', help='Do not put the header at the top',
> +                        action="store_true")
> +    parser.add_argument('--discard-empty', action="store_true",
> +                        help='Discard the empty rows (defconfigs that do not enable at least one option)')
> +
> +    find_defconfigs.update_parser_with_default_options(parser)
> +    args = parser.parse_args()
> +
> +    # generate db file if needed or requested
> +    # The job of generating the db is actually done by tools/moveconfig.py
> +    # (called with -B)
> +    if args.rebuild_db or not os.path.isfile(CONFIG_DATABASE):
> +        find_defconfig_args = ["--{} '{}'".format(f, getattr(args, f))
> +                               for f in find_defconfigs.get_default_options()
> +                               if getattr(args, f)]
> +        if args.jobs:
> +            jobs_option = "-j {}".format(args.jobs)
> +        else:
> +            jobs_option = ""
> +
> +        rc = os.system(
> +            "tools/find_defconfigs.py {} | tools/moveconfig.py -B {} -d - 1>&2 "
> +            .format(" ".join(find_defconfig_args), jobs_option))
> +        if rc:
> +            sys.exit(1)

This is fine, but I wonder why you don't import this module and call
it directly? It could return a dict, perhaps?

> +
> +    # get a list of defconfigs matching the rules
> +    targets = [t for t in find_defconfigs.get_matching_boards(args)]
> +    defconfigs = [t.defconfig for t in targets]
> +
> +    # create a list of binary types we are interested in
> +    binary_types = []
> +    if args.spl:
> +        binary_types.append("SPL")
> +    if args.tpl:
> +        binary_types.append("TPL")
> +    if args.u_boot or not binary_types:
> +        binary_types.append("u-boot")
> +
> +    # define a function used to filter on the options
> +    rules = [re.compile("   {}=".format(cfg_opt))
> +             for cfg_opt in args.options]
> +
> +    def match_any_rule(line):
> +        for r in rules:
> +            if r.match(line):
> +                return True
> +        return False
> +
> +    # read the database
> +    db = read_db(defconfigs, match_any_rule, binary_types)
> +
> +    target_dict = {}
> +    for t in targets:
> +        target_dict[t.defconfig] = t
> +
> +    def populate_left_columns(target=None, dic=None, header=True):
> +        if header:
> +            return ["vendor", "soc", "defconfig", "type"]
> +        else:
> +            dic[target]["vendor"] = target_dict[target.defconfig].vendor
> +            dic[target]["soc"] = target_dict[target.defconfig].soc
> +            dic[target]["defconfig"] = target.defconfig
> +            dic[target]["type"] = target.binary_type
> +
> +    def populate_left_columns_diff(target=None, dic=None, header=True):
> +        if header:
> +            return ["vendor", "soc", "defconfig"]
> +        else:
> +            dic[target]["vendor"] = target_dict[target.defconfig].vendor
> +            dic[target]["soc"] = target_dict[target.defconfig].soc
> +            dic[target]["defconfig"] = target.defconfig
> +
> +    if args.output:
> +        out = open(args.output, "w")
> +    else:
> +        out = sys.stdout
> +
> +    if args.diff:
> +        db.diff().output_csv(output=out, show_X=False,
> +                             header=not args.no_header,
> +                             discard_empty_rows=args.discard_empty,
> +                             left_columns=populate_left_columns_diff)
> +    else:
> +        db.output_csv(output=out, show_X=args.X, header=not args.no_header,
> +                      discard_empty_rows=args.discard_empty,
> +                      left_columns=populate_left_columns)
> +
> +    if out != sys.stdout:
> +        out.close()
> +
> +if __name__ == '__main__':
> +    main()
> --
> 2.7.4
>

Regards,
Simon


More information about the U-Boot mailing list