[PATCH v2 1/2] lib: zstd: update to latest Linux zstd 1.5.2

[Simon Glass]

Hi Brandon,

On Tue, 27 Dec 2022 at 09:16, Brandon Maier <brandon.maier at collins.com> wrote:
>
> Update the zstd implementation to match Linux zstd 1.5.2 from commit
> 2aa14b1ab2.
>
> This was motivated by running into decompression corruption issues when
> trying to uncompress files compressed with newer versions of zstd. zstd
> users also claim significantly improved decompression times with newer
> zstd versions which is a side benefit.
>
> Original zstd code was copied from Linux commit 2aa14b1ab2 which is a
> custom-built implementation based on zstd 1.3.1. Linux switched to an
> implementation that is a copy of the upstream zstd code in Linux commit
> e0c1b49f5b, this results in a large code diff. However this should make
> future updates easier along with other benefits[1].
>
> This commit is a straight mirror of the Linux zstd code, except to:
> - update a few #include that do not translate cleanly
>   - linux/swab.h -> asm/byteorder.h
>   - linux/limits.h -> linux/kernel.h
>   - linux/module.h -> linux/compat.h
> - remove assert() from debug.h so it doesn't conflict with u-boot's
>   assert()
> - strip out the compressor code as was done in the previous u-boot zstd
> - update existing zstd users to the new Linux zstd API
> - add a new KConfig option that sets zstd build options to minify code
>   based on zstd's ZSTD_LIB_MINIFY[2].
>
> These changes were tested by booting a zstd 1.5.2 compressed kernel inside a
> FIT. And the squashfs changes by loading a file from zstd compressed squashfs
> with sqfsload. buildman was used to compile test other boards and check for
> binary bloat, as follows:
>
> > $ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS
> > Summary of 3 commits for 9 boards (8 threads, 1 job per thread)
> > 01: Merge tag 'tpm-23122022' of https://source.denx.de/u-boot/custodians/u-boot-tpm
> >        arm:  w+   m53menlo dh_imx6
> > 02: lib: zstd: update to latest Linux zstd 1.5.2
> >    aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0
> >        arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6

That's a pretty stiff penalty. Do you know what has changed? Does it
support a lot more optional features? If so, could we make it optional
(even if default y)?


> >    sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0
>
> [1] https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8
> [2] https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31
>
> Signed-off-by: Brandon Maier <brandon.maier at collins.com>
> ---
> v2:
> - zstd_decompress(): Handle junk at the end of the decompression buffer
>   by checking actual size with zstd_find_frame_compressed_size().
> ---
>  fs/squashfs/sqfs_decompressor.c               |   15 +-
>  include/linux/zstd.h                          | 1244 ++------
>  include/linux/zstd_errors.h                   |   77 +
>  include/linux/zstd_lib.h                      | 2551 +++++++++++++++++
>  lib/Kconfig                                   |   15 +
>  lib/zstd/Makefile                             |   35 +-
>  lib/zstd/bitstream.h                          |  344 ---
>  lib/zstd/common/bitstream.h                   |  446 +++
>  lib/zstd/common/compiler.h                    |  184 ++
>  lib/zstd/common/cpu.h                         |  194 ++
>  lib/zstd/common/debug.c                       |   24 +
>  lib/zstd/common/debug.h                       |   92 +
>  lib/zstd/common/entropy_common.c              |  357 +++
>  lib/zstd/common/error_private.c               |   56 +
>  lib/zstd/common/error_private.h               |  145 +
>  lib/zstd/common/fse.h                         |  711 +++++
>  lib/zstd/common/fse_decompress.c              |  390 +++
>  lib/zstd/common/huf.h                         |  358 +++
>  lib/zstd/common/mem.h                         |  261 ++
>  lib/zstd/common/portability_macros.h          |   93 +
>  lib/zstd/common/zstd_common.c                 |   83 +
>  lib/zstd/common/zstd_deps.h                   |  125 +
>  lib/zstd/common/zstd_internal.h               |  443 +++
>  lib/zstd/decompress.c                         | 2516 ----------------
>  lib/zstd/decompress/huf_decompress.c          | 1740 +++++++++++
>  lib/zstd/decompress/zstd_ddict.c              |  241 ++
>  lib/zstd/decompress/zstd_ddict.h              |   44 +
>  lib/zstd/decompress/zstd_decompress.c         | 2131 ++++++++++++++
>  lib/zstd/decompress/zstd_decompress_block.c   | 2072 +++++++++++++
>  lib/zstd/decompress/zstd_decompress_block.h   |   68 +
>  .../decompress/zstd_decompress_internal.h     |  228 ++
>  lib/zstd/decompress_sources.h                 |   34 +
>  lib/zstd/entropy_common.c                     |  213 --
>  lib/zstd/error_private.h                      |   43 -
>  lib/zstd/fse.h                                |  545 ----
>  lib/zstd/fse_decompress.c                     |  302 --
>  lib/zstd/huf.h                                |  182 --
>  lib/zstd/huf_decompress.c                     |  930 ------
>  lib/zstd/mem.h                                |  142 -
>  lib/zstd/zstd.c                               |   52 +-
>  lib/zstd/zstd_common.c                        |   66 -
>  lib/zstd/zstd_common_module.c                 |   32 +
>  lib/zstd/zstd_decompress_module.c             |  105 +
>  lib/zstd/zstd_internal.h                      |  253 --
>  lib/zstd/zstd_opt.h                           | 1004 -------
>  45 files changed, 13638 insertions(+), 7548 deletions(-)
>  create mode 100644 include/linux/zstd_errors.h
>  create mode 100644 include/linux/zstd_lib.h
>  delete mode 100644 lib/zstd/bitstream.h
>  create mode 100644 lib/zstd/common/bitstream.h
>  create mode 100644 lib/zstd/common/compiler.h
>  create mode 100644 lib/zstd/common/cpu.h
>  create mode 100644 lib/zstd/common/debug.c
>  create mode 100644 lib/zstd/common/debug.h
>  create mode 100644 lib/zstd/common/entropy_common.c
>  create mode 100644 lib/zstd/common/error_private.c
>  create mode 100644 lib/zstd/common/error_private.h
>  create mode 100644 lib/zstd/common/fse.h
>  create mode 100644 lib/zstd/common/fse_decompress.c
>  create mode 100644 lib/zstd/common/huf.h
>  create mode 100644 lib/zstd/common/mem.h
>  create mode 100644 lib/zstd/common/portability_macros.h
>  create mode 100644 lib/zstd/common/zstd_common.c
>  create mode 100644 lib/zstd/common/zstd_deps.h
>  create mode 100644 lib/zstd/common/zstd_internal.h
>  delete mode 100644 lib/zstd/decompress.c
>  create mode 100644 lib/zstd/decompress/huf_decompress.c
>  create mode 100644 lib/zstd/decompress/zstd_ddict.c
>  create mode 100644 lib/zstd/decompress/zstd_ddict.h
>  create mode 100644 lib/zstd/decompress/zstd_decompress.c
>  create mode 100644 lib/zstd/decompress/zstd_decompress_block.c
>  create mode 100644 lib/zstd/decompress/zstd_decompress_block.h
>  create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h
>  create mode 100644 lib/zstd/decompress_sources.h
>  delete mode 100644 lib/zstd/entropy_common.c
>  delete mode 100644 lib/zstd/error_private.h
>  delete mode 100644 lib/zstd/fse.h
>  delete mode 100644 lib/zstd/fse_decompress.c
>  delete mode 100644 lib/zstd/huf.h
>  delete mode 100644 lib/zstd/huf_decompress.c
>  delete mode 100644 lib/zstd/mem.h
>  delete mode 100644 lib/zstd/zstd_common.c
>  create mode 100644 lib/zstd/zstd_common_module.c
>  create mode 100644 lib/zstd/zstd_decompress_module.c
>  delete mode 100644 lib/zstd/zstd_internal.h
>  delete mode 100644 lib/zstd/zstd_opt.h
>