[PATCH v2 0/4] fit: allow signing with an OpenSSL engine

[Quentin Schulz]

I have a couple of products whose U-Boot FIT is signed via a proprietary
OpenSSL engine which only expects the name of a "slot" to select the key
to sign data with.

Currently mkimage fit support expects either a key-dir (-k) or a
key-file (-G) as a toggle for signing, however this doesn't apply to our
usecase because we use an OpenSSL engine (so no key-file to provide)
which doesn't mimic a directory layout like key-dir implies. Moreover,
binman really expects private keys (.key extension) to be available in
this key-dir directory, which we of course cannot provide.

This series allows to sign a FIT image with mkimage (and binman) with
an OpenSSL engine, including PKCS11 and custom engines. If a key-dir
needs to be passed (which is typical for PKCS11), one can do so by using
fit,engine-keydir.

Note that the public key (.crt extension) still needs to be available if
one wants to embed it for signature verification (which is probably what
one wants to do :) ). It is probably possible to use the engine for
getting the public key instead of storing it on disk, but this needs to
be added to fdt_add_pubkey and then binman, through a mechanism
different from fit,engine*.

One issue though is that since binman resolves key paths absolutely and
that I don't believe an OpenSSL engine would happen to have the exact
same key_id value than a local absolute path, fit,encrypt and
fit,engine cannot cohabit. An issue for the next person who wants
an OpenSSL engine AND encrypt the same FIT image, I don't.

Note that LibreSSL supports neither engines nor providers as far as I
could tell (engine support has been explicitly removed).

Note that OpenSSL engines have been deprecated since 3.0 (Q3-2021),
however note that OpenSSL 3.5 still seems to support engines (git grep)
and is EOL end of Q1 2030.

I am not entirely sure if the setup for the PKCS11 tests package
requirements is implemented the proper way (the double try:catch).

+Cc Eddie who's working on OpenSSL provider support, maybe we can work
together on supporting both engines and providers via the same API we
expose to the user?
+Cc Wolfgang who seems interested in signing with PKCS11 engine which
should now be supported.
+Cc Peter who seems interested according to v1

@Wolfgang, key-name-hint will only be a hint to use when verifying (as
well as being used for the node name), but if you have a
fit,engine-keydir which contains object= or id= in it, key-name-hint
won't be passed to your engine.

Signed-off-by: Quentin Schulz <quentin.schulz at cherry.de>
---
Changes in v2:
- added R-b on patch 1,
- did NOT add T-b and R-B on patch 3 since there's been some rework,
- added binman test for signing with dummy RSA engine,
- added binman test for signing with SoftHSMv2 PKCS11 engine,
- added binman test for signing with SoftHSMv2 PKCS11 engine and
  specific keydir (which contains an object= to tackle that special
  case)
- added dummy RSA engine for binman test,
- renamed property to fit,engine as my hunch is that it could be used
  for encrypting as well (from the same engine with the same keydir but
  likely a different key-name-hint so that a different keypair is used)
- added fit,engine-keydir to easily support PKCS11 with arguments other
  than object= taken from key-name-hint,
- Link to v1: https://patch.msgid.link/20251031-binman-engine-v1-0-c13c1b5dac43@cherry.de

---
Quentin Schulz (4):
      fit: support signing with only an engine_id
      tools: binman: mkimage: add support for passing the engine
      tools: binman: fit: add support for OpenSSL engines
      tools: binman: fit: add tests for signing with an OpenSSL engine

 tools/binman/btool/mkimage.py                      |   5 +-
 tools/binman/btool/softhsm2_util.py                |  21 +++
 tools/binman/entries.rst                           |  45 ++++-
 tools/binman/etype/fit.py                          |  61 ++++++-
 tools/binman/ftest.py                              | 193 +++++++++++++++++++++
 tools/binman/test/340_dummy-rsa4096.crt            |  31 ++++
 tools/binman/test/340_fit_signature_engine.dts     |  99 +++++++++++
 .../test/340_fit_signature_engine_pkcs11.dts       |  99 +++++++++++
 .../340_fit_signature_engine_pkcs11_object.dts     | 100 +++++++++++
 tools/binman/test/340_openssl.conf                 |  10 ++
 tools/binman/test/340_softhsm2.conf                |  16 ++
 tools/binman/test/Makefile                         |   6 +-
 tools/binman/test/dummy-rsa-engine.c               | 149 ++++++++++++++++
 tools/fit_image.c                                  |   3 +-
 tools/image-host.c                                 |   4 +-
 15 files changed, 830 insertions(+), 12 deletions(-)
---
base-commit: 089d92860e1ba63c52fba9b2ac8aa42bf03ebc03
change-id: 20251030-binman-engine-e349b02696d0

Best regards,
-- 
Quentin Schulz <quentin.schulz at cherry.de>