[RFC] Plan to Migrate U-Boot from OpenSSL Engines to Providers

[Enric Balletbo i Serra]

Hi all,

As many of you may be aware, recent Linux distributions (such as
Fedora) are removing support for OpenSSL engines, following the
deprecation of the ENGINE API in OpenSSL 3.x. This is already
impacting the ability to build U-Boot with hardware-backed signing on
these systems, and the situation is likely to become more widespread
as other distros follow suit.

To address this, we would like to propose the following plan for U-Boot:

* Introduce a new config option OPENSSL_NO_DEPRECATED.
This option will allow building U-Boot without any ENGINE support.
When enabled, hardware-backed signing (PKCS#11, etc.) will not be
available, but U-Boot will be buildable on distributions that have
removed ENGINE support from OpenSSL.

* Add support for OpenSSL Providers
Migrate the relevant signing and key-loading code to use the OpenSSL
Provider API, which is the officially supported mechanism in OpenSSL
3.x and beyond. Providers are the modern way to access both software
and hardware cryptography, including PKCS#11, and are supported in all
current OpenSSL releases.

Arguments for this change:

* Future-proofing: The ENGINE API is deprecated and will eventually be
removed from OpenSSL. Providers are the supported path forward.
* Compatibility: Many distributions are already disabling or removing
ENGINE support, making it impossible to build U-Boot with signing
support out-of-the-box.
* Security: The Provider API receives active maintenance and security
updates, while ENGINE is legacy code.
* Flexibility: Providers can support both software and hardware
cryptography, and are more flexible for future enhancements.


Impact:

Users who require hardware-backed signing will need to ensure their
provider (e.g., PKCS#11 provider) is available and configured.

The transition will be gradual: ENGINE support can be kept as an
option for a while, but the default and recommended path will be
providers.


Questions for the community:

* Do we need to maintain ENGINE compatibility for a transition period,
or can we remove ENGINE support entirely and move fully to providers?

* Are there users or workflows that still depend on ENGINE-based
hardware signing that cannot be migrated to providers?

* Any other concerns or suggestions regarding this migration?


Next Steps:

* Add the OPENSSL_NO_DEPRECATED config and update documentation.
* Begin refactoring the signing code to use providers by default.


Note that we feel comfortable with adding the OPENSSL_NO_DEPRECATED
config option and can proceed with that change. However, we are not
very familiar with the details of migrating from OpenSSL Engines to
Providers. Any help, guidance, or code contributions from community
members experienced with the OpenSSL provider API would be greatly
appreciated.

Please let us know your thoughts, concerns, or if you are interested
in helping or you are already working with this migration.

Best regards,

  Eddie and Enric