[U-Boot] [PATCH 2/4] rsa: Verify RSA padding programatically

[Andrew Duda]

Simon,

So I looked into this more after you asked this, and it looks very
platform dependent. I tested on two builds: sandbox and a version of
x86-common. The before/after for sandbox image was
5486016-5486800(+784). The before/after for my x86 build was
3306100-3305908(-192). So memory saving is anywhere from a few bytes
to actually more space. But the big motivation is the next two patches
depend on this change.

Thanks,
Andrew

On Fri, Nov 11, 2016 at 8:17 AM, Simon Glass <sjg at chromium.org> wrote:
> On 8 November 2016 at 11:53, aduda <aduda at meraki.com> wrote:
>> From: Andrew Duda <aduda at meraki.com>
>>
>> Padding verification was done against static SHA/RSA pair arrays which
>> take up a lot of static memory, are mostly 0xff, and cannot be reused
>> for additional SHA/RSA pairings. The padding can be easily computed
>> according to PKCS#1v2.1 as:
>>
>>   EM = 0x00 || 0x01 || PS || 0x00 || T
>>
>> where PS is (emLen - tLen - 3) octets of 0xff and T is DER encoding
>> of the hash.
>>
>> Store DER prefix in checksum_algo and create rsa_verify_padding
>> function to handle verification of a message for any SHA/RSA pairing.
>>
>> Signed-off-by: Andrew Duda <aduda at meraki.com>
>> Signed-off-by: aduda <aduda at meraki.com>
>> ---
>>
>>  common/image-sig.c            |   9 ++--
>>  include/image.h               |   3 +-
>>  include/u-boot/rsa-checksum.h |   4 --
>>  include/u-boot/sha1.h         |   3 ++
>>  include/u-boot/sha256.h       |   3 ++
>>  lib/rsa/rsa-checksum.c        | 121 ------------------------------------------
>>  lib/rsa/rsa-verify.c          |  38 ++++++++++++-
>>  lib/sha1.c                    |   5 ++
>>  lib/sha256.c                  |   6 +++
>>  9 files changed, 61 insertions(+), 131 deletions(-)
>
> Reviewed-by: Simon Glass <sjg at chromium.org>
>
> How much memory does this save?