[PATCH 1/1] lib: rsa: fix padding_pss_verify

[Heinrich Schuchardt]

On 8/31/22 15:46, Simon Glass wrote:
> Hi Heinrich,
> 
> On Wed, 31 Aug 2022 at 03:32, Heinrich Schuchardt
> <heinrich.schuchardt at canonical.com> wrote:
>>
>> Check the that the hash length is shorter than the message length. This
>> avoids:
>>
>>      ./tools/../lib/rsa/rsa-verify.c:275:11: warning:
>>      ‘*db’ may be used uninitialized [-Wmaybe-uninitialized]
>>        275 |         db[0] &= 0xff >> leftmost_bits;
>>
>> Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt at canonical.com>
>> ---
>>   lib/rsa/rsa-verify.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/rsa/rsa-verify.c b/lib/rsa/rsa-verify.c
>> index 1d95cfbdee..255f99be24 100644
>> --- a/lib/rsa/rsa-verify.c
>> +++ b/lib/rsa/rsa-verify.c
>> @@ -234,6 +234,9 @@ int padding_pss_verify(struct image_sign_info *info,
>>          uint8_t leftmost_mask;
>>          struct checksum_algo *checksum = info->checksum;
>>
>> +       if (db_len <= 0)
>> +               return 1;
>> +
>>          /* first, allocate everything */
>>          db_mask = malloc(db_len);
>>          db = malloc(db_len);
>> --
>> 2.37.2
>>
> 
> While you are here can you please add a 'Returns' comment for this
> function? I think in this case you are return 1, meaning 'different' ?

Looking at the rest of the code 0 means ok and non-zero means not ok.

At another code position ret = EINVAL. So let me update the patch.

Best regards

Heinrich

> 
> Regards,
> Simon