[PATCH 04/45] rsa: Avoid warning in padding_pss_verify()

[Heinrich Schuchardt]

On 9/25/22 17:02, Simon Glass wrote:
> With gcc 12 we ge the following warning:

%s/ge/get/

>
> In file included from tools/lib/rsa/rsa-verify.c:1:
> lib/rsa/rsa-verify.c:275:11: warning: ‘*db’ may be used uninitialized
>    275 |         db[0] &= 0xff >> leftmost_bits;
>
> Check the value of db_len to ensure this cannot happen.
>
> Signed-off-by: Simon Glass <sjg at chromium.org>
> ---
>
>   lib/rsa/rsa-verify.c | 5 +++++
>   1 file changed, 5 insertions(+)
>
> diff --git a/lib/rsa/rsa-verify.c b/lib/rsa/rsa-verify.c
> index 1d95cfbdee0..81c39251e59 100644
> --- a/lib/rsa/rsa-verify.c
> +++ b/lib/rsa/rsa-verify.c
> @@ -234,6 +234,11 @@ int padding_pss_verify(struct image_sign_info *info,
>   	uint8_t leftmost_mask;
>   	struct checksum_algo *checksum = info->checksum;
>
> +	if (db_len <= 0) {

Why are length fields msg_len, hash_len, db_len signed? I cannot imagine
an array with negative length. Any of the parameters msg_len and
hash_len being negative would be an error.

The check here should be

	if (msg_len >= hash_len)

to allow db_len moving to unsigned types.

struct padding_algo() should be corrected to use size_t.

Best regards

Heinrich

> +		ret = -EINVAL;
> +		goto out;
> +	}
> +
>   	/* first, allocate everything */
>   	db_mask = malloc(db_len);
>   	db = malloc(db_len);