[PATCH] lib/rsa: Sanity check db_len

[Joel Stanley]

When building with GCC 12:

../tools/../lib/rsa/rsa-verify.c:275:11: warning: ‘*db’ may be used uninitialized [-Wmaybe-uninitialized]
  275 |         db[0] &= 0xff >> leftmost_bits;
      |         ~~^~~

If msg_len or hash_len were nonsense values, db_len would end up invalid
and the rest of the function wil not work, so detect this case and
return early. If this was host code we could assert, but as this is
target code print an error and return.

Fixes: 061daa0b61f0 ("rsa: add support of padding pss")
Signed-off-by: Joel Stanley <joel at jms.id.au>
---
 lib/rsa/rsa-verify.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/rsa/rsa-verify.c b/lib/rsa/rsa-verify.c
index 1d95cfbdee0c..a7c87e40afa2 100644
--- a/lib/rsa/rsa-verify.c
+++ b/lib/rsa/rsa-verify.c
@@ -234,6 +234,11 @@ int padding_pss_verify(struct image_sign_info *info,
 	uint8_t leftmost_mask;
 	struct checksum_algo *checksum = info->checksum;
 
+	if (db_len <= 0) {
+		printf("%s: invalid db length\n", __func__);
+		return -EINVAL;
+	}
+
 	/* first, allocate everything */
 	db_mask = malloc(db_len);
 	db = malloc(db_len);
-- 
2.35.1