[PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

[Tom Rini]

On Tue, Jul 07, 2020 at 10:57:26PM +0200, Heinrich Schuchardt wrote:

> From: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
> 
> When calculating rrtmp/rr rsa_gen_key_prop() tries to make
> (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
> (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
> with rlen being num_bits * 2
> 
> On a 4096bit key this comes down to to 257 uint32_t elements
> in rr and 256 elements in rrtmp but with the current allocation
> rr and rrtmp only have 129 uint32_t elements.
> 
> On 2048bit keys this works by chance as the defined max_rsa_size=4096
> allocates a suitable number of elements, but with an actual 4096bit key
> this results in other memory parts getting overwritten.
> 
> So as suggested by Heinrich Schuchardt just use the actual bit-size
> of the key as base for the size calculation, in turn making the code
> compatible to any future keysizes.
> 
> Suggested-by: Heinrich Schuchardt <xypron.debian at gmx.de>
> Signed-off-by: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
> Reviewed-by: Simon Glass <sjg at chromium.org>
> 
> rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
> 
> Reviewed-by: Heinrich Schuchardt <xypron.glpk at gmx.de>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200708/441a74bd/attachment.sig>