[PATCH v4 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

[Simon Glass]

On Fri, 22 May 2020 at 08:13, Heiko Stuebner <heiko at sntech.de> wrote:
>
> From: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
>
> When calculating rrtmp/rr rsa_gen_key_prop() tries to make
> (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
> (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
> with rlen being num_bits * 2
>
> On a 4096bit key this comes down to to 257 uint32_t elements
> in rr and 256 elements in rrtmp but with the current allocation
> rr and rrtmp only have 129 uint32_t elements.
>
> On 2048bit keys this works by chance as the defined max_rsa_size=4096
> allocates a suitable number of elements, but with an actual 4096bit key
> this results in other memory parts getting overwritten.
>
> so double the number of elements in rr and rrtmp so that it matches
> the needed number and should increase nicely if max_rsa_size gets
> increased in the future.
>
> Signed-off-by: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
> ---
> changes in v4:
> - new patch
>
>  lib/rsa/rsa-keyprop.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Simon Glass <sjg at chromium.org>