[PATCH v5 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

[Heinrich Schuchardt]

On 25.05.20 22:43, Philipp Tomsich wrote:
>
>
>> On 22.05.2020, at 16:19, Heiko Stuebner <heiko at sntech.de> wrote:
>>
>> From: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
>>
>> When calculating rrtmp/rr rsa_gen_key_prop() tries to make
>> (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
>> (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
>> with rlen being num_bits * 2
>>
>> On a 4096bit key this comes down to to 257 uint32_t elements
>> in rr and 256 elements in rrtmp but with the current allocation
>> rr and rrtmp only have 129 uint32_t elements.
>>
>> On 2048bit keys this works by chance as the defined max_rsa_size=4096
>> allocates a suitable number of elements, but with an actual 4096bit key
>> this results in other memory parts getting overwritten.
>>
>> so double the number of elements in rr and rrtmp so that it matches
>> the needed number and should increase nicely if max_rsa_size gets
>> increased in the future.
>>
>> Signed-off-by: Heiko Stuebner <heiko.stuebner at theobroma-systems.com>
>
> Reviewed-by: Philipp Tomsich <philipp.tomsich at theobroma-systems.com>
>

Why would we allocate memory according to max_rsa_size and not according
to the actual size (*prop)->num_bits = (rsa_key.n_sz - i) * 8;

Who stops a user from using 8192 or 16384 bits? We should avoid any
constant here if we do not validate that it is not exceeded.

openssl genrsa -out mykey.pem 8192
works fine for me.

Best regards

Heinrich