[PATCH v1 1/1] rsa: fix incorrect positive EINVAL in padding_pss_verify()

[Quentin Schulz]

Hi Aristo,

On 2/28/26 9:50 AM, Aristo Chen wrote:
> All other error paths in padding_pss_verify() return negative error
> codes (-EINVAL, -ENOMEM), which is the same as the rest of the U-Boot
> RSA stack. The positive value is inconsistent with U-Boot's convention

It is consistent with the function doc though.

> and is a potential bug: any caller that tests (ret < 0) to detect
> failure would incorrectly treat a malformed PSS signature as a success.
> 

They would also miss an error in step 14, since memcmp returns non-zero 
(but potentially positive) return value.

There's no documentation for the crypto padding verify function pointer, 
c.f. elixir.bootlin.com/u-boot/v2026.01/source/include/image.h#L1604 so 
maybe we should be explicit that it may return >0 for an error too.

The main question is, is there anyone in our code base that is checking 
< 0 for an error?

Cheers,
Quentin