[U-Boot] [PATCH 2/2] mtd: fix Coverity integer handling issue

[Miquel Raynal]

A Coverity robot reported an integer handling issue
(OVERFLOW_BEFORE_WIDEN) in the potentially overflowing expression:

    (mtd_div_by_ws(mtd->size, mtd) - mtd_div_by_ws(offs, mtd)) *
    mtd_oobavail(mtd, ops)

While such overflow will certainly never happen due to the numbers
handled, it is cleaner to fix this operation anyway.

The problem is that all the maths include 32-bit quantities, while the
result is stored in an explicit 64-bit value.

As maxooblen will just be compared with a size_t, let's change the
type of the variable to a size_t. This will not fix anything but will
clarify a bit the situation. Then, do an explicit cast to fix Coverity
warning.

Addresses-Coverity-ID: 184180 ("Integer handling issues")
Signed-off-by: Miquel Raynal <miquel.raynal at bootlin.com>
---
 drivers/mtd/mtdcore.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/mtd/mtdcore.c b/drivers/mtd/mtdcore.c
index fb6c779abb..df12535d32 100644
--- a/drivers/mtd/mtdcore.c
+++ b/drivers/mtd/mtdcore.c
@@ -1030,13 +1030,13 @@ static int mtd_check_oob_ops(struct mtd_info *mtd, loff_t offs,
 		return -EINVAL;
 
 	if (ops->ooblen) {
-		u64 maxooblen;
+		size_t maxooblen;
 
 		if (ops->ooboffs >= mtd_oobavail(mtd, ops))
 			return -EINVAL;
 
-		maxooblen = ((mtd_div_by_ws(mtd->size, mtd) -
-			      mtd_div_by_ws(offs, mtd)) *
+		maxooblen = ((size_t)(mtd_div_by_ws(mtd->size, mtd) -
+				      mtd_div_by_ws(offs, mtd)) *
 			     mtd_oobavail(mtd, ops)) - ops->ooboffs;
 		if (ops->ooblen > maxooblen)
 			return -EINVAL;
-- 
2.17.1