[PATCH] tools: mkimage: make size_inc a signed type

[Rasmus Villemoes]

In the Fixes commit, I initialized size_inc from the return value of
the new fit_estimate_hash_sig_size() helper. That helper may fail and
report that by returning a negative value, but I overlooked that
size_inc had type size_t, and hence the error check doesn't work.

Change size_inc to have type int so the error check works.  Inside the
loop, it is passed to another function as a size_t parameter, but
that's fine, because we know it is non-negative, and its value may be
incremented in steps of 1024 and is capped at ~64K, so it will
certainly never overflow an int.

Fixes: 7d4eacb0e68 ("mkimage: do a rough estimate for the size needed for hashes/signatures")
Addresses-Coverity-ID: 569495: Integer handling issues  (NEGATIVE_RETURNS)
Signed-off-by: Rasmus Villemoes <ravi at prevas.dk>
---

I'm a little puzzled by the report

919     			break;
920     		}
>>>     CID 569495:         Integer handling issues  (NEGATIVE_RETURNS)
>>>     "size_inc" is passed to a parameter that cannot be negative.
921     		ret = fit_add_file_data(params, size_inc, tmpfile);
922     		if (!ret || ret != -ENOSPC)

I would instead expect it to point out the lines

	size_inc = fit_estimate_hash_sig_size(params, bakfile);
	if (size_inc < 0)
		goto err_system;

where that < 0 test is currently dead code.

Nevertheless, there is obviously an issue, and I do think this should
make Coverity happy as well.

 tools/fit_image.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/fit_image.c b/tools/fit_image.c
index 8717dc9a3b1..ad0ffa39c6a 100644
--- a/tools/fit_image.c
+++ b/tools/fit_image.c
@@ -846,7 +846,7 @@ static int fit_handle_file(struct image_tool_params *params)
 	char tmpfile[MKIMAGE_MAX_TMPFILE_LEN];
 	char bakfile[MKIMAGE_MAX_TMPFILE_LEN + 4] = {0};
 	char cmd[MKIMAGE_MAX_DTC_CMDLINE_LEN];
-	size_t size_inc;
+	int size_inc;
 	int ret = EXIT_FAILURE;
 
 	/* Flattened Image Tree (FIT) format  handling */
-- 
2.50.0