[U-Boot] [RFC][DO NOT APPLY] binman: Switch to 'python-coverage'

[Simon Glass]

Hi Tom,

On 1 June 2018 at 09:36, Tom Rini <trini at konsulko.com> wrote:
> On Fri, Jun 01, 2018 at 08:27:03AM -0600, Simon Glass wrote:
>
>> Hi Tom,
>>
>> On 26 May 2018 at 11:06, Tom Rini <trini at konsulko.com> wrote:
>> > The most portable way to get access to coverage is to invoke it as
>> > 'python-coverage'.
>> >
>> > Cc: Simon Glass <sjg at chromium.org>
>> > Signed-off-by: Tom Rini <trini at konsulko.com>
>> > ---
>> > So, here's the set of problems I have.  First, running 'pip install
>> > coverage' doesn't get me a 'coverage' binary, so I've just never made
>> > good use of 'make tests' and so never have these run.  It also means
>> > they aren't strictly in travis-ci, so they're never run automatically.
>> > With this patch, make tests executes for me, and now fails noting that a
>> > lot of things don't have coverage.  I'm honestly not sure what the best
>> > path forward here is.
>> > ---
>> >  tools/binman/README    | 3 +--
>> >  tools/binman/binman.py | 4 ++--
>> >  2 files changed, 3 insertions(+), 4 deletions(-)
>> >
>>
>> Reviewed-by: Simon Glass <sjg at chromium.org>
>>
>> For me this patch works fine. The -T option works correctly.
>>
>> Typing 'python-coverage html' produces this report:
>>
>> https://pste.eu/p/NI7q.html
>>
>> Note that your patch needs to update this line too:
>>
>>         print "Type 'coverage html' to get a report in htmlcov/index.html"
>>
>> Can you send me the output you see?
>
> So, I need to amend this a bit.  On my Docker-based ubuntu 16.04 setup,
> I see as non-fatal (and not coverage related, but binman test):
> <unittest.result.TestResult run=79 errors=0 failures=0>
> <unittest.result.TestResult run=11 errors=0 failures=4>
> Traceback (most recent call last):
>   File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 218, in testGood
>     self.assertEqual(result.ok, True)
> AssertionError: False != True
>
> Traceback (most recent call last):
>   File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 252, in testIndent
>     self.assertEqual(len(result.problems), 1)
> AssertionError: 2 != 1
>
> Traceback (most recent call last):
>   File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 230, in testNoSignoff
>     self.assertEqual(len(result.problems), 1)
> AssertionError: 2 != 1
>
> Traceback (most recent call last):
>   File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 241, in testSpaces
>     self.assertEqual(len(result.problems), 3)
> AssertionError: 4 != 3
>

This is new to the SPDX change in checkpatch. I'll send a pull request
with a fix for this soon.

> Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/
> <unittest.result.TestResult run=35 errors=0 failures=1>
> Traceback (most recent call last):
>   File "/home/trini/work/u-boot/u-boot/tools/buildman/test.py", line 426, in testToolchainDownload
>     self.toolchains.LocateArchUrl('arm'))
> AssertionError: 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.9.0/x86_64-gcc-4.9.0-nolibc_arm-unknown-linux-gnueabi.tar.xz' != 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/x86_64-gcc-7.3.0-nolibc_arm-linux-gnueabi.tar.xz'

This is due to the version number changes. It needs a fix to the test.

>
> <unittest.result.TestResult run=12 errors=0 failures=0>
> <unittest.result.TestResult run=79 errors=0 failures=0>
>
> And then the coverage part works fine.  With the changes in this RFC
> patch so that my desktop itself could run coverage stuff I also see:
> Missing tests for u_boot_with_ucode_ptr.py, u_boot_spl_bss_pad.py, entry.py, u_boot_spl.py, u_boot_spl_with_ucode_ptr.py, u_boot_spl_dtb.py, u_boot_img.py, intel_vbt.py, u_boot_dtb.py, intel_vga.py, intel_descriptor.py, u_boot_dtb_with_ucode.py, intel_cmc.py, x86_start16.py, u_boot_nodtb.py, u_boot.py, intel_fsp.py, blob.py, intel_mrc.py, u_boot_spl_nodtb.py, u_boot_ucode.py, x86_start16_spl.py, intel_me.py
> ValueError: Test coverage failure
> Tests FAILED
> Makefile:1710: recipe for target 'tests' failed
> make: *** [tests] Error 1

This looks like it is not getting any coverage info at all. Perhaps
there is a complete test failure? Can you try:

binman -t

and see what tests fail?

>
> So I guess there's something wrong with trying to use coverage this way,
> rather than via pip.

Maybe, but it works OK for me. Given that it works fine on travis
(right?) I think we should apply your patch, with the additional
change I mentioned in my previous email.

Regards,
Simon