Frustrations of running testing

[Tom Rini]

On Fri, Feb 19, 2021 at 12:28:16PM -0600, Alex G. wrote:

> Hi,
> 
> I keep being hit by two frustrating issues when trying to run tests. My
> expectation is that I can run a test at any time when working on something
> -- usually to check the correctness of that something. That's not the case
> today.
> 
> 
> The first one is an error message about mrproper:
> 
> 	$ ./test/py/test.py --bd sandbox_spl
> 	.. is not clean, please run 'make mrproper'
> 
> I don't think this message is very useful. Running 'mrproper' is out of the
> question, as it would wipe out the branch I'm currently working on. So what
> I end up doing is deleting the mrimproper check from the Makefile. And while
> it's a fast workaround, it dirties the tree and gets in the way of rebasing
> patches or working with git.
> 
> I can continue just fine without 'mrproper death' so I really don't
> understand the over-abundance of caution in breaking the build.

So, I looked a bit here.  The problem is that our logic in the top-level
Makefile for "Is this a separate srctree/objtree build?" is very out of
date / wrong.  Re-syncing this with a current Linux Makefile should
resolve this problem.

> The second issue is the how the tests are trying to build graphics:
> 
> 	$ ./test/py/test.py --bd sandbox_spl -k "whatever"
> 	make[1]: sdl2-config: Command not found
> 	../arch/sandbox/cpu/sdl.c:10:10: fatal error: SDL2/SDL.h:
> 	No such file or directory
> 	   10 | #include <SDL2/SDL.h>
> 
> The solution is to set NO_SDL=1, but this is far from obvious. The first
> instinct is always to look up the command help:
> 
> 	$ ./test/py/test.py --help
> 
> But this really isn't at all useful. A simple grep for sdl2-config also
> doesn't immediately reveal the solution. Most u-boot tools get their
> arguments via the commandline. Thus, a reasonably competent u-boot developer
> will not think that environment variables are the solution. Ergo,
> environment variables are not the ideal way to solve this.
> 
> Maybe we could have a commandline option, and at the very least, catch this
> error and print something useful on the console.
> 
> 
> I'm not sure what the preferred way would be to solve the above. For me,
> these issues cause a significant enough disruption to my workflow, that I am
> very likely to not run tests regularly. I suspect I'm not alone.

Part of the problem is that if you don't have graphics enabled in
sandbox, you in turn don't get other tests run I believe.  A check in
arch/sandbox/config.mk for NO_SDL being unset AND not having
$(SDL_CONFIG) exist would provide a better user experience, yes.

Thanks for the feedback!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210219/d5917e34/attachment.sig>