[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/23/19 12:17 PM, Tom Rini wrote:
> On Wed, Oct 23, 2019 at 12:04:30PM -0600, Stephen Warren wrote:
>> On 10/23/19 11:29 AM, Stephen Warren wrote:
>>> On 10/23/19 11:12 AM, Tom Rini wrote:
>>>> On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
>>>>> On 10/23/19 10:58 AM, Tom Rini wrote:
>>>>>> On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
>>>>>>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>>>>>>> Now that we have moved to being based on pytest for
>>>>>>>> python3 we need to
>>>>>>>> make our test.py wrapper more robust in terms of only calling python3
>>>>>>>> rather than possibly finding and using python2.  To
>>>>>>>> do this, change from
>>>>>>>> execvp()'ing pytest to invoking the package itself
>>>>>>>> via python.  In the
>>>>>>>> event that pytest is unavailable we still get a user-friendly error:
>>>>>>>
>>>>>>> I guess if this works, it's fine. But I think it'd work
>>>>>>> just as well to
>>>>>>> simply s/py.test/py.test-3/ in the existing code?
>>>>>>
>>>>>> We can't because pip only provides "py.test" / "pytest" binaries.
>>>>>
>>>>> Hmm. It sucks that virtualenv does things differently to Ubuntu
>>>>> packaging
>>>>> then. That means this series will probably break my test system, which
>>>>> doesn't use virtualenv. I guess I'll apply it and see.
>>>>
>>>> This should be what works everywhere tho.  Instead of hoping that
>>>> "py.test" will be the right python version we tell python3 to use the
>>>> pytest module it has.  So if you have system python3-pytest, that works,
>>>> if you (python3) pip install pytest locally, that works and if you
>>>> virtualenv + python3 + pip install pytest, that works.
>>>
>>> On Ubuntu 16.04, with everything installed from Ubuntu packages rather
>>> than using virtualenv, I get this:
>>>
>>>> + ./src/u-boot/test/py/test.py --bd sandbox --build-dir /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox
>>>> -k 'not sleep'
>>>> Traceback (most recent call last):
>>>>    File "./src/u-boot/test/py/test.py", line 20, in <module>
>>>>      sys.exit(load_entry_point('pytest', 'console_scripts',
>>>> 'pytest')(args))
>>>>    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
>>>> line 542, in load_entry_point
>>>>      return get_distribution(dist).load_entry_point(group, name)
>>>>    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
>>>> line 2568, in load_entry_point
>>>>      raise ImportError("Entry point %r not found" % ((group, name),))
>>>> ImportError: Entry point ('console_scripts', 'pytest') not found
>>>
>>>> [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
>>>> ii  python-pytest    2.8.7-4
>>>> ii  python3-pytest   2.8.7-4
>>>
>>> I guess that's much older than the pytest in requirements.txt (5.2.1!!).
>>> While I think we can require a newish version of Python (3.5) since
>>> that's widely available on supported OSs, I don't think we want to force
>>> people to have a very new version of all the packages (which supported
>>> distros don't package yet), or change they way they run pytest (by
>>> requiring use of a virtualenv and requirements.txt) since that mechanism
>>> isn't supported on old branches of U-Boot, which will then require all
>>> kinds of special cases in scripts that invoke the U-Boot test system.
>>
>> Ah. The issue is the name of the entry point; the last parameter to
>> load_entry_point(). "py.test" is supported everywhere, but only newer
>> versions of pytesst have a "pytest" entry point. Changing that parameter to
>> "py.test" makes the wrapper script work. Now every other test fails; I will
>> see if there's an easy fix for that or not.
> 
> We're going to have to make a cut-off point somewhere it looks like.
> pytest 2.8.7 (which we had pinned to) is incompatible with pytest 4.x
> and throws a lot of warnings I believe on pytest 3.x..
> 
> What I really want to avoid is the problem we have today where we've
> actively pinned ourselves to a now quite out of date version and have a
> lot of problems showing up.
> 
> So we aren't going to have one environment that works for history and
> future but I am hoping / trying to plan such that if we have
> requirements.txt we can say from here until the next new best practices
> (which today would be using pipenv, but that starts out with pip install
> pipenv, sigh...) we can say to use requirements.txt and for history,
> python2 + pytest 2.8.7 and local setups could even dump that in to a
> requirements.txt file if that makes scripting easier.

Indeed the pytest 4 changes were a lot of the problem, but there was 
plenty more. It turned out to be too painful to fix it all.

After a lot of pain, I've switched my system over to using a virtualenv 
when there's a test/py/requirements.txt file in the source. Using a 
virtualenv broke some of the HW remote control scripts, which were 
written for Python2 but just said "#!/usr/bin/env python". Fixing that 
to request Python 2 explicitly fixed that issue. Another pain was that 
the ftdi1 module (used by the HW remote control scripts) isn't available 
from pip so can't be installed in a virtualenv. Luckily explicitly using 
python2 for those scripts makes this irrelevant, since system libraries 
are used, since the python2 interpreter doesn't come from the virtualenv 
in that case. However, if that didn't fix the issue, then I'd have 
needed extra options to set up the virtualenv in order to rely on 
Ubuntu-provided package for the module. I'm quoting them below just in 
case someone searching the archives finds this later in the future!

> +if [ -f ./src/u-boot/test/py/requirements.txt ]; then
> +    rm -rf "${venv_dir}"
> +    # --system-site-packages: Need ftdi1 module which is not available via pip,
> +    #   but is installed on the system
> +    virtualenv --system-site-packages -p /usr/bin/python3 "${venv_dir}"
> +    . "${venv_dir}/bin/activate"
> +    # -U: upgrade: Ensure the package is installed in the venv, at the version
> +    #   requested in requirements.txt, even if already available on the host.
> +    pip install -U -r ./src/u-boot/test/py/requirements.txt
> +fi

So in summary, with the Python 3.5 fix I already sent, then this series,
Tested-by: Stephen Warren <swarren at nvidia.com>