[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Simon Glass]

Hi Stephen,

On 30 November 2015 at 10:13, Stephen Warren <swarren at wwwdotorg.org> wrote:
>
> On 11/26/2015 07:52 PM, Simon Glass wrote:
>>
>> Hi Stephen,
>>
>> On 24 November 2015 at 13:28, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>>
>>> On 11/24/2015 12:04 PM, Simon Glass wrote:
>>>>
>>>>
>>>> Hi Stephen,
>>>>
>>>> On 23 November 2015 at 21:44, Stephen Warren <swarren at wwwdotorg.org>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 11/23/2015 06:45 PM, Simon Glass wrote:
>>>>>>
>>>>>>
>>>>>> On 22 November 2015 at 10:30, Stephen Warren <swarren at wwwdotorg.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 11/21/2015 09:49 AM, Simon Glass wrote:
>>>
>>>
>>>
>>>>>>>> OK I got it working thank you. It is horribly slow though - do you
>>>>>>>> know what is holding it up? For me to takes 12 seconds to run the
>>>>>>>> (very basic) tests.
>>>
>>>
>>> ..
>>>
>>>>> I put a bit of time measurement into run_command() and found that on my
>>>>> system at work, for p.send("the shell command to execute") was actually
>>>>> (marginally) slower on sandbox than on real HW, despite real HW being a
>>>>> 115200 baud serial port, and the code splitting the shell commands into
>>>>> chunks that are sent and waited for synchronously to avoid overflowing
>>>>> UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it
>>>>> seems to be non-blocking, so I don't think termios VMIN/VTIME come into
>>>>> play (setting them to 0 made no difference), and the two raw modes took
>>>>> the same time. I meant to look into pexpect's termios settings to see if
>>>>> there was anything to tweak there, but forgot today.
>>>>>
>>>>> I did do one experiment to compare expect (the Tcl version) and pexpect.
>>>>> If I do roughly the following in both:
>>>>>
>>>>> spawn u-boot (sandbox)
>>>>> wait for prompt
>>>>> 100 times:
>>>>>       send "echo $foo\n"
>>>>>       wait for "echo $foo"
>>>>>       wait for shell prompt
>>>>> send "reset"
>>>>> wait for "reset"
>>>>> send "\n"
>>>>>
>>>>> ... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I
>>>>> remove all the "wait"s, then IIRC Tcl was about 15x faster or more.
>>>>> That's a pity. Still, I'm sure as heck not going to rewrite all this in
>>>>> Tcl:-( I wonder if something similar to pexpect but more targetted at
>>>>> simple "interactive shell" cases would remove any of that overhead.
>>>>
>>>>
>>>>
>>>> It is possible that we should use sandbox in 'cooked' mode so that
>>>> lines an entered synchronously. The -t option might help here, or we
>>>> may need something else.
>>>
>>>
>>>
>>> I don't think cooked mode will work, since I believe cooked is
>>> line-buffered, yet when U-Boot emits the shell prompt there's no \n printed
>>> afterwards.
>>
>>
>> Do you mean we need fflush() after writing the prompt? If so, that
>> should be easy to arrange. We have a similar problem with the LCD, and
>> added lcd_sync().
>
>
> Anything U-Boot does will only affect its own buffer when sending into the PTY.
>
> If the test program used cooked mode for its reading side of the PTY, then even with fflush() on the sending side, I don't believe reading from the PTY would return characters until a \n appeared.

It normally works for me - do you have the PTY set up correctly?

>
> FWIW, passing "-t cooked" to U-Boot (which affects data in the other direction to the discussion above) (plus hacking the code to disable terminal-level input echoing) doesn't make any difference to the test timing. That's not particularly surprising, since the test program sends each command as a single write, so it's likely that U-Boot reads each command into its stdin buffers in one go anyway.

Yes, I'm not really sure what is going on. But we should try to avoid
unnecessary waits and delays in the test framework, and spend as much
effort as possible actually running test rather than dealing with I/O,
etc.

>
>>> FWIW, I hacked out pexpect and replaced it with some custom code. That
>>> reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time
>>> against real HW didn't seem to be affected at all. Some features like
>>> timeouts and complete error handling are still missing, but I don't think
>>> that would affect the execution time. See my github tree for the WIP patch.
>>
>>
>> Interesting, that's a big improvement. I wonder if we should look at
>> building U-Boot with SWIG to remove all these overheads? Then the
>> U-Boot command line (and any other feature we want) could become a
>> Python class. Of course that would only work for sandbox.
>
>
> SWIG doesn't seem like a good direction; it would re-introduce different paths between sandbox and non-sandbox again. One of the main benefits of the test/py/ approach is that sandbox and real HW are treated the same.

At present we don't have a sensible test framework for anything other
than sandbox, so to me the main benefit is that with your setup, we
do.

The benefit of the existing sandbox tests is that they are very fast.
We could bisect for a test failure in a few minutes. I'd like to make
sure that we can still write C tests (that are called from your
framework with results integrated into it) and that the Python tests
are also fast.

How do we move this forward? Are you planing to resend the patch with
the faster approach?

Regards,
Simon