[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Stephen Warren]

On 11/26/2015 07:52 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 24 November 2015 at 13:28, Stephen Warren <swarren at wwwdotorg.org> wrote:
>> On 11/24/2015 12:04 PM, Simon Glass wrote:
>>>
>>> Hi Stephen,
>>>
>>> On 23 November 2015 at 21:44, Stephen Warren <swarren at wwwdotorg.org>
>>> wrote:
>>>>
>>>> On 11/23/2015 06:45 PM, Simon Glass wrote:
>>>>>
>>>>> On 22 November 2015 at 10:30, Stephen Warren <swarren at wwwdotorg.org>
>>>>> wrote:
>>>>>>
>>>>>> On 11/21/2015 09:49 AM, Simon Glass wrote:
>>
>>
>>>>>>> OK I got it working thank you. It is horribly slow though - do you
>>>>>>> know what is holding it up? For me to takes 12 seconds to run the
>>>>>>> (very basic) tests.
>>
>> ..
>>
>>>> I put a bit of time measurement into run_command() and found that on my
>>>> system at work, for p.send("the shell command to execute") was actually
>>>> (marginally) slower on sandbox than on real HW, despite real HW being a
>>>> 115200 baud serial port, and the code splitting the shell commands into
>>>> chunks that are sent and waited for synchronously to avoid overflowing
>>>> UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it
>>>> seems to be non-blocking, so I don't think termios VMIN/VTIME come into
>>>> play (setting them to 0 made no difference), and the two raw modes took
>>>> the same time. I meant to look into pexpect's termios settings to see if
>>>> there was anything to tweak there, but forgot today.
>>>>
>>>> I did do one experiment to compare expect (the Tcl version) and pexpect.
>>>> If I do roughly the following in both:
>>>>
>>>> spawn u-boot (sandbox)
>>>> wait for prompt
>>>> 100 times:
>>>>       send "echo $foo\n"
>>>>       wait for "echo $foo"
>>>>       wait for shell prompt
>>>> send "reset"
>>>> wait for "reset"
>>>> send "\n"
>>>>
>>>> ... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I
>>>> remove all the "wait"s, then IIRC Tcl was about 15x faster or more.
>>>> That's a pity. Still, I'm sure as heck not going to rewrite all this in
>>>> Tcl:-( I wonder if something similar to pexpect but more targetted at
>>>> simple "interactive shell" cases would remove any of that overhead.
>>>
>>>
>>> It is possible that we should use sandbox in 'cooked' mode so that
>>> lines an entered synchronously. The -t option might help here, or we
>>> may need something else.
>>
>>
>> I don't think cooked mode will work, since I believe cooked is
>> line-buffered, yet when U-Boot emits the shell prompt there's no \n printed
>> afterwards.
>
> Do you mean we need fflush() after writing the prompt? If so, that
> should be easy to arrange. We have a similar problem with the LCD, and
> added lcd_sync().

Anything U-Boot does will only affect its own buffer when sending into 
the PTY.

If the test program used cooked mode for its reading side of the PTY, 
then even with fflush() on the sending side, I don't believe reading 
from the PTY would return characters until a \n appeared.

FWIW, passing "-t cooked" to U-Boot (which affects data in the other 
direction to the discussion above) (plus hacking the code to disable 
terminal-level input echoing) doesn't make any difference to the test 
timing. That's not particularly surprising, since the test program sends 
each command as a single write, so it's likely that U-Boot reads each 
command into its stdin buffers in one go anyway.

>> FWIW, I hacked out pexpect and replaced it with some custom code. That
>> reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time
>> against real HW didn't seem to be affected at all. Some features like
>> timeouts and complete error handling are still missing, but I don't think
>> that would affect the execution time. See my github tree for the WIP patch.
>
> Interesting, that's a big improvement. I wonder if we should look at
> building U-Boot with SWIG to remove all these overheads? Then the
> U-Boot command line (and any other feature we want) could become a
> Python class. Of course that would only work for sandbox.

SWIG doesn't seem like a good direction; it would re-introduce different 
paths between sandbox and non-sandbox again. One of the main benefits of 
the test/py/ approach is that sandbox and real HW are treated the same.