[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Simon Glass]

Hi Stephen,

On 24 November 2015 at 13:28, Stephen Warren <swarren at wwwdotorg.org> wrote:
> On 11/24/2015 12:04 PM, Simon Glass wrote:
>>
>> Hi Stephen,
>>
>> On 23 November 2015 at 21:44, Stephen Warren <swarren at wwwdotorg.org>
>> wrote:
>>>
>>> On 11/23/2015 06:45 PM, Simon Glass wrote:
>>>>
>>>> On 22 November 2015 at 10:30, Stephen Warren <swarren at wwwdotorg.org>
>>>> wrote:
>>>>>
>>>>> On 11/21/2015 09:49 AM, Simon Glass wrote:
>
>
>>>>>> OK I got it working thank you. It is horribly slow though - do you
>>>>>> know what is holding it up? For me to takes 12 seconds to run the
>>>>>> (very basic) tests.
>
> ..
>
>>> I put a bit of time measurement into run_command() and found that on my
>>> system at work, for p.send("the shell command to execute") was actually
>>> (marginally) slower on sandbox than on real HW, despite real HW being a
>>> 115200 baud serial port, and the code splitting the shell commands into
>>> chunks that are sent and waited for synchronously to avoid overflowing
>>> UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it
>>> seems to be non-blocking, so I don't think termios VMIN/VTIME come into
>>> play (setting them to 0 made no difference), and the two raw modes took
>>> the same time. I meant to look into pexpect's termios settings to see if
>>> there was anything to tweak there, but forgot today.
>>>
>>> I did do one experiment to compare expect (the Tcl version) and pexpect.
>>> If I do roughly the following in both:
>>>
>>> spawn u-boot (sandbox)
>>> wait for prompt
>>> 100 times:
>>>      send "echo $foo\n"
>>>      wait for "echo $foo"
>>>      wait for shell prompt
>>> send "reset"
>>> wait for "reset"
>>> send "\n"
>>>
>>> ... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I
>>> remove all the "wait"s, then IIRC Tcl was about 15x faster or more.
>>> That's a pity. Still, I'm sure as heck not going to rewrite all this in
>>> Tcl:-( I wonder if something similar to pexpect but more targetted at
>>> simple "interactive shell" cases would remove any of that overhead.
>>
>>
>> It is possible that we should use sandbox in 'cooked' mode so that
>> lines an entered synchronously. The -t option might help here, or we
>> may need something else.
>
>
> I don't think cooked mode will work, since I believe cooked is
> line-buffered, yet when U-Boot emits the shell prompt there's no \n printed
> afterwards.

Do you mean we need fflush() after writing the prompt? If so, that
should be easy to arrange. We have a similar problem with the LCD, and
added lcd_sync().

>
> FWIW, I hacked out pexpect and replaced it with some custom code. That
> reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time
> against real HW didn't seem to be affected at all. Some features like
> timeouts and complete error handling are still missing, but I don't think
> that would affect the execution time. See my github tree for the WIP patch.

Interesting, that's a big improvement. I wonder if we should look at
building U-Boot with SWIG to remove all these overheads? Then the
U-Boot command line (and any other feature we want) could become a
Python class. Of course that would only work for sandbox.

Regards,
Simon