[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Stephen Warren]

On 11/24/2015 12:04 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 23 November 2015 at 21:44, Stephen Warren <swarren at wwwdotorg.org> wrote:
>> On 11/23/2015 06:45 PM, Simon Glass wrote:
>>> On 22 November 2015 at 10:30, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>>> On 11/21/2015 09:49 AM, Simon Glass wrote:

>>>>> OK I got it working thank you. It is horribly slow though - do you
>>>>> know what is holding it up? For me to takes 12 seconds to run the
>>>>> (very basic) tests.
..
>> I put a bit of time measurement into run_command() and found that on my
>> system at work, for p.send("the shell command to execute") was actually
>> (marginally) slower on sandbox than on real HW, despite real HW being a
>> 115200 baud serial port, and the code splitting the shell commands into
>> chunks that are sent and waited for synchronously to avoid overflowing
>> UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it
>> seems to be non-blocking, so I don't think termios VMIN/VTIME come into
>> play (setting them to 0 made no difference), and the two raw modes took
>> the same time. I meant to look into pexpect's termios settings to see if
>> there was anything to tweak there, but forgot today.
>>
>> I did do one experiment to compare expect (the Tcl version) and pexpect.
>> If I do roughly the following in both:
>>
>> spawn u-boot (sandbox)
>> wait for prompt
>> 100 times:
>>      send "echo $foo\n"
>>      wait for "echo $foo"
>>      wait for shell prompt
>> send "reset"
>> wait for "reset"
>> send "\n"
>>
>> ... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I
>> remove all the "wait"s, then IIRC Tcl was about 15x faster or more.
>> That's a pity. Still, I'm sure as heck not going to rewrite all this in
>> Tcl:-( I wonder if something similar to pexpect but more targetted at
>> simple "interactive shell" cases would remove any of that overhead.
>
> It is possible that we should use sandbox in 'cooked' mode so that
> lines an entered synchronously. The -t option might help here, or we
> may need something else.

I don't think cooked mode will work, since I believe cooked is 
line-buffered, yet when U-Boot emits the shell prompt there's no \n 
printed afterwards.

FWIW, I hacked out pexpect and replaced it with some custom code. That 
reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time 
against real HW didn't seem to be affected at all. Some features like 
timeouts and complete error handling are still missing, but I don't 
think that would affect the execution time. See my github tree for the 
WIP patch.