[U-Boot] [PATCH] test/py: support running sandbox under gdbserver

[Stephen Warren]

On 02/06/2016 01:39 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 6 February 2016 at 13:34, Stephen Warren <swarren at wwwdotorg.org> wrote:
>> On 02/06/2016 01:30 PM, Simon Glass wrote:
>>> On 4 February 2016 at 16:11, Stephen Warren <swarren at wwwdotorg.org> wrote:
>>>> Implement command--line option --gdbserver COMM, which does two things:
>>>>
>>>> a) Run the sandbox process under gdbserver, using COMM as gdbserver's
>>>>     communication channel.
>>>>
>>>> b) Disables all timeouts, so that if U-Boot is halted under the debugger,
>>>>     tests don't fail. If the user gives up in the middle of a debugging
>>>>     session, they can simply CTRL-C the test script to abort it.
>>>>
>>>> This allows easy debugging of test failures without having to manually
>>>> re-create the failure conditions. Usage is:
>>>>
>>>> Window 1:
>>>> ./test/py/test.py --bd sandbox --gdbserver localhost:1234
>>>>
>>>> Window 2:
>>>> gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
>>>>
>>>> When using this option, it likely makes sense to use pytest's -k option
>>>> to limit the set of tests that are executed.
>>>>
>>>> Simply running U-Boot directly under gdb (rather than gdbserver) was
>>>> also considered. However, this was rejected because:
>>>>
>>>> a) gdb's output would then be processed by the test script, and likely
>>>>     confuse it causing false failures.
>>>>
>>>> b) pytest by default hides stdout from tests, which would prevent the
>>>>     user from interacting with gdb.
>>>>
>>>>     While gdb can be told to redirect the debugee's stdio to a separate
>>>>     PTY, this would appear to leave gdb's stdio directed at the test
>>>>     scripts and the debugee's stdio directed elsewhere, which is the
>>>>     opposite of the desired effect. Perhaps some complicated PTY muxing
>>>>     and process hierarchy could invert this. However, the current scheme
>>>>     is simple to implement and use, so it doesn't seem worth complicating
>>>>     matters.
>>>>
>>>> c) Using gdbserver allows arbitrary debuggers to be used, even those with
>>>>     a GUI. If the test scripts invoked the debugger themselves, they'd have
>>>>     to know how to execute arbitary applications. While the user could hide
>>>>     this all in a wrapper script, this feels like extra complication.
>>>>
>>>> An interesting future idea might be a --gdb-screen option, which could
>>>> spawn both U-Boot and gdb separately, and spawn the screen into a newly
>>>> created window under screen. Similar options could be envisaged for
>>>> creating a new xterm/... too.
>>>>
>>>> --gdbserver  currently only supports sandbox, and not real hardware.
>>>> That's primarily because the test hooks are responsible for all aspects of
>>>> hardware control, so there's nothing for the test scripts themselves can
>>>> do to enable gdbserver on real hardware. We might consider introducing a
>>>> separate --disable-timeouts option to support use of debuggers on real
>>>> hardware, and having --gdbserver imply that option.
>>>>
>>>> Signed-off-by: Stephen Warren <swarren at nvidia.com>
>>>> ---
>>>>   test/py/conftest.py               |  8 ++++++++
>>>>   test/py/tests/test_sleep.py       |  7 ++++---
>>>>   test/py/u_boot_console_base.py    |  3 ++-
>>>>   test/py/u_boot_console_sandbox.py |  5 ++++-
>>>>   test/py/u_boot_spawn.py           | 12 ++++++++----
>>>>   5 files changed, 26 insertions(+), 9 deletions(-)
>>>
>>> Can you please add info about this to the docs?
>>>
>>> Also for me this worked up to the point where it ran the
>>> test_sandbox_exit.py test. Then the gdb process said that U-Boot
>>> exited normally. Is that test not compatible with this feature?
>>
>> The sandbox_exit test deliberately causes the sandbox process to exit,
>> to make sure that the "reset" command and "typing" Ctrl-C work. To
>> continue the test, simply re-run gdb to re-attach to the new gdbserver
>> and U-Boot process.
>
> That's a bit annoying. Perhaps we should have a flag that disabled
> such tests? Or perhaps a way to specify what tests are run?

I would expect the user to only run the specific test that needed 
debugging in order to speed up the whole process. So, unless someone was 
debugging the reset tests, they wouldn't notice this.

There's already a standard pytest option to select which tests to run; -k.

> Anyway, can you add this to the docs too?

Sure.