[tbot] Testcase / Scripting JTAG Debugger

[Harald Seiler]

Hello Stefano, Wolfgang,

On Mon, 2018-12-10 at 09:32 +0100, Stefano Babic wrote:
> On 09/12/18 18:54, Wolfgang Denk wrote:
> > Hallo Harald,
> > 
> > In message <4c44dbd1e71be5555964e5e9a5c9fdd480590569.camel at denx.de> you wrote:
> > > 
> > > > - maybe even provide a generic way to "open a connection" to some
> > > >   other device and run interactive commands on it.  This could abso
> > > >   be (for example) a SSH connection to some other board which serves
> > > >   as a test partner, say when running CAN or TCP/IP tests between
> > > >   two boards.
> > > 
> > > SSH connections are already possible[1], they are used for example for the
> > > build-host.
> > 
> > Hm... it's a pity that SSH is special.  Can we not abstract fro the
> > actual communcation media and protocol?  I mean, "ssh foo", "rlogin
> > bar", "telnet baz nnn" or "cu system" more or less all do the same
> > thing: setting up a communication channel.

It's not ssh that is special, it is the heterogeneity of the other side
we are talking to.  We can't and in my opinion shouldn't even attempt to
throw these vastly different systems into one bucket.  A bash works
different than U-Boot's hush and the bdi's console probably has completely
different semantics all together.  I think it is good to make these
distinctions on the type level so mistakes can be spotted early on by a
static type checker.

> > I would find it really nice if we would leave the decision which
> > tools [cu, kermit, minicom, ...] / media [serial, TCP/IP, CAN, ...]
> > / protocols one wants to use to the end user, without needing to add
> > specific code for each of these.
> 
> If I understand well, this is already done with the Channel abstraction.

You are correct.  tbot just takes any channel, it is up to you how you
create it.  Although you channel does have to adhere to some implicit rules:

It needs to be "similar" to a serial connection in that tbot (if your config
says so) waits for U-Boot to show up and later on a getty.  If you were to
create a channel using ssh, you obviously can't wait for U-Boot to answer ...

> We define a method connect() (but this in the board setupt) to identify
> the protocol. If we think the BDI as a board to be tested, we could
> write something like:
> 
> class BDI(board.board):
> 	def connect(self) -> channel.channel
> 	return self.lh.new_channel("telnet bdi")
> 
> However, BDI is not a board. It is (like console) just another way to
> drive the device under test.

Hmm, I wouldn't equate the BDI with a board, this sounds more like a forced
abstraction to me than one that we actually benefit from.  We also can't
use the same command-exectution code on a bdi, because it has different
shell semantics.  It might make sense to refactor tbot a bit to allow more
easily defining new "shell-types" though ...

> > 
> > Is this possible?
> > 
> > > > I think is should be possible to have several such "connections"
> > > > open in a test case - console and BDI are just two examples of what
> > > > we frequently need.
> > > 
> > > Yes, you can have as many connections as you want! (Actually, at the moment it
> > > is limited by the number of channels configured for your sshd but if that ever
> > > gets to be a problem, I could implement opening secondary sessions ...)
> > 
> > Well, the BDI connection would use "telnet" protocol so I don;t
> > think this is related to the sshd configuration?
> > 

Ah, sorry, maybe I should have worded this differently.  tbot connects to the
lab-host via ssh and this ssh connection limits the number of concurrent channels.

> > > In tbot's design this looks a little different, but essentially this
> > > is what is planned.  At the moment I only have Linux and U-Boot, as
> > > soon as we add gdbserver, BDIs, etc, I need to refactor that code a
> > > little, but that's not a big deal ...
> > 
> > This is what I mean: I don;t want to add gdbserver or BDI support.
> > I just want to have the ability to connect myself to such tools.
> 
> Right - anyway, we need some way to assign a BDI to a board - something
> like in a testcase:
> 	lh = cx.enter_context(tbot.acquire_lab())
> 	b = cx.enter_context(tbot.acquire_board(lh))
> 	bdi = lh.acquire_bdi(b)
> 	bdi.exec (<raw command as in debugger's manual>)

Yes, this is a very good way to do it. (Although you forgot the context
manager ;)  The bdi needs to be bound to the physical board, because its
power is also switched alongside.

> > Actually I have no idea what else one might want to use - but I can
> > come up with a pretty long lsit - more tools / types of coneections
> > you ever want to implement as special cases :-)

No matter what hardware you want to connect to, and no matter how you do
it, you will always have to specify at least a few properties of this
hardware.  In tbot's design philosophy, this should be done in code rather
than config, which is why you would write a new class for eg the BDI.
The boilerplate is minimal, but the benefits are quite big:

* You have absolutely no constraints, you can make your machine do *anything*
* Static type-checking allows to easily spot mistakes very early on
* You don't need a DSL for configuration, everything is python, and as long as
  you can read python you can understand what the Machine does (if it is well
  written of course ...).

At the moment the only machines are U-Boot and Linux.  With debuggers now
entering the stage, this part of tbot will need some refactoring to allow
defining new machines with as little boilerplate as possible.  I'll take
a look over the next few days and report back once I have a good solution.

> Regards,
> Stefano
> 

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws at denx.de