[tbot] [Discussion] Calling tbot from within the source directory

Sat Dec 1 12:20:01 UTC 2018

Hi,

Quoting Stefano Babic (2018-11-30 10:10:57)
> Hi Claudius,
> 
> On 29/11/18 13:46, Claudius Heine wrote:
> > Hi Harald,
> > 
> > Quoting Harald Seiler (2018-11-29 12:33:42)
> >> On Thu, 2018-11-29 at 10:47 +0100, Stefano Babic wrote:
> >>> On 29/11/18 10:41, Claudius Heine wrote:
> >>>> Just a suggestion from me, calling tbot like this:
> >>>>
> >>>>   tbot -c config.ini my-testcase
> >>>>
> >>>> Where the ini file would contain something like this:
> >>>>
> >>>>   [tbot]
> >>>>   verbose=2
> >>>>   cwd=/home/user/work/tbot/tbot-myproduct
> >>>>   lab=lab/my-lab.py
> >>>>   board=board/my-board.py
> >>>>   flags=flag1 flag2
> >>>>   testcase=my-default-testcase
> >>>>
> >>>>   [project]
> >>>>   sources=work/source/my-project-source
> >>>>   outdir=work/source/my-project-source/out
> >>>>   nfsdir=/srv/nfs/my-project/user/
> >>>>   tftpdir=/srv/tftp/my-project/user/
> >>>>
> >>>> What do you think?
> >>>
> >>> See start of the thread, it was already proposed, but Harald prefers to
> >>> avoid such as configuration file.
> >>
> >> I think, I should elaborate on why:
> >>
> >> As I mentioned in my talk, the version of tbot you know is actually the second
> >> rewrite.  The first one had an extensive configuration system, supporting
> >> overlaying different config files, having configs depend on each other, and
> >> generating config values on the fly from python code.  The reason it got so
> >> complex was because I stumbled upon a lot of edge-cases that needed special
> >> handling.
> >>
> >> It worked.  But it wasn't pretty.  And it was really hard to reason about.
> >> I was just short of adding something like `bitbake -e` which would allow
> >> you to trace where each value came from.
> > 
> > I agree that configuration can be done very complicated, but there is
> > always some middle ground. Just because we don't like 100% make
> > everything configurable and overwriteable and what ever, 10% or 50%
> > configurable might still be nice to have.
> 
> Just a new point: configurations like we thought (a .tbotrc, for
> example) can often hide an issue an make a comparison between working
> and not working systems difficult. As tbot-new is still young, I would
> agree to try to go on without introducing too many variables: each
> configuration file introduces different behaviors. We can also think
> about if we can reach what you want without configuration files.

Ok, but that is an argument about implicitly used configuration files.
Not something against:

    tbot -c tbot.cfg

I remember Harald saying to me that he will investigate if the current
'flag' system can be expanded to provide some configuration options as
well. Maybe I misunderstood him or don't remember correctly, but if he
does that, then we would have the option to set some values via the
command line that will be available in the testcases.

If that is the case and gets implemented, then the command line of tbot
might become even larger. So allowing just to set the default command options
of tbot in a configuration file that is used like I shown above, might
be a good middle ground here.

Thinking about the flag system here, that is a bit of a strange design
choice. It is currently some sort of configuration via the command line that is
available in the python files, but only fits one bit of information.
I would say that is should eighter be removed with the hint that for
each customization a seperate class should be written or it should allow flags
to contain an optional arbitrary value like Harald suggested IIRC.
Because in its current form I think that this might cause a lot of frustration
because its so limited in its use. But if I understood him correctly and he
thinks about expanding it, then this is a non issue and welcomed by me :)

> > Eighter use some simple config parser or just eval
> > some python file to get your configuration in case of tbot. My hope is that if
> > tbot provides some simple mechanism to get key-value pairs into testcases
> > then people will not try to implement their own in each and every
> > testcase making them incompatible to each other.
> 
> But maybe we can have some "tbot libraries" or "3rd party" components
> that are not part of tbot core, as many projects do.

Hopefully all the changes to get this sort of thing working are small
and easy enough that we don't need some external library to house them.

I would reserve additional libraries for more complicated and completly
experimental stuff, like Windows support or what ever ;)

Eighter way, if we do that, then we might have to think about a
plugin system for tbot. And that is a whole different can of worms to
open...

> > Why should they want complicated configuration mechanism if they already have
> > powerful composition in this version of tbot? I am not talking about
> > takeing that away, but just about providing some options to have some
> > project/user configuration that is valid for tbot based test
> > environments.
> > 
> > Also tbot command can be shorter even without alias or wrapper scripts, typing
> > less it IMO always better especially with tbot were that is the ultimate
> > goal :)
> > 
> > 
> > However if I understand Heiko correctly then all of this can be easily
> > done in a lab configuration. So we would have a "Lab" for every
> > project/user.
> 
> I think this is a great idea and we can try to explore better (or for
> me, to better understand). IMHO lab is the "setup" fo the project, and I
> guess your use case can be done via methods of the lab object. What about a
>         lh = cx.enter_context(tbot.acquire_lab())
>         src = lh.get_sources()
> 
> > Then it should be easy to reuse other "Lab"s to recreate
> > some sort of project/userconfiguration. I don't know how easy that is
> 
> Me too

I am currently just unsure about the best way to use tbot in my dev
environments and how to change tbot in order for it to better integrated
within.

For example if I want to use and expand the lab definitions in tbot-denx
from my tbot-project, does the tbot-project needs to know the paths to
the tbot-denx repository? Should tbot-denx be a submodule of
tbot-project? What if I want to use it on a different lab?

Mabye the inheritance be defined via the tbot command line? Like:

    tbot -l my-project.py:/path/to/my/user/tbot.py:/path/to/tbot-denx/labs/pollux.py ...

Just to be more flexible. That needs to be implemented in tbot first of
course. But I don't know if that is a good idea or not.

My problem here I think is that lab-specific configuration, user specific
configuration and project specific configuration are sort of orthogonal and
all of them come together at the command line of tbot.

In this example:

my-project.py:
  - contains the paths to the project sources

/path/to/my/user/tbot.py:
  - contains customization to the base lab classes, so that login etc.
    works (ssh-key key path, choosen build server etc.)

/path/to/tbot-denx/labs/pollux.py:
  - contains base defintions of the available lab hardware/build server

I think thats just a lot of composition necessary. So that either has to
be so easy, like specifying it via the command line or handled
seperatly. That was one thing we learned with kas, at some point
creating a file for every permutation is just to much, so we implemented
a mechanism to combine configuration files with colons in the command
line.

Maybe I am just missing a good example of how things should work if you
have multiple users build multiple projects in different labs and
trying to share as much as possible with each other.

So enough rambled on, have a nice weekend,
Claudius

--
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch at denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: signature
URL: <http://lists.denx.de/pipermail/tbot/attachments/20181201/9a2ce0aa/attachment.sig>