[tbot] [Discussion] Calling tbot from within the source directory

[Harald Seiler]

Hello Claudius,

On Sat, 2018-12-01 at 13:20 +0100, Claudius Heine wrote:
> Hi,
> 
> Quoting Stefano Babic (2018-11-30 10:10:57)
> > Hi Claudius,
> > 
> > On 29/11/18 13:46, Claudius Heine wrote:
> > > Hi Harald,
> > > 
> > > Quoting Harald Seiler (2018-11-29 12:33:42)
[...]
> > > > It worked.  But it wasn't pretty.  And it was really hard to reason about.
> > > > I was just short of adding something like `bitbake -e` which would allow
> > > > you to trace where each value came from.
> > > 
> > > I agree that configuration can be done very complicated, but there is
> > > always some middle ground. Just because we don't like 100% make
> > > everything configurable and overwriteable and what ever, 10% or 50%
> > > configurable might still be nice to have.
> > 
> > Just a new point: configurations like we thought (a .tbotrc, for
> > example) can often hide an issue an make a comparison between working
> > and not working systems difficult. As tbot-new is still young, I would
> > agree to try to go on without introducing too many variables: each
> > configuration file introduces different behaviors. We can also think
> > about if we can reach what you want without configuration files.
> 
> Ok, but that is an argument about implicitly used configuration files.
> Not something against:
> 
>     tbot -c tbot.cfg
> 
> I remember Harald saying to me that he will investigate if the current
> 'flag' system can be expanded to provide some configuration options as
> well. Maybe I misunderstood him or don't remember correctly, but if he
> does that, then we would have the option to set some values via the
> command line that will be available in the testcases.
> 
> If that is the case and gets implemented, then the command line of tbot
> might become even larger.

This is exactly the reason I refrained from implementing this feature up
until now.  I don't want to bloat the commandline unnecessarily.

> So allowing just to set the default command options
> of tbot in a configuration file that is used like I shown above, might
> be a good middle ground here.
> 
> Thinking about the flag system here, that is a bit of a strange design
> choice. It is currently some sort of configuration via the command line that is
> available in the python files, but only fits one bit of information.

I think this is a little bit of a misunderstanding.  The flag-system is *not*
meant to be used for configuration that is eg. static for each user, like
the config in our dcu repo.  It is instead meant to be used in cases where
you often need to toggle a feature.  For example to switch from booting via
nfs to booting from internal flash.  The idea was (and still is) to be able to
toggle these sorts of things easily from the commandline and not having to edit
the board-config every time.  It was never meant for "static" configuration ...

> I would say that is should eighter be removed with the hint that for
> each customization a seperate class should be written or it should allow flags
> to contain an optional arbitrary value like Harald suggested IIRC.
> Because in its current form I think that this might cause a lot of frustration
> because its so limited in its use. But if I understood him correctly and he
> thinks about expanding it, then this is a non issue and welcomed by me :)
> 
> > > Eighter use some simple config parser or just eval
> > > some python file to get your configuration in case of tbot. My hope is that if
> > > tbot provides some simple mechanism to get key-value pairs into testcases
> > > then people will not try to implement their own in each and every
> > > testcase making them incompatible to each other.
> > 
> > But maybe we can have some "tbot libraries" or "3rd party" components
> > that are not part of tbot core, as many projects do.
> 
> Hopefully all the changes to get this sort of thing working are small
> and easy enough that we don't need some external library to house them.
> 
> I would reserve additional libraries for more complicated and completly
> experimental stuff, like Windows support or what ever ;)

I don't think windows-support would ever be possible nor do I think
anyone would be willing to implement it ;)

> Eighter way, if we do that, then we might have to think about a
> plugin system for tbot. And that is a whole different can of worms to
> open...

If you do it right, you don't.  That is the beauty of python.  I'd prefer
many small modules over one big monolithic tbot because it will A) keep
the project way more manageable and B) makes it easier for a user who can
mix and match tbot with things he already knows.  If tbot provides its own
solutions for everything, we quickly land in a mess similar to django where
you have to learn everything from scratch.

> > > Why should they want complicated configuration mechanism if they already have
> > > powerful composition in this version of tbot? I am not talking about
> > > takeing that away, but just about providing some options to have some
> > > project/user configuration that is valid for tbot based test
> > > environments.
> > > 
> > > Also tbot command can be shorter even without alias or wrapper scripts, typing
> > > less it IMO always better especially with tbot were that is the ultimate
> > > goal :)
> > > 
> > > 
> > > However if I understand Heiko correctly then all of this can be easily
> > > done in a lab configuration. So we would have a "Lab" for every
> > > project/user.
> > 
> > I think this is a great idea and we can try to explore better (or for
> > me, to better understand). IMHO lab is the "setup" fo the project, and I
> > guess your use case can be done via methods of the lab object. What about a
> >         lh = cx.enter_context(tbot.acquire_lab())
> >         src = lh.get_sources()
> > 
> > > Then it should be easy to reuse other "Lab"s to recreate
> > > some sort of project/userconfiguration. I don't know how easy that is
> > 
> > Me too
> 
> I am currently just unsure about the best way to use tbot in my dev
> environments and how to change tbot in order for it to better integrated
> within.
> 
> For example if I want to use and expand the lab definitions in tbot-denx
> from my tbot-project, does the tbot-project needs to know the paths to
> the tbot-denx repository? Should tbot-denx be a submodule of
> tbot-project? What if I want to use it on a different lab?

I'd put tbot-denx into a submodule and import the pollux lab in yours like this:

	from denx.labs import pollux

	class ClaudiusLabHost(pollux.PolluxLab):
	    ...

> Mabye the inheritance be defined via the tbot command line? Like:
> 
>     tbot -l my-project.py:/path/to/my/user/tbot.py:/path/to/tbot-denx/labs/pollux.py ...
> 
> Just to be more flexible. That needs to be implemented in tbot first of
> course. But I don't know if that is a good idea or not.

Most definitely not.  This is completely against the python idoms, breaks syntax-checking,
static type-checking and is really hard to reason about ... Apart from that, implementing
it would be living hell, the python import system is already horrible enough ...

Inheritance for your lab is something you need to specify in your lab-config as shown
above, I am afraid ...

> My problem here I think is that lab-specific configuration, user specific
> configuration and project specific configuration are sort of orthogonal and
> all of them come together at the command line of tbot.

Don't get me wrong, I can also see this issue.  But as you mentioned before,
bloating the commandline isn't really a nice solution ...

> In this example:
> 
> my-project.py:
>   - contains the paths to the project sources
> 
> /path/to/my/user/tbot.py:
>   - contains customization to the base lab classes, so that login etc.
>     works (ssh-key key path, choosen build server etc.)
> 
> /path/to/tbot-denx/labs/pollux.py:
>   - contains base defintions of the available lab hardware/build server
> 
> I think thats just a lot of composition necessary. So that either has to
> be so easy, like specifying it via the command line or handled
> seperatly. That was one thing we learned with kas, at some point
> creating a file for every permutation is just to much, so we implemented
> a mechanism to combine configuration files with colons in the command
> line.
> 
> Maybe I am just missing a good example of how things should work if you
> have multiple users build multiple projects in different labs and
> trying to share as much as possible with each other.

I don't have one either at this point ... But I think we need to find a pythonic
solution for this and not force a custom one on the user as I did in my first
version.  At the moment, I have the following ideas floating around:

* As I did in the dcu repo, use a configparser.  I think this works well enough
  for this specific case.
* Add a config.py that is either imported or sourced using `eval`.  I have seen
  this pattern in a lot of other projects, where it seems to work really well.
  The advantage is, that you can now modify literally everything from your config
  because of pythons dynamic nature.  On the flipside, confining the room what a
  config can change might also be good, to keep the code simple to reason about.
* Each user creates their own lab.py which inherits the projects lab.py.  This
  would also allow all possible customizations and would have one nice side-effect:
    If the projects lab-config has sane defaults, most people could get started right
  away without needing any custom config file at all ... The downside I see is that
  you need to document elsewhere, what should be configured per user, which can be
  seen right away with the other two solutions.

What I do not like is adding some fixed solution to tbot.  I have done this in the
past and it was for the worse.  You will always stumble upon an edge-case where the
system is not expressive enough and with a fixed solution, you have no option to
change it (easily).  By going this route, you force downstream users to implement
hacks for solving their problems ...

I think you know what I am talking about (we have this exact same issue in isar all
the time!), but for others, I want to make an example:
   Just a few days ago, I was writing a testcase to bisect U-Boot.  The current
design for building U-Boot in tbot is one that is strictly configured (and does not
really use composition).  The issue arose that the `uboot.build` testcase internally
checks out the U-Boot repository.  During a git-bisect, this will make troubles because
we do not want to checkout the current master but build on the revision the bisect
put us on.  How to solve this?  Well, you can't.  You have to change the `uboot.build`
testcase.  Luckily you can do this without actually touching tbot code, but the solution
is still less than ideal:

	class BuildInfo(getattr(tbot.selectable.UBootMachine, "build")):
	    def checkout(self, clean):
	        return super().checkout(False)

	builddir = uboot.build(
	    bh,
	    BuildInfo,
	    clean=True,
	)

This code overwrites the checkout method in the U-Boot build-info (which is a composable
part of this config!) and tells tbot to always do a dirty checkout, even if we supply
clean=True to `uboot.build`.  This is arguably not good programming and definitely not
pythonic in any way.  If the U-Boot build was more composable instead of this config-approach,
this issue would never arise.

The reason I am so adamant about this is because I believe it is really *really* hard to
think about everything when using the config approach so nobody will ever have the issue
that his needs are not supported.  If you implement a compositing approach, however, you
nicely delegate this responsibility to the downstream user and in doing so solve a lot of
headaches implicitly.

I think I am getting off topic ... And this mail is getting way too long anyway.

Hope I was able to clear things up a little ... And that I haven't hurt anyones
feelings by having such a strong opinion about this ...

Regards,
	Harald

> So enough rambled on, have a nice weekend,
> Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws at denx.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://lists.denx.de/pipermail/tbot/attachments/20181203/bcb99493/attachment.sig>