[tbot] [Discussion] Calling tbot from within the source directory

[Claudius Heine]

Hi Harald,

Quoting Harald Seiler (2018-12-03 10:27:08)
> Hello Claudius,
> On Sat, 2018-12-01 at 13:20 +0100, Claudius Heine wrote:
[...]
> > So allowing just to set the default command options
> > of tbot in a configuration file that is used like I shown above, might
> > be a good middle ground here.
> > 
> > Thinking about the flag system here, that is a bit of a strange design
> > choice. It is currently some sort of configuration via the command line that is
> > available in the python files, but only fits one bit of information.
> 
> I think this is a little bit of a misunderstanding.  The flag-system is *not*
> meant to be used for configuration that is eg. static for each user, like
> the config in our dcu repo.  It is instead meant to be used in cases where
> you often need to toggle a feature.  For example to switch from booting via
> nfs to booting from internal flash.  The idea was (and still is) to be able to
> toggle these sorts of things easily from the commandline and not having to edit
> the board-config every time.  It was never meant for "static" configuration ...

Ok, I just saw the tbot configuration at siemens and there we have a
flag for which device I want to use. If we had more than 2 we would need
two flags. And I just found that very curious.

Maybe that was just some short hack, but if I had to create a board configuration
and class for each instance of a board, thats just much more efford. So instead
a dev tries to use shortcuts like this.

> > > > Eighter use some simple config parser or just eval
> > > > some python file to get your configuration in case of tbot. My hope is that if
> > > > tbot provides some simple mechanism to get key-value pairs into testcases
> > > > then people will not try to implement their own in each and every
> > > > testcase making them incompatible to each other.
> > > 
> > > But maybe we can have some "tbot libraries" or "3rd party" components
> > > that are not part of tbot core, as many projects do.
> > 
> > Hopefully all the changes to get this sort of thing working are small
> > and easy enough that we don't need some external library to house them.
> > 
> > I would reserve additional libraries for more complicated and completly
> > experimental stuff, like Windows support or what ever ;)
> 
> I don't think windows-support would ever be possible nor do I think
> anyone would be willing to implement it ;)

Who knows? I think Microsoft got Ubuntu running on Windows without
virtualization... I think people are trying to get bitbake and OE
running on it. Maybe Microsoft will come along and port tbot to Windows
because they discover that they need it urgently :P

> > Eighter way, if we do that, then we might have to think about a
> > plugin system for tbot. And that is a whole different can of worms to
> > open...
> 
> If you do it right, you don't.  That is the beauty of python.  I'd prefer
> many small modules over one big monolithic tbot because it will A) keep
> the project way more manageable and B) makes it easier for a user who can
> mix and match tbot with things he already knows.  If tbot provides its own
> solutions for everything, we quickly land in a mess similar to django where
> you have to learn everything from scratch.

Of course you can hack around everything in python, that is not the
point. But if you want people to use that and not be annoyed by every
minor tbot update that they now have to fix their plugin again, then you
have to think about which API should plugins be written against and be
considered "stable".

> > > > Why should they want complicated configuration mechanism if they already have
> > > > powerful composition in this version of tbot? I am not talking about
> > > > takeing that away, but just about providing some options to have some
> > > > project/user configuration that is valid for tbot based test
> > > > environments.
> > > > 
> > > > Also tbot command can be shorter even without alias or wrapper scripts, typing
> > > > less it IMO always better especially with tbot were that is the ultimate
> > > > goal :)
> > > > 
> > > > 
> > > > However if I understand Heiko correctly then all of this can be easily
> > > > done in a lab configuration. So we would have a "Lab" for every
> > > > project/user.
> > > 
> > > I think this is a great idea and we can try to explore better (or for
> > > me, to better understand). IMHO lab is the "setup" fo the project, and I
> > > guess your use case can be done via methods of the lab object. What about a
> > >         lh = cx.enter_context(tbot.acquire_lab())
> > >         src = lh.get_sources()
> > > 
> > > > Then it should be easy to reuse other "Lab"s to recreate
> > > > some sort of project/userconfiguration. I don't know how easy that is
> > > 
> > > Me too
> > 
> > I am currently just unsure about the best way to use tbot in my dev
> > environments and how to change tbot in order for it to better integrated
> > within.
> > 
> > For example if I want to use and expand the lab definitions in tbot-denx
> > from my tbot-project, does the tbot-project needs to know the paths to
> > the tbot-denx repository? Should tbot-denx be a submodule of
> > tbot-project? What if I want to use it on a different lab?
> 
> I'd put tbot-denx into a submodule and import the pollux lab in yours like this:
> 
>         from denx.labs import pollux
> 
>         class ClaudiusLabHost(pollux.PolluxLab):
>             ...

And that exact example it something that makes working with this so
difficult. Because I would work on differnt projects and on different
labs I would need to create a Python class for every permutation. But I
guess you already understood that.

> 
> > Mabye the inheritance be defined via the tbot command line? Like:
> > 
> >     tbot -l my-project.py:/path/to/my/user/tbot.py:/path/to/tbot-denx/labs/pollux.py ...
> > 
> > Just to be more flexible. That needs to be implemented in tbot first of
> > course. But I don't know if that is a good idea or not.
> 
> Most definitely not.  This is completely against the python idoms, breaks syntax-checking,
> static type-checking and is really hard to reason about ... Apart from that, implementing
> it would be living hell, the python import system is already horrible enough ...
> 
> Inheritance for your lab is something you need to specify in your lab-config as shown
> above, I am afraid ...

That wasn't an example about how that should be implemented, but just
about what I would need. Creating python class everytime I call a
permutation of these configuration files is just takes IMO to much
efford.

> 
> > My problem here I think is that lab-specific configuration, user specific
> > configuration and project specific configuration are sort of orthogonal and
> > all of them come together at the command line of tbot.
> 
> Don't get me wrong, I can also see this issue.  But as you mentioned before,
> bloating the commandline isn't really a nice solution ...

Thats why I suggested putting your configuration into one configuration
file and call tbot with that.

> 
> > In this example:
> > 
> > my-project.py:
> >   - contains the paths to the project sources
> > 
> > /path/to/my/user/tbot.py:
> >   - contains customization to the base lab classes, so that login etc.
> >     works (ssh-key key path, choosen build server etc.)
> > 
> > /path/to/tbot-denx/labs/pollux.py:
> >   - contains base defintions of the available lab hardware/build server
> > 
> > I think thats just a lot of composition necessary. So that either has to
> > be so easy, like specifying it via the command line or handled
> > seperatly. That was one thing we learned with kas, at some point
> > creating a file for every permutation is just to much, so we implemented
> > a mechanism to combine configuration files with colons in the command
> > line.
> > 
> > Maybe I am just missing a good example of how things should work if you
> > have multiple users build multiple projects in different labs and
> > trying to share as much as possible with each other.
> 
> I don't have one either at this point ... But I think we need to find a pythonic
> solution for this and not force a custom one on the user as I did in my first
> version.  At the moment, I have the following ideas floating around:
> 
> * As I did in the dcu repo, use a configparser.  I think this works well enough
>   for this specific case.

Then this solves the problem for just one test case repo. I would have
copy paste this into every other test case repo, if I want to use that.
And I could not use test cases from others that have used a different
mechanism or just called their variables differently.

> * Add a config.py that is either imported or sourced using `eval`.  I have seen
>   this pattern in a lot of other projects, where it seems to work really well.
>   The advantage is, that you can now modify literally everything from your config
>   because of pythons dynamic nature.  On the flipside, confining the room what a
>   config can change might also be good, to keep the code simple to reason about.

The first mechanism of kas used python callback functions in order to
do everything. People went crazy with that. It was a mess. If you have
to know the intrinsics of a software in order to make it fit for your
purpose, then that will go bad. People are lazy and will rather
implement some hacks, instead of investigating and using well tested functionality
that is provided by that software already.

Maybe that is different in tbot, but IMO expecting users to write code in order
to use a software in for its common case is an invitation for chaos.

> * Each user creates their own lab.py which inherits the projects lab.py.  This
>   would also allow all possible customizations and would have one nice side-effect:
>     If the projects lab-config has sane defaults, most people could get started right
>   away without needing any custom config file at all ... The downside I see is that
>   you need to document elsewhere, what should be configured per user, which can be
>   seen right away with the other two solutions.

As I already mentioned, we need m:n relations between users, labs and
projects. Inheritance only provides 1:n in one direction.

> What I do not like is adding some fixed solution to tbot.  I have done this in the
> past and it was for the worse.  You will always stumble upon an edge-case where the
> system is not expressive enough and with a fixed solution, you have no option to
> change it (easily).  By going this route, you force downstream users to implement
> hacks for solving their problems ...

If downstream always has to implement its own code, it will start with
hacks right out of the box...

IMO the common case should be doable without much coding. For special
corner cases we can have some kind of mechanism to load custom classes,
etc.

As I said I like hybrid systems.

> I think you know what I am talking about (we have this exact same issue in isar all
> the time!), but for others, I want to make an example:

Yes bitbake has gone too far in many cases, I agree totally here. And
isar especially has still a very long road ahead. But if I compare
bitbake with for instance wic. I prefere bitbake. Why? Because in order
to implmement by custom functionality in wic I have to implement my own
plugins everytime. Very often just copying an upstream plugin, changing
a view lines and pasting it into my own layer under a random new name.

>    Just a few days ago, I was writing a testcase to bisect U-Boot.  The current
> design for building U-Boot in tbot is one that is strictly configured (and does not
> really use composition).  The issue arose that the `uboot.build` testcase internally
> checks out the U-Boot repository.  During a git-bisect, this will make troubles because
> we do not want to checkout the current master but build on the revision the bisect
> put us on.  How to solve this?  Well, you can't.  You have to change the `uboot.build`
> testcase.  Luckily you can do this without actually touching tbot code, but the solution
> is still less than ideal:
> 
>         class BuildInfo(getattr(tbot.selectable.UBootMachine, "build")):
>             def checkout(self, clean):
>                 return super().checkout(False)
> 
>         builddir = uboot.build(
>             bh,
>             BuildInfo,
>             clean=True,
>         )
> 
> This code overwrites the checkout method in the U-Boot build-info (which is a composable
> part of this config!) and tells tbot to always do a dirty checkout, even if we supply
> clean=True to `uboot.build`.  This is arguably not good programming and definitely not
> pythonic in any way.  If the U-Boot build was more composable instead of this config-approach,
> this issue would never arise.

And my suggestions whould IMO not change a thing about the ability to
implement custom classes and inherit and overwrite methods from existing
ones. For me configuration does not change anything about what is done,
which is implemented in testcases, etc. just the environment and some
parameters of it.

For me there is not a configuration *xor* composability, but as I
understand it both handle different things. You write classes and
testcases and lab/board definitions to express the process and
configuration expresses the parameters of it.

> 
> The reason I am so adamant about this is because I believe it is really *really* hard to
> think about everything when using the config approach so nobody will ever have the issue
> that his needs are not supported.  If you implement a compositing approach, however, you
> nicely delegate this responsibility to the downstream user and in doing so solve a lot of
> headaches implicitly.
> 
> I think I am getting off topic ... And this mail is getting way too long anyway.
> 
> Hope I was able to clear things up a little ... And that I haven't hurt anyones
> feelings by having such a strong opinion about this ...

No absolutly not. I like our discussion and I guess we both see just
different perspectives of the same problem. And maybe also have just
different understandings of what 'configuration' and 'composition' is.

kind regards,
Claudius

--
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch at denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: signature
URL: <http://lists.denx.de/pipermail/tbot/attachments/20181203/e88bd526/attachment-0001.sig>