[tbot] [Discussion] Calling tbot from within the source directory

[Stefano Babic]

Hi Harald,

thanks for your clarifications:

On 03/12/18 10:27, Harald Seiler wrote:
> Hello Claudius,
> 
> On Sat, 2018-12-01 at 13:20 +0100, Claudius Heine wrote:
>> Hi,
>>
>> Quoting Stefano Babic (2018-11-30 10:10:57)
>>> Hi Claudius,
>>>
>>> On 29/11/18 13:46, Claudius Heine wrote:
>>>> Hi Harald,
>>>>
>>>> Quoting Harald Seiler (2018-11-29 12:33:42)
> [...]
>>>>> It worked.  But it wasn't pretty.  And it was really hard to reason about.
>>>>> I was just short of adding something like `bitbake -e` which would allow
>>>>> you to trace where each value came from.
>>>>
>>>> I agree that configuration can be done very complicated, but there is
>>>> always some middle ground. Just because we don't like 100% make
>>>> everything configurable and overwriteable and what ever, 10% or 50%
>>>> configurable might still be nice to have.
>>>
>>> Just a new point: configurations like we thought (a .tbotrc, for
>>> example) can often hide an issue an make a comparison between working
>>> and not working systems difficult. As tbot-new is still young, I would
>>> agree to try to go on without introducing too many variables: each
>>> configuration file introduces different behaviors. We can also think
>>> about if we can reach what you want without configuration files.
>>
>> Ok, but that is an argument about implicitly used configuration files.
>> Not something against:
>>
>>     tbot -c tbot.cfg
>>
>> I remember Harald saying to me that he will investigate if the current
>> 'flag' system can be expanded to provide some configuration options as
>> well. Maybe I misunderstood him or don't remember correctly, but if he
>> does that, then we would have the option to set some values via the
>> command line that will be available in the testcases.
>>
>> If that is the case and gets implemented, then the command line of tbot
>> might become even larger.
> 
> This is exactly the reason I refrained from implementing this feature up
> until now.  I don't want to bloat the commandline unnecessarily.
> 

+

>> So allowing just to set the default command options
>> of tbot in a configuration file that is used like I shown above, might
>> be a good middle ground here.
>>
>> Thinking about the flag system here, that is a bit of a strange design
>> choice. It is currently some sort of configuration via the command line that is
>> available in the python files, but only fits one bit of information.
> 
> I think this is a little bit of a misunderstanding.  The flag-system is *not*
> meant to be used for configuration that is eg. static for each user, like
> the config in our dcu repo.  It is instead meant to be used in cases where
> you often need to toggle a feature.  For example to switch from booting via
> nfs to booting from internal flash.  The idea was (and still is) to be able to
> toggle these sorts of things easily from the commandline and not having to edit
> the board-config every time.  It was never meant for "static" configuration ...

Thanks - this is clearer now.

> 
>> I would say that is should eighter be removed with the hint that for
>> each customization a seperate class should be written or it should allow flags
>> to contain an optional arbitrary value like Harald suggested IIRC.
>> Because in its current form I think that this might cause a lot of frustration
>> because its so limited in its use. But if I understood him correctly and he
>> thinks about expanding it, then this is a non issue and welcomed by me :)
>>
>>>> Eighter use some simple config parser or just eval
>>>> some python file to get your configuration in case of tbot. My hope is that if
>>>> tbot provides some simple mechanism to get key-value pairs into testcases
>>>> then people will not try to implement their own in each and every
>>>> testcase making them incompatible to each other.
>>>
>>> But maybe we can have some "tbot libraries" or "3rd party" components
>>> that are not part of tbot core, as many projects do.
>>
>> Hopefully all the changes to get this sort of thing working are small
>> and easy enough that we don't need some external library to house them.
>>
>> I would reserve additional libraries for more complicated and completly
>> experimental stuff, like Windows support or what ever ;)
> 
> I don't think windows-support would ever be possible nor do I think
> anyone would be willing to implement it ;)
> 
>> Eighter way, if we do that, then we might have to think about a
>> plugin system for tbot. And that is a whole different can of worms to
>> open...
> 
> If you do it right, you don't.  That is the beauty of python.  I'd prefer
> many small modules over one big monolithic tbot because it will A) keep
> the project way more manageable and B) makes it easier for a user who can
> mix and match tbot with things he already knows.  If tbot provides its own
> solutions for everything, we quickly land in a mess similar to django where
> you have to learn everything from scratch.

Well, django is python, too...

This proofs that it does not matter which language is chosen to make a
mess...

> 
>>>> Why should they want complicated configuration mechanism if they already have
>>>> powerful composition in this version of tbot? I am not talking about
>>>> takeing that away, but just about providing some options to have some
>>>> project/user configuration that is valid for tbot based test
>>>> environments.
>>>>
>>>> Also tbot command can be shorter even without alias or wrapper scripts, typing
>>>> less it IMO always better especially with tbot were that is the ultimate
>>>> goal :)
>>>>
>>>>
>>>> However if I understand Heiko correctly then all of this can be easily
>>>> done in a lab configuration. So we would have a "Lab" for every
>>>> project/user.
>>>
>>> I think this is a great idea and we can try to explore better (or for
>>> me, to better understand). IMHO lab is the "setup" fo the project, and I
>>> guess your use case can be done via methods of the lab object. What about a
>>>         lh = cx.enter_context(tbot.acquire_lab())
>>>         src = lh.get_sources()
>>>
>>>> Then it should be easy to reuse other "Lab"s to recreate
>>>> some sort of project/userconfiguration. I don't know how easy that is
>>>
>>> Me too
>>
>> I am currently just unsure about the best way to use tbot in my dev
>> environments and how to change tbot in order for it to better integrated
>> within.
>>
>> For example if I want to use and expand the lab definitions in tbot-denx
>> from my tbot-project, does the tbot-project needs to know the paths to
>> the tbot-denx repository? Should tbot-denx be a submodule of
>> tbot-project? What if I want to use it on a different lab?
> 
> I'd put tbot-denx into a submodule and import the pollux lab in yours like this:
> 
> 	from denx.labs import pollux
> 
> 	class ClaudiusLabHost(pollux.PolluxLab):
> 	    ...
> 
>> Mabye the inheritance be defined via the tbot command line? Like:
>>
>>     tbot -l my-project.py:/path/to/my/user/tbot.py:/path/to/tbot-denx/labs/pollux.py ...
>>
>> Just to be more flexible. That needs to be implemented in tbot first of
>> course. But I don't know if that is a good idea or not.
> 
> Most definitely not.  This is completely against the python idoms, breaks syntax-checking,
> static type-checking and is really hard to reason about ... Apart from that, implementing
> it would be living hell, the python import system is already horrible enough ...
> 
> Inheritance for your lab is something you need to specify in your lab-config as shown
> above, I am afraid ...
> 
>> My problem here I think is that lab-specific configuration, user specific
>> configuration and project specific configuration are sort of orthogonal and
>> all of them come together at the command line of tbot.
> 
> Don't get me wrong, I can also see this issue.  But as you mentioned before,
> bloating the commandline isn't really a nice solution ...

I agree that this should not be done extending the command line - and
board, lab, project *are* configuration file.

But as Claudius says they are orthogonal - that means they should not
have any dependencies. I am unsure which is the best way to derive from
base class.

In fact, let's assume I have a board. I should be able to test myboard
without change anything in the board configuration file if I test in
another environment (lab). So I would like to have:

	tbot -b <myboard> -l <mylab> testcase

or

	tbot -b <myboard> -l <company lab> testcase

This allows to create a database of boards that can be reused in any
lab. This is currently not the case because the board file should be
aware of the lab, as it was discussed in previous thread. So I have
something in my board file like:

if tbot.selectable.LabHost.name == "pollux":
    # Use pollux specific config
    import denx
    BaseBoard = denx.DenxBoard
    BaseUBootBuild = denx.DenxUBootBuildInfo
elif tbot.selectable.LabHost.name == "local" or
tbot.selectable.LabHost.name == "papero":
    import augsburg
    # Use your personal lab config
    BaseBoard = augsburg.MylabBoard
    BaseUBootBuild = augsburg.MylabUBootBuildInfo
else:
    raise NotImplementedError("Board not available on this labhost!")

and this does not scale well because everyone has its own lab that must
be imported. I do not know if this is the only case, but the "poweron"
and "poweroff" methods are bound to the board - should they not be bound
to the lab ?

I could write in the board something like:


class MylabBoard(board.Board):

    def poweron(self) -> None:
	with lab or tbot.acquire_lab() as lh:
        	lh.poweron(self.name)

but it is not portable if a lab does not export the method. Should it do
it ?

> 
>> In this example:
>>
>> my-project.py:
>>   - contains the paths to the project sources
>>
>> /path/to/my/user/tbot.py:
>>   - contains customization to the base lab classes, so that login etc.
>>     works (ssh-key key path, choosen build server etc.)
>>
>> /path/to/tbot-denx/labs/pollux.py:
>>   - contains base defintions of the available lab hardware/build server
>>
>> I think thats just a lot of composition necessary. So that either has to
>> be so easy, like specifying it via the command line or handled
>> seperatly. That was one thing we learned with kas, at some point
>> creating a file for every permutation is just to much, so we implemented
>> a mechanism to combine configuration files with colons in the command
>> line.
>>
>> Maybe I am just missing a good example of how things should work if you
>> have multiple users build multiple projects in different labs and
>> trying to share as much as possible with each other.
> 
> I don't have one either at this point ... But I think we need to find a pythonic
> solution for this and not force a custom one on the user as I did in my first
> version.  At the moment, I have the following ideas floating around:
> 
> * As I did in the dcu repo, use a configparser.  I think this works well enough
>   for this specific case.
> * Add a config.py that is either imported or sourced using `eval`.  I have seen
>   this pattern in a lot of other projects, where it seems to work really well.
>   The advantage is, that you can now modify literally everything from your config
>   because of pythons dynamic nature.  On the flipside, confining the room what a
>   config can change might also be good, to keep the code simple to reason about.
> * Each user creates their own lab.py which inherits the projects lab.py.  This
>   would also allow all possible customizations and would have one nice side-effect:
>     If the projects lab-config has sane defaults, most people could get started right
>   away without needing any custom config file at all ... The downside I see is that
>   you need to document elsewhere, what should be configured per user, which can be
>   seen right away with the other two solutions.
> 
> What I do not like is adding some fixed solution to tbot.  I have done this in the
> past and it was for the worse.  You will always stumble upon an edge-case where the
> system is not expressive enough and with a fixed solution, you have no option to
> change it (easily).  By going this route, you force downstream users to implement
> hacks for solving their problems ...
> 
> I think you know what I am talking about (we have this exact same issue in isar all
> the time!), but for others, I want to make an example:
>    Just a few days ago, I was writing a testcase to bisect U-Boot.  The current
> design for building U-Boot in tbot is one that is strictly configured (and does not
> really use composition).  The issue arose that the `uboot.build` testcase internally
> checks out the U-Boot repository.  During a git-bisect, this will make troubles because
> we do not want to checkout the current master but build on the revision the bisect
> put us on.  How to solve this?  Well, you can't.  You have to change the `uboot.build`
> testcase.  Luckily you can do this without actually touching tbot code, but the solution
> is still less than ideal:
> 
> 	class BuildInfo(getattr(tbot.selectable.UBootMachine, "build")):
> 	    def checkout(self, clean):
> 	        return super().checkout(False)
> 
> 	builddir = uboot.build(
> 	    bh,
> 	    BuildInfo,
> 	    clean=True,
> 	)
> 
> This code overwrites the checkout method in the U-Boot build-info (which is a composable
> part of this config!) and tells tbot to always do a dirty checkout, even if we supply
> clean=True to `uboot.build`.  This is arguably not good programming and definitely not
> pythonic in any way.  If the U-Boot build was more composable instead of this config-approach,
> this issue would never arise.
> 
> The reason I am so adamant about this is because I believe it is really *really* hard to
> think about everything when using the config approach so nobody will ever have the issue
> that his needs are not supported.  If you implement a compositing approach, however, you
> nicely delegate this responsibility to the downstream user and in doing so solve a lot of
> headaches implicitly.
> 
> I think I am getting off topic ... And this mail is getting way too long anyway.
> 
> Hope I was able to clear things up a little ... And that I haven't hurt anyones
> feelings by having such a strong opinion about this ...
> 

Regards,
Stefano


-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================