[tbot] [Discussion] Calling tbot from within the source directory

[Wolfgang Denk]

Dear Harald,

In message <23e351927d17bd495f54e151e5b2bcd1ddff415c.camel at denx.de> you wrote:
>
> Examples:
> - Power is both lab AND board dependent.  You can't put it just into one
>   of the two, because it is different for each combination.

Is it? I think it should not be.  Power switching is (or should be) a
property of the lab.  The lab should provide a unified way to switch
power for all boards in that lab.  When you use "remote_power" on
pollux or xpert you are probably not even aware that this supports 5
totally different types of hardware for switching, each with it's
own control tool and interface.

If there are board specific differences in a lab, they should/could
be hidden by a wraper script.

We have the same topic for console connection - it should not matter
whether this is a direct serial port or a USB serial adapter or
through a (real or emulated) terminal server.

We hide these details to tbot, don't we?

> - Toolchains are definitely lab-specific, but a board requires the
>   right toolchain to be available.

Yes, but dependencies of bigher level properties are nothing specal,
or are they?

> I'd say this is possible right now, as long as the board doesn't require
> any special fancy features from the labhost.  But as soon as it does,
> they need to know of each other, ie where the kernel is found or the
> serverip ... We can't abstract that away ...

Hm.... I would expect that in most cases such things map more or
less naturally to either board or lab specific properties.  For
example, the serverip is something I would instinctively assume to
be a lab property.

> Wolfgang seems to like the idea of a database, too:
> > What I would like to see is that lab configurations, board
> > configurations and test cases can be made publicly available.

...only in the widest sense of "database". A collection of files /
directories that somehow "plays with each other" would be fine for
me, too (and definitely prefereed over a real DB).

> I agree about testcases, because they are supposed to be shareable.  Sharing
> testcases also works right now, because I specifically wrote tbot in a way
> to allow for generic testcases.  There might be some board-specific testcases, but
> those don't make sense sharing anyway.
>   For sharing testcase, you'd just pack them into a python module and ship
> that to anyone who wants to use them/make it available online.

Yes, but this works only for isolated developers.  What about
company use like we do internally at DENX, or some of our customers
do?

When I want to run a test in our VLAB, for example on pollux, I do
not want to set up tbot from scratch.  I know others are already
using tbot there - for example, Heiko is running regulra regression
tests on - say - the BBB board.  Now I want to run some of these
tests myself.  In this case I do not want to copy all of Heiko's
stuff.  I just want to be able to use it.

Or if I want to add new test cases for the BBB board, I would like
to be able to this is some kind of private meta layer on top of
Heiko's existing configuration.

> With lab-configs I'd argue, sharing them does not make that much sense.  A
> lab config doesn't contain much "valuable" information, for the most part
> it is "where do I store and find things on this specific machine?", which
> will never be portable to any other machine.

I tend to disagree.  The lab configs for pollux, sisyphus, hercules
(and xpert) are pretty much the same, aren't they?

>   Now, to not confuse anyone, sharing here is in the sense of sharing one
> config between multiple projects/setups.  Multiple developers working in
> our VLAB can definitely use the same lab-config.

Actually I think they _should_ by default use the same lab config.

We should consider tbot not to be a private tool for the individual
developer (it should ork as such, in a strictly isolated and private
setup, too - of course), but as a multi-site, multi-lab, multi-user tool.

> If a downstream project is developed in our VLAB, it will probably require
> changes to the lab-config.  To allow for this, I'd suggest the downstream
> lab inherits the pollux config and overwrites things as needed:
> 
> 	[tbot_denx.pollux] <- [my_project.lab]

Yes!  I definitely want to be able to inherit such common
configurations (I don't care about the implementation here).

> In code:
> 
> 	from tbot_denx.labs import pollux
> 
> 
> 	class MyProjectLab(pollux.PolluxLab):
> 	    # A project-specific config, our projects testcases might depend
> 	    # on this, but this would mean they can't be shared with other
> 	    # projects anymore ...
> 	    def tftp_folder(self, board: str) -> "linux.Path[MyProjectLab]":
> 	        return linux.Path(self, "/tftpboot") / board / "hws"

Ummm... This looks wrong to me.  The TFTP path is not lab specific,
but a board property, isn't it?

> Now, this will "break down" in the following scenario:  You have done your work
> and want to give a customer your tbot config so he can use it as well.  The
> customer doesn't have access to pollux, they have their own lab.  Now I'd say
> this means the customer needs to write their own lab-config, not dependent on
> pollux but their own stuff.  To make tbot notice if they forget something, we
> can make use of pythons multiple-inheritance:

I would expect that the customer has will be using his own lab
config, and that he may have to adapt the board config to match his
environment.  Ideally, _only_ the board config would need changes.

> Although ... This is quite a complex case.  We should wait and see how often
> these situations actually arise before deciding on a best practice.

I think we are in the middle of it already.  I know there are many
tbot users on pollux, for example.  I want to use this existing
setup without setting up tbot myself :-)

> Hmm, good point ... Well, I'd argue "poweron" is sort of difficult, because
> it depends on both the board and the lab.  Making it a lab-only thing is bad,
> because you now need to edit your lab-config when adding a new board.  Making

I disagree.  This _is_ a lab property.  See comments above.

> My issue with just making it a lab-function is the following:  Not everyone has
> a setup as tidy as ours, where every board is powered on and off in the same way.

In this case they should write a wrapper script that hides these
internal differences to the user.  I would want to have this anyway.
I would consider it a major PITA if I had to rember which specific
switch type or console connection type a board has.

> By making poweron a lab thing, we would make this sort of setup mandatory.

I see no problem with that.

If an issue is complicated to solve on our side, but as simple as a
ten lines shell script on the other side, then we should not try to
solve it.

> One more thing:  I think we are seeing a few different use-cases for tbot here.
> 
> On the one hand the way I use tbot right now:  To support me while hacking on some
> customer issue.  In this "mode", tbot just supports me by automating eg compile+flash+run_test.
> The config and testcases I create during working like this are all "throw-away" code, I don't
> intend it to be pretty or used later on for anything else.  A lot of it is copy pasted between
> different projects for the quickest way to solving whatever the issue is.  I really hack
> into tbot to get solutions quick and dirty.  I try, however, to find acceptable solutions
> that don't require internal knowledge of tbot and if I can't, I will come up with upstream
> solutions ...

Agreed.

> There is also the "nightly-tests/ci" mode, which would be writing tests that are intended to be
> run from ci.  Code written here should be robust and adhere to coding styles but doesn't need to
> be as generic as the following:

Agreed.

> Finally there is what Claudius and Stefano seem to intend, a use-case where tbot is more
> than just a helper and code written is intended to be "distributable" and extendable.  Personally,
> I don't quite see this being applicable yet, but I see that it might come in the future and we should
> think about it when making design decisions.  Wolfgangs public board database also plays in this
> direction.

Future is now.

We are already in the situation that we want to use test cases for
our daily work during development, then again as regression test for
making a delivery, and then pass it on to the customer who want's to
run the same thing as acceptance tests.  I'm talking about existing
projects with not-too-far-away deadlines here...

> I think the first thing we should get "public" is a bunch of testcases.  Right now the only real
> builtin testcase in tbot is building U-Boot (and git-bisect if you want to count that).  There is
> definitely room for improvements here, like testing a running U-Boot and the whole GNU+Linux stack ...

I would also like to see "public" lab configurations at least on the
major build hosts in DENX, minimum on pollux and xpert.

> Anyway, I got carried away again ... have a nice evening!

It's nice to see how this project starts to live on its own :-)

Thanks!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Never underestimate the power of human stupidity  when  it  comes  to
using technology they don't understand.