[tbot] [Discussion] Calling tbot from within the source directory
wd at denx.de
Tue Dec 4 07:29:42 UTC 2018
In message <23e351927d17bd495f54e151e5b2bcd1ddff415c.camel at denx.de> you wrote:
> - Power is both lab AND board dependent. You can't put it just into one
> of the two, because it is different for each combination.
Is it? I think it should not be. Power switching is (or should be) a
property of the lab. The lab should provide a unified way to switch
power for all boards in that lab. When you use "remote_power" on
pollux or xpert you are probably not even aware that this supports 5
totally different types of hardware for switching, each with it's
own control tool and interface.
If there are board specific differences in a lab, they should/could
be hidden by a wraper script.
We have the same topic for console connection - it should not matter
whether this is a direct serial port or a USB serial adapter or
through a (real or emulated) terminal server.
We hide these details to tbot, don't we?
> - Toolchains are definitely lab-specific, but a board requires the
> right toolchain to be available.
Yes, but dependencies of bigher level properties are nothing specal,
or are they?
> I'd say this is possible right now, as long as the board doesn't require
> any special fancy features from the labhost. But as soon as it does,
> they need to know of each other, ie where the kernel is found or the
> serverip ... We can't abstract that away ...
Hm.... I would expect that in most cases such things map more or
less naturally to either board or lab specific properties. For
example, the serverip is something I would instinctively assume to
be a lab property.
> Wolfgang seems to like the idea of a database, too:
> > What I would like to see is that lab configurations, board
> > configurations and test cases can be made publicly available.
...only in the widest sense of "database". A collection of files /
directories that somehow "plays with each other" would be fine for
me, too (and definitely prefereed over a real DB).
> I agree about testcases, because they are supposed to be shareable. Sharing
> testcases also works right now, because I specifically wrote tbot in a way
> to allow for generic testcases. There might be some board-specific testcases, but
> those don't make sense sharing anyway.
> For sharing testcase, you'd just pack them into a python module and ship
> that to anyone who wants to use them/make it available online.
Yes, but this works only for isolated developers. What about
company use like we do internally at DENX, or some of our customers
When I want to run a test in our VLAB, for example on pollux, I do
not want to set up tbot from scratch. I know others are already
using tbot there - for example, Heiko is running regulra regression
tests on - say - the BBB board. Now I want to run some of these
tests myself. In this case I do not want to copy all of Heiko's
stuff. I just want to be able to use it.
Or if I want to add new test cases for the BBB board, I would like
to be able to this is some kind of private meta layer on top of
Heiko's existing configuration.
> With lab-configs I'd argue, sharing them does not make that much sense. A
> lab config doesn't contain much "valuable" information, for the most part
> it is "where do I store and find things on this specific machine?", which
> will never be portable to any other machine.
I tend to disagree. The lab configs for pollux, sisyphus, hercules
(and xpert) are pretty much the same, aren't they?
> Now, to not confuse anyone, sharing here is in the sense of sharing one
> config between multiple projects/setups. Multiple developers working in
> our VLAB can definitely use the same lab-config.
Actually I think they _should_ by default use the same lab config.
We should consider tbot not to be a private tool for the individual
developer (it should ork as such, in a strictly isolated and private
setup, too - of course), but as a multi-site, multi-lab, multi-user tool.
> If a downstream project is developed in our VLAB, it will probably require
> changes to the lab-config. To allow for this, I'd suggest the downstream
> lab inherits the pollux config and overwrites things as needed:
> [tbot_denx.pollux] <- [my_project.lab]
Yes! I definitely want to be able to inherit such common
configurations (I don't care about the implementation here).
> In code:
> from tbot_denx.labs import pollux
> class MyProjectLab(pollux.PolluxLab):
> # A project-specific config, our projects testcases might depend
> # on this, but this would mean they can't be shared with other
> # projects anymore ...
> def tftp_folder(self, board: str) -> "linux.Path[MyProjectLab]":
> return linux.Path(self, "/tftpboot") / board / "hws"
Ummm... This looks wrong to me. The TFTP path is not lab specific,
but a board property, isn't it?
> Now, this will "break down" in the following scenario: You have done your work
> and want to give a customer your tbot config so he can use it as well. The
> customer doesn't have access to pollux, they have their own lab. Now I'd say
> this means the customer needs to write their own lab-config, not dependent on
> pollux but their own stuff. To make tbot notice if they forget something, we
> can make use of pythons multiple-inheritance:
I would expect that the customer has will be using his own lab
config, and that he may have to adapt the board config to match his
environment. Ideally, _only_ the board config would need changes.
> Although ... This is quite a complex case. We should wait and see how often
> these situations actually arise before deciding on a best practice.
I think we are in the middle of it already. I know there are many
tbot users on pollux, for example. I want to use this existing
setup without setting up tbot myself :-)
> Hmm, good point ... Well, I'd argue "poweron" is sort of difficult, because
> it depends on both the board and the lab. Making it a lab-only thing is bad,
> because you now need to edit your lab-config when adding a new board. Making
I disagree. This _is_ a lab property. See comments above.
> My issue with just making it a lab-function is the following: Not everyone has
> a setup as tidy as ours, where every board is powered on and off in the same way.
In this case they should write a wrapper script that hides these
internal differences to the user. I would want to have this anyway.
I would consider it a major PITA if I had to rember which specific
switch type or console connection type a board has.
> By making poweron a lab thing, we would make this sort of setup mandatory.
I see no problem with that.
If an issue is complicated to solve on our side, but as simple as a
ten lines shell script on the other side, then we should not try to
> One more thing: I think we are seeing a few different use-cases for tbot here.
> On the one hand the way I use tbot right now: To support me while hacking on some
> customer issue. In this "mode", tbot just supports me by automating eg compile+flash+run_test.
> The config and testcases I create during working like this are all "throw-away" code, I don't
> intend it to be pretty or used later on for anything else. A lot of it is copy pasted between
> different projects for the quickest way to solving whatever the issue is. I really hack
> into tbot to get solutions quick and dirty. I try, however, to find acceptable solutions
> that don't require internal knowledge of tbot and if I can't, I will come up with upstream
> solutions ...
> There is also the "nightly-tests/ci" mode, which would be writing tests that are intended to be
> run from ci. Code written here should be robust and adhere to coding styles but doesn't need to
> be as generic as the following:
> Finally there is what Claudius and Stefano seem to intend, a use-case where tbot is more
> than just a helper and code written is intended to be "distributable" and extendable. Personally,
> I don't quite see this being applicable yet, but I see that it might come in the future and we should
> think about it when making design decisions. Wolfgangs public board database also plays in this
Future is now.
We are already in the situation that we want to use test cases for
our daily work during development, then again as regression test for
making a delivery, and then pass it on to the customer who want's to
run the same thing as acceptance tests. I'm talking about existing
projects with not-too-far-away deadlines here...
> I think the first thing we should get "public" is a bunch of testcases. Right now the only real
> builtin testcase in tbot is building U-Boot (and git-bisect if you want to count that). There is
> definitely room for improvements here, like testing a running U-Boot and the whole GNU+Linux stack ...
I would also like to see "public" lab configurations at least on the
major build hosts in DENX, minimum on pollux and xpert.
> Anyway, I got carried away again ... have a nice evening!
It's nice to see how this project starts to live on its own :-)
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Never underestimate the power of human stupidity when it comes to
using technology they don't understand.
More information about the tbot