tbot testcases

[Heiko Schocher]

Hello Michal, Harald,

Am 16.06.2020 um 17:36 schrieb Harald Seiler:
> Hi Michal,
> 
> I've put the list on CC because this might be interesting to others as
> well.
> 
> On Tue, 2020-06-16 at 16:28 +0200, Michal Simek wrote:
>> Hi Harald,
>>
> [....]
>>
>> I expect you and Heiko and maybe Simon are users who are using it the
>> most. It means I expect you also have your private tests what you run.
>> Is there any reason not to share them?
> 
> I think the biggest reason for me is that I didn't yet take the time to
> organize my mess and bring it into a form that can be published.  That's

Yes, same for me...

> for the most part because my personal collection of 'testcases' is
> actually mostly ops stuff like building, flashing, or running different
> software.
> 
> Sadly they are also very project specific for the most part so not really
> useful for others in that form.  Though, where ever possible I am trying

Yep.

> to extract a generic subset and push that upstream (for example
> uboot_testpy [1]).  If the stuff I do have does sound interesting to you,
> I can take a look what else I can share.
> 
> For actual tests, on my side the situation is a bit sparse right now.
> I think Heiko (cc) has more in that regard.  Maybe he's willing to share
> some stuff / I think he has published it somewhere already?

Yes, but this is really hacky state ...

https://github.com/EmbLux-Kft/tbot-tbot2go

"interesting" stuff may in
https://github.com/EmbLux-Kft/tbot-tbot2go/tree/devel/tc/commonhelper

But there are also parts which are superseeded in the meantime...

For U-Boot I start mainly test.py (which is nowadays integrated in tbot)

For linux tests may interesting:

https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/tc/commonhelper/generic.py#L356

where I check the values in a file which contain a registerdump [1]. The file
of course created with a tbot testcase [2]

So once you have this reference file, you can check on registerlevel if a
new linux version sets the same values...

[1] https://github.com/EmbLux-Kft/tbot-tbot2go/tree/devel/tc/aristainetos/files
[2] https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/tc/commonhelper/generic.py#L404

Later same tests for U-Boot

U-Boot (I2C check a dump from device):

https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/tc/commonhelper/generic.py#L653

Unfortunately I had not the time to convert all tests from my old
tbot implementation

>> Anyway the setup you have mentioned on tbot repo with tbot host PC, then
>> lab host PC connected to board. That's one the most used setup.
>> But in tbot case at least that configuration I tried based on manual
>> that lab host is the guy who is building images which is problematic in
>> my case.
>> The reason is simply that lab host PC are arm based boards or pretty
>> weak x86 PCs and there is no value to offload this kind of work to them.
>> They really just manage connected boards.
>>
>> I expect that there should be an option to tell to tbot where images
>> should be built.
> 
> Yes, having a separate build-host is a common use-case.  That was also
> documented at some point but unfortunately it looks like that part got
> lost ... I'll have to add that again.
> 
> In essence, your lab-config also needs to define a build-host.  This
> build-host needs to be reachable from the lab-host and needs to be
> a `Builder` [2].  As an example:
> 
>      from tbot.machine import linux, connector
> 
>      # This is the build-host
>      #
>      # It needs to define what toolchains are available
>      class MyBuildHost(connector.SSHConnector, linux.Bash, linux.Builder):
>          hostname = "192.0.2.1"
>          username = "build-user"
>          # Authentication via private key
> 
>          @property
>          def toolchains(self):
>              return {
>                  "armv7a": linux.build.DistroToolchain("arm", "arm-linux-gnueabi-"),
>                  "armv8": linux.build.DistroToolchain("aarch64", "aarch64-linux-gnu-"),
>                  "mipsel": linux.build.DistroToolchain("mips", "mipsel-linux-gnu-"),
>              }
> 
>      # This is the lab-host
>      #
>      # The important part is the build() method which tells tbot which
>      # host to use for building.
>      class MyLabHost(connector.SubprocessConnector, linux.Bash, linux.Lab):
>          def build(self):
>              return MyBuildHost(self)
> 
>      LAB = MyLabHost
> 
> [1]: https://tbot.tools/modules/tc.html#u-boot-test-py
> [2]: https://tbot.tools/modules/machine_linux.html#builder
> 

I collect the builders in a seperate file, as different labs can use the
same build machines and define a tbot flag for each build machine, so I
can select when starting tbot which build machine tbot uses.

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de