[tbot] [DENX] tbot: board hangs if no autoload

Harald Seiler hws at denx.de
Thu Nov 15 15:59:02 UTC 2018


Hi Stefano, Heiko,

you two keep posting so much that I can't even keep up
with it ;) I'll try to chime in on some aspects to maybe
clear things up.  I hope it's not too confusing ...

On Thu, 2018-11-15 at 15:51 +0100, Stefano Babic wrote:
> Hallo Heiko,
> 
> On 15/11/18 13:01, Heiko Schocher wrote:
> > Hello Stefano,
> > 
> 
> [snip]
> 
> > > Well, developers have his own way and each of us do things in a
> > > different and preferred way. It is just to start a flame to use just Vim
> > > instead of Emacs...
> > 
> > Of course! I can not force anyone to use tbot ...
> > 
> > But I hope that others are also lazy, and want to automate as much tasks
> > as they can.

TBot lives in the middle of two worlds:  On one hand it is an automation
tool solely to support a developer in his everyday work.  That means automating
repetitive tasks.

As an example, you might be trying to get some U-Boot feature working on a new
hardware.  In that case, without TBot, you'd manually compile, copy, flash, reboot,
try if it works, repeat.  This is what TBot tries to help you with.  With TBot,
you'd have testcases for all these steps and also one that does all at once.

So instead of having to run 10 commands that are always the same (or almost the same),
you just run one: tbot.  While in theory, you could also just use a shellscript,
TBot has the nice benefit of also managing your hardware and juggle multiple
hosts (labhost, buildhost, localhost, ...) for you.


On the other hand, TBot is also made for testing/ci use.  As a testtool for a
customer, I think, TBot would be used more for trying to deploy artifacts that
are built elsewhere and verify they work.

> > > But what developers surely need (and this is why I put functional tests
> > > on top of priorities) is a way to validate what they did and to have
> > > regression tests without a lot of effort. And in both of them, tbot
> > > excels.
> > 
> > Isn;t it for example also a valid testcase to ensure, that u-boot for
> > example compiles?
> > 
> > Just yesterday I posted a patch on U-Boot ML, which compiled, but dropped
> > warnings I did not check, because I build U-Boot with bitbake ... :-(
> 
> As maintainer, my current work-flow is with buildman and/or travis. I
> get an e-mail if travis reports error, if not I sent a PR...
> 
> > Or if customer uses swupdate for updating, write a testcase for it?
> 
> Exactly, this is the functional test case. And you understand why I am
> not so interested about a specific setup for installing U-Boot and/or
> kernel on the target. Specific install details are already hidden by
> SWUpdate and I have not to take care of it. My testcase is to push the
> SWU to the target, and this can be done generic because the project
> specific parts are already handled.
> 
> > > I would not say that there won't be a customer who wants to have this,
> > > but as far as I know, most customers rely on already known way to build
> > > software (Jenkins / bitbake /..) and I guess that building from U-Boot
> > > is not the first priority for them. But testing that the build works is
> > > on the top of the list.
> > 
> > Ok. But thats the benefit of using tbot. You (as developer) can automate
> > *all* your task you have ... and pass the customer only the testcase for
> > example which start testing functionality for his board ...
> 
> Yes, I think anyone should find the most profitable way to use the tool.
> 
> > > > And if I have one command for doing all the boring stuff from
> > > > scratch ... this is nice. Also if you get at the end a documentation
> > > > with all the steps for the customer, how to reproduce this.
> > > > 
> > > > > If we start to convert how to install software on the board, we start
> > > > > with a lot of single different cases, because this is absolutely board
> > > > > specific.
> > > > 
> > > > Yes ... so write for the board specific part a board specific testcase
> > > > which is called from a generic part ...
> > > 
> > > I am just looking to the current status and what we have available. To
> > > do this, I am expecting that class Board has additional methods like
> > > "install_uboot" and/or "install_linux" near poweron / poweroff/.., see
> > > machine/board/board.py. So I guess we are not ready for it and it is
> > > better to start with testcases that do not imply to have a very specific
> > > setup for each board.
> > 
> > I rather have in mind, not to fullfill the class with a lot of tasks,
> > instead
> > let tbot as simpel as possible and do the hard work in testcases ...
> 
> ok

The Board class is part of TBot's core and should stay as small as possible.
`install_uboot` should definitely be a testcase and because it is very board
specific I don't even think it belongs in TBot at all.

> 
> > But may I am here on the wrong way ...
> > 
> > > > > My vote goes to start with the more general cases, that is: Software is
> > > > > on the board, does the board work as expected ? Things like:
> > > > > 
> > > > > - U-Boot:
> > > > >      - does network work ?
> > > > >      - does storage work ?
> > > > >      - do other u-boot peripherals work ?
> > > > 
> > > > Of course also a valid starting point!
> > > > 
> > > > But also you must define a way, how to find out, what devices are
> > > > oon the board... I did for example "help date" and if this is
> > > > successfull, I can test the date command ...
> > > 
> > > I think this can be ok to put into the board configuration file. It is a
> > > static configuration and does not depend on the runtime.

One could technically detect these features at runtime but this might be error
prone as a failing detection will lead to a feature not being tested when it should.

I think the boards features should be explicitly stated in the config and in
board-specific testcases.  For example:

	from tbot import tc
	from tbot.tc import uboot

	def p2020rdb_uboot_tests() -> None:
	    with tbot.acquire_lab() as lh:
	        tc.testsuite(
	            # These should be testcases provided by TBot
	            # you list them in this board-specific testcase
                    # to explicitly run them for this board

	            uboot.tests.i2c,
	            uboot.tests.spi,
	            uboot.tests.hush,
	            uboot.tests.network,
	            uboot.tests.mmc,
	        )

I'll add some info about this to our Repo's README ...

> > 
> > Hmm... really .. think on the capes of the beagleboneblack ...
> > 
> > I would say, write a board specific testcase, which calls all the (maybe
> > generic) testcases you want to run on the board ... or test what testcases
> > it can run ...>
> > > > Or parse help output and decide then?
> > > 
> > > Also a good idea, too.
> > > 
> > > > Parse U-Boots config and/or DTS ?
> > > > 
> > > > > Such cases - they are unaware of which board is running, and we can at
> > > > > the early beginning have more general test cases. Same thing for linux,
> > > > > but of course we can have much more.
> > > > 
> > > > see above.
> > > > 
> > > > Also as you can call testcases from another testcase, you can write
> > > > a board specific testcase, in which you (as board maintainer) should
> > > > now, which generic testcases you can call ...
> > > 
> > > That is nice ! I wait for tomorrow when testcases will be put into
> > > tbot-denx. It will help me to understand better.
> > 
> > At least with the old tbot you can do this ... and I am sure Haralds
> > newer version can do this!
> > 
> > I had/have variables which hold the name of a testcase ... so you can
> > write a generic testcase, which calls testcases you can configure
> > in the board config file ...
> > 
> > For example:
> > https://github.com/hsdenx/tbot/blob/master/src/tc/demo/u-boot/tc_demo_compile_install_test.py#L134
> > 
> > 
> > if tb.config.tc_demo_uboot_test_update != 'none':
> > 
> > call testcase with the name in this variable ... so you can write a
> > board specific testcase, which installs SPL/U-Boot on your specific
> > board ...
> > 
> > so you can set (old tbot!) in your board or lab config file:
> > 
> > tc_demo_uboot_test_update = 'tc_install_uboot_on_p2020rdb.py'

Yes, I know you make use of this design pattern.  I would argue against it though, as
this callback style unnecessarily limits you in what you can do.
    You are much more flexible if TBot just provides generic helpers for things that
are generic and implement the rest as board-specific testcases.  In this case, a
`p2020rdb_install_uboot` testcase which does the specific install routine for p2020rdb
and finally calls a testcase provided by TBot that might be called `uboot.check_version`.

> ok, maybe you are getting the point because this is what I do not like.
> I prefer a more generic approach as I see in the example. I can test any
> linux command simply with:
> 
>             lh = cx.enter_context(tbot.acquire_lab())
>             b = cx.enter_context(tbot.acquire_board(lh))
>             lnx = cx.enter_context(tbot.acquire_linux(b))
> 
> This hides which board (mira in my case), which lab, how u-boot is
> started and how linux is started. There is not a "boot_linux_on_mira",
> it is simply tbot.acquire_linux(b) ! Very nice !

Yes, the idea is to be able to write testcases as generic as possible.
(If applicable!)

> The class hides the specific part (which u-boot variables I must set,
> which commands, ..). That means that this testcase runs as it is on any
> board, from power-off until it is turned off again, and full resuse is
> possible. Something like:
> 
> @tbot.testcase
>  def iinstall_uboot() -> None:
>      with tbot.acquire_lab() as lh:
>          with tbot.acquire_board(lh) as b:
>              with tbot.acquire_uboot(b) as ub:
>                  ub.install_uboot()
> 
> And this is generic and I just need to define a set of commands for my
> board to istall u-boot (like "boot_command" array, I mean).

`install_uboot` is something I would not implement generically as this
can be vastly different depending on the hardware.  In this case I'd rather
add board specific `p2020rdb_install_uboot`/`mira_install_uboot` testcases,
that just directly run the commands necessary for installation.

I guess my approach can be summarized as:  Prefer composition over configuration

Prefer writing testcases that call smaller, generic testcases over
writing one monolith that can be configured to do everything.

If a testcase is generic enough in it's nature, configuration is ok.
For example I have implemented `uboot.build` as a testcase that takes
configuration from the board.  I made this choice, because the U-Boot build
process doesn't differ much between different hardware (At least in
my experience).

`uboot.install`, however, doesn't make much sense as a configurable testcase
in my opinion.  I'd rather write board specific testcases each time, as
there isn't much shared functionality anyway.  The shared pieces that do
exist can be refactored into TBot's builtin testcases of course!

> In your example, I cannot set a generic testcase like
> tc_demo_uboot_test_update to work for any board, because it must
> reference to a specific file / function (install_uboot_p2020rdb). IMHO I
> like to have an abstraction to hide the specific part of a board (as the
> enter_context above let me do).
> 
> > and the generic testcase will call this board specific function for
> > installing SPL/U-Boot for the p2020rdb board ...
> > 
> > You got the idea ?
> > 
> > Hope I do not make Harald now headaches :-P
> 
> Maybe I have added some headaches...

No worries!

> > > > > > - create a register dump file
> > > > > >     write register content into a register dump file
> > > > > > - do register checks
> > > > > >     open register dump file and check if register content
> > > > > >     is the same as in the file
> > > > > > - convert all DUTS testcases
> > > > > >     http://git.denx.de/?p=duts.git
> > > > > 
> > > > > I do not thing this is a great idea. This "duts" is obsolete and I
> > > > > think
> > > > > we have now a more generic and better concept with tbot. I think we
> > > > > should just have a list of test cases and then translate them in
> > > > > @tbot.testcase, without looking at the past. IMHO duts is quite broken
> > > > > and we should not care of it, it can just confuse us and it could be a
> > > > > waste of time.
> > > > 
> > > > But there are a lot of valid tests!
> > > 
> > > That is the reason I think we should have a list of testcases, and then
> > > implement them as @tbot.testcase
> > 
> > Yes!
> > 
> > > > It is just an idea ... I converted some of them (not finished all)
> > > > and made based on the results a U-Boot cmdline documentation, as
> > > > we had with the DULG.
> > > 
> > > ok, I'll wait for ;-)
> > 
> > :-P
> > 
> > Not ready for the new tbot ... patches are welcome!
> > 
> > > > > >     goal, create at the end a u-boot commandline documentation
> > > > > > 
> > > > > > - call pytest from u-boot?
> > > > > 
> > > > > Do we ?
> > > > 
> > > > I meant: call U-Boot testframework which is in "tools/py"
> > > > from tbot.
> > > > 
> > > > > > - if new u-boot does not boot, switch bootmode and unbreak it
> > > > > 
> > > > > This is also very board specific and it does not always work. I prefer
> > > > > to start with a more generic approach.
> > > > > 
> > > > > For example, start with testing network in U-Boot. How can I split
> > > > > between Lab setup and board setup ? Let's say the tftp server. I can
> > > > > set
> > > > > in the board file a "setenv serverip", but this is broken, because a
> > > > > board could belong to different Labs (I have a mira here and I have my
> > > > > own Lab setup). Is there a a way to do this ? Where should I look for
> > > > > such a cases ?
> > > > 
> > > > Than the serverip should be a lab specific variable.
> > > 
> > > Should not be an attribute of UbootMachine class, that I can overwrite
> > > in my lab.py ?
> > 
> > Or better, it maybe is detectable through a testcase, executed on the
> > lab PC ?
> > 
> > The tftp serverip is configured somewhere on the lab PC ... so write a
> > testcase
> 
> But a lab PC is not strictly required, and we have not. If you see the
> code, I manage to add my lab simply having my own board/lab.py instead
> of board/denx.py, and I inherit my board (mira) from it.

By default, TBot just uses your localhost as a "lab PC".

In the denx repo, board/denx.py is just a convenience to reduce code
duplication.  If you want to add your own lab, you'd be better off creating
a labs/sbabic.py.  But I am not sure if that should reside in our
denx repo.


If you want a board to work in multiple labs, you could make
the board-config check which lab was selected and use a different
base class depending on that:

	if tbot.selectable.LabHost.name == "pollux":
	    # Use pollux specific config
	    import denx
	    BoardBase = denx.DenxBoard
	elif tbot.selectable.LabHost.name == "sbabic":
	    # Use your personal lab config
	    import sbabic
	    BoardBase = sbabic.SbabicBoard
	else:
	    raise NotImplementedError("Board not available on this labhost!")

	class Mira(BaseBoard):
	    ...

> > for it, which returns the ip ... and you do not need to configure it!

If `serverip` proves to be needed enough times, we could add it as an option
to the labhost:

	class PolluxLab(lab.SSHLabHost):
	    ...
	    serverip = "192.168.1.1"

and check it using

	# Board config
	tbot.selectable.LabHost.serverip
	# or cleaner (maybe even with a default value?):
	getattr(tbot.selectable.LabHost, "serverip", "192.168.0.1")

> > > > > > Linux:
> > > > > > 
> > > > > > - get sources
> > > > > > - may apply patches to it
> > > > > > - install linux on the board
> > > > > > - check if booted version is the expected one
> > > > > > - create a register dump file
> > > > > >     write register content into a register dump file
> > > > > > - do register checks
> > > > > 
> > > > > See above. I think this is useful during a porting, but it is less
> > > > > useful for a customer who wants to test functionalities. I would
> > > > > like to
> > > > 
> > > > I have here another opinion.
> > > 
> > > Well, of course ;-). We should not always agree, we get more improvement
> > > when we discuss and have different opinions ! ;-)
> > 
> > Yep!
> > 
> > I like this discussion ... 4 years nearly nobody was interested in my
> > old tbot.
> > Ok, it was a big misuse of python ... but it worked ;-)
> > 
> > I could not say it to much ... many thanks to Harald!
> > 
> > > > This is also interesting for a customer.
> > > > 
> > > > Which customer does never change a DTS or does not try a linux update
> > > > on his own?
> > > > 
> > > > If he have an automated check, if all important registers are setup
> > > > as expected ... this is nice.
> > > > 
> > > > This testcase could be done very generic...
> > > > 
> > > > > have first a catalog of testcases with functionalities, like:
> > > > > 
> > > > >      - is network working ?
> > > > >      - are peripherals working (SPI / I2C /....) ?
> > > > 
> > > > Yes. My hope is, that we get a lot of users, so we will get a lot of
> > > > testcases ;-)
> > > 
> > > ok
> > > 
> > > > > In the ideal case, DT is parsed to get a list of testcases...
> > > > 
> > > > Yes.
> > > > 
> > > > > >     open register dump file and check if register content
> > > > > >     is the same as in the file
> > > > > > - look if a list of string are in dmesg output
> > > > > > 
> > > > > > - look for example at the LTP project, what they test
> > > > > > 
> > > > > 
> > > > > +1
> > > > > 
> > > > > LTP contains a lot of useful testcases, but of course they are meant to
> > > > > run as scripts directly on the target / host. Anyway, they have
> > > > > testcases for a lot of things.
> > > > 
> > > > Yes, and we may can use this scripts! Start them and analyse the
> > > > results.
> > > > 
> > > 
> > > ok, I let this for later, it is not clear to me how...
> > 
> > I am also just speculating. But executing a script on the board is easy...
> 
> I see a lot of calls to something related to LTP (tst_res, tst_resm,..).
> Most testcases are simple, we could have most of them in tbot as own
> testcases as python code.
> 
> > > > > > - check if ptest-runner is in the rootfs and call it
> > > > > 
> > > > > ptest-runner means python. Do we have it on most projects ? Some yes,
> > > > > some not...
> > > > 
> > > > Therefore is "check if ptest-runner" exists ;-)
> > > > 
> > > > > > ...
> > > > > > 
> > > > > > yocto:
> > > > > > - get the sources
> > > > > > - configure
> > > > > > - bake
> > > > > > - check if files you are interested in are created
> > > > > > - install new images
> > > > > > - boot them
> > > > > > - check if rootfsversion is correct
> > > > > 
> > > > > See above - IMHO it is better to split between functional tests on
> > > > > target and build, and to start with the functional tests.
> > > > 
> > > > Of course. Both parts can be done independently
> > > 
> > > Sure !

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws at denx.de 



More information about the tbot mailing list