[tbot] [DENX] tbot: board hangs if no autoload

Heiko Schocher hs at denx.de
Fri Nov 16 05:41:09 UTC 2018

Hello Harald, Stefano,

Am 15.11.2018 um 19:22 schrieb Stefano Babic:
> Hi Harald,
> On 15/11/18 16:59, Harald Seiler wrote:
>> Hi Stefano, Heiko,
>> you two keep posting so much that I can't even keep up
>> with it ;) I'll try to chime in on some aspects to maybe
>> clear things up.  I hope it's not too confusing ...
> :-)


>> On Thu, 2018-11-15 at 15:51 +0100, Stefano Babic wrote:
>>> Hallo Heiko,
>>> On 15/11/18 13:01, Heiko Schocher wrote:
>>>> Hello Stefano,
>>> [snip]
>>>>> Well, developers have his own way and each of us do things in a
>>>>> different and preferred way. It is just to start a flame to use just Vim
>>>>> instead of Emacs...
>>>> Of course! I can not force anyone to use tbot ...
>>>> But I hope that others are also lazy, and want to automate as much tasks
>>>> as they can.
>> TBot lives in the middle of two worlds:  On one hand it is an automation
>> tool solely to support a developer in his everyday work.  That means automating
>> repetitive tasks.
>> As an example, you might be trying to get some U-Boot feature working on a new
>> hardware.  In that case, without TBot, you'd manually compile, copy, flash, reboot,
>> try if it works, repeat.  This is what TBot tries to help you with.  With TBot,
>> you'd have testcases for all these steps and also one that does all at once.
>> So instead of having to run 10 commands that are always the same (or almost the same),
>> you just run one: tbot.  While in theory, you could also just use a shellscript,
>> TBot has the nice benefit of also managing your hardware and juggle multiple
>> hosts (labhost, buildhost, localhost, ...) for you.
>> On the other hand, TBot is also made for testing/ci use.  As a testtool for a
>> customer, I think, TBot would be used more for trying to deploy artifacts that
>> are built elsewhere and verify they work.
> Fine.


>>>>> But what developers surely need (and this is why I put functional tests
>>>>> on top of priorities) is a way to validate what they did and to have
>>>>> regression tests without a lot of effort. And in both of them, tbot
>>>>> excels.
>>>> Isn;t it for example also a valid testcase to ensure, that u-boot for
>>>> example compiles?
>>>> Just yesterday I posted a patch on U-Boot ML, which compiled, but dropped
>>>> warnings I did not check, because I build U-Boot with bitbake ... :-(
>>> As maintainer, my current work-flow is with buildman and/or travis. I
>>> get an e-mail if travis reports error, if not I sent a PR...
>>>> Or if customer uses swupdate for updating, write a testcase for it?
>>> Exactly, this is the functional test case. And you understand why I am
>>> not so interested about a specific setup for installing U-Boot and/or
>>> kernel on the target. Specific install details are already hidden by
>>> SWUpdate and I have not to take care of it. My testcase is to push the
>>> SWU to the target, and this can be done generic because the project
>>> specific parts are already handled.


>>>>> I would not say that there won't be a customer who wants to have this,
>>>>> but as far as I know, most customers rely on already known way to build
>>>>> software (Jenkins / bitbake /..) and I guess that building from U-Boot
>>>>> is not the first priority for them. But testing that the build works is
>>>>> on the top of the list.
>>>> Ok. But thats the benefit of using tbot. You (as developer) can automate
>>>> *all* your task you have ... and pass the customer only the testcase for
>>>> example which start testing functionality for his board ...
>>> Yes, I think anyone should find the most profitable way to use the tool.
>>>>>> And if I have one command for doing all the boring stuff from
>>>>>> scratch ... this is nice. Also if you get at the end a documentation
>>>>>> with all the steps for the customer, how to reproduce this.
>>>>>>> If we start to convert how to install software on the board, we start
>>>>>>> with a lot of single different cases, because this is absolutely board
>>>>>>> specific.
>>>>>> Yes ... so write for the board specific part a board specific testcase
>>>>>> which is called from a generic part ...
>>>>> I am just looking to the current status and what we have available. To
>>>>> do this, I am expecting that class Board has additional methods like
>>>>> "install_uboot" and/or "install_linux" near poweron / poweroff/.., see
>>>>> machine/board/board.py. So I guess we are not ready for it and it is
>>>>> better to start with testcases that do not imply to have a very specific
>>>>> setup for each board.
>>>> I rather have in mind, not to fullfill the class with a lot of tasks,
>>>> instead
>>>> let tbot as simpel as possible and do the hard work in testcases ...
>>> ok
>> The Board class is part of TBot's core and should stay as small as possible.
>> `install_uboot` should definitely be a testcase and because it is very board
>> specific I don't even think it belongs in TBot at all.
> ok, got it.

Yes, I think also, we must keep the core as small as possible.

>>>> But may I am here on the wrong way ...
>>>>>>> My vote goes to start with the more general cases, that is: Software is
>>>>>>> on the board, does the board work as expected ? Things like:
>>>>>>> - U-Boot:
>>>>>>>       - does network work ?
>>>>>>>       - does storage work ?
>>>>>>>       - do other u-boot peripherals work ?
>>>>>> Of course also a valid starting point!
>>>>>> But also you must define a way, how to find out, what devices are
>>>>>> oon the board... I did for example "help date" and if this is
>>>>>> successfull, I can test the date command ...
>>>>> I think this can be ok to put into the board configuration file. It is a
>>>>> static configuration and does not depend on the runtime.
>> One could technically detect these features at runtime but this might be error
>> prone as a failing detection will lead to a feature not being tested when it should.
>> I think the boards features should be explicitly stated in the config and in
>> board-specific testcases.  For example:
>> 	from tbot import tc
>> 	from tbot.tc import uboot
>> 	def p2020rdb_uboot_tests() -> None:
>> 	    with tbot.acquire_lab() as lh:
>> 	        tc.testsuite(
>> 	            # These should be testcases provided by TBot
>> 	            # you list them in this board-specific testcase
>>                      # to explicitly run them for this board
>> 	            uboot.tests.i2c,
>> 	            uboot.tests.spi,
>> 	            uboot.tests.hush,
>> 	            uboot.tests.network,
>> 	            uboot.tests.mmc,
>> 	        )
>> I'll add some info about this to our Repo's README ...
> ok, thanks !

May a practical example would help?

May we use the BeagleBoneBlack as a reference board, as yocto do, and
write examples for it?

>>>> Hmm... really .. think on the capes of the beagleboneblack ...
>>>> I would say, write a board specific testcase, which calls all the (maybe
>>>> generic) testcases you want to run on the board ... or test what testcases
>>>> it can run ...>
>>>>>> Or parse help output and decide then?
>>>>> Also a good idea, too.
>>>>>> Parse U-Boots config and/or DTS ?
>>>>>>> Such cases - they are unaware of which board is running, and we can at
>>>>>>> the early beginning have more general test cases. Same thing for linux,
>>>>>>> but of course we can have much more.
>>>>>> see above.
>>>>>> Also as you can call testcases from another testcase, you can write
>>>>>> a board specific testcase, in which you (as board maintainer) should
>>>>>> now, which generic testcases you can call ...
>>>>> That is nice ! I wait for tomorrow when testcases will be put into
>>>>> tbot-denx. It will help me to understand better.
>>>> At least with the old tbot you can do this ... and I am sure Haralds
>>>> newer version can do this!
>>>> I had/have variables which hold the name of a testcase ... so you can
>>>> write a generic testcase, which calls testcases you can configure
>>>> in the board config file ...
>>>> For example:
>>>> https://github.com/hsdenx/tbot/blob/master/src/tc/demo/u-boot/tc_demo_compile_install_test.py#L134
>>>> if tb.config.tc_demo_uboot_test_update != 'none':
>>>> call testcase with the name in this variable ... so you can write a
>>>> board specific testcase, which installs SPL/U-Boot on your specific
>>>> board ...
>>>> so you can set (old tbot!) in your board or lab config file:
>>>> tc_demo_uboot_test_update = 'tc_install_uboot_on_p2020rdb.py'
>> Yes, I know you make use of this design pattern.  I would argue against it though, as
>> this callback style unnecessarily limits you in what you can do.
>>      You are much more flexible if TBot just provides generic helpers for things that
>> are generic and implement the rest as board-specific testcases.  In this case, a
>> `p2020rdb_install_uboot` testcase which does the specific install routine for p2020rdb
>> and finally calls a testcase provided by TBot that might be called `uboot.check_version`.

Yes. I had no seperation between generic tbot testcases and board specific
ones. May we have more than one testcase "repo" usable with "-T" parameter
of tbot?

>>> ok, maybe you are getting the point because this is what I do not like.
>>> I prefer a more generic approach as I see in the example. I can test any
>>> linux command simply with:
>>>              lh = cx.enter_context(tbot.acquire_lab())
>>>              b = cx.enter_context(tbot.acquire_board(lh))
>>>              lnx = cx.enter_context(tbot.acquire_linux(b))
>>> This hides which board (mira in my case), which lab, how u-boot is
>>> started and how linux is started. There is not a "boot_linux_on_mira",
>>> it is simply tbot.acquire_linux(b) ! Very nice !
>> Yes, the idea is to be able to write testcases as generic as possible.
>> (If applicable!)
> Right


>>> The class hides the specific part (which u-boot variables I must set,
>>> which commands, ..). That means that this testcase runs as it is on any
>>> board, from power-off until it is turned off again, and full resuse is
>>> possible. Something like:
>>> @tbot.testcase
>>>   def iinstall_uboot() -> None:
>>>       with tbot.acquire_lab() as lh:
>>>           with tbot.acquire_board(lh) as b:
>>>               with tbot.acquire_uboot(b) as ub:
>>>                   ub.install_uboot()
>>> And this is generic and I just need to define a set of commands for my
>>> board to istall u-boot (like "boot_command" array, I mean).
>> `install_uboot` is something I would not implement generically as this
>> can be vastly different depending on the hardware.  In this case I'd rather
>> add board specific `p2020rdb_install_uboot`/`mira_install_uboot` testcases,
>> that just directly run the commands necessary for installation.
> ok, understood, it is fine.
>> I guess my approach can be summarized as:  Prefer composition over configuration
>> Prefer writing testcases that call smaller, generic testcases over
>> writing one monolith that can be configured to do everything.

Full Ack, that was also my approach ...

But ended in testcases which did all tasks for a specific board ...
which called all the small generic testcases, where possible.

And I had a lot fo duplication in them, so I tried to write generic
testcases, which call here and there board specific testcases,
when a board specific task was needed.

But I am an old C programmer, so I hope you can show me the correct
way with phython :-D

>> If a testcase is generic enough in it's nature, configuration is ok.
>> For example I have implemented `uboot.build` as a testcase that takes
>> configuration from the board.  I made this choice, because the U-Boot build
>> process doesn't differ much between different hardware (At least in
>> my experience).

Yes, this should be possible as a generic testcase.

>> `uboot.install`, however, doesn't make much sense as a configurable testcase
>> in my opinion.  I'd rather write board specific testcases each time, as
>> there isn't much shared functionality anyway.  The shared pieces that do
>> exist can be refactored into TBot's builtin testcases of course!

Yes ... I defined in my U-Boot Environment variable setup a tbot_upd_spl
and tbot_upd_uboot variable, which does the necessary steps and call
them from the generic approach ... but I never was happy with it.

>>> In your example, I cannot set a generic testcase like
>>> tc_demo_uboot_test_update to work for any board, because it must
>>> reference to a specific file / function (install_uboot_p2020rdb). IMHO I
>>> like to have an abstraction to hide the specific part of a board (as the
>>> enter_context above let me do).
>>>> and the generic testcase will call this board specific function for
>>>> installing SPL/U-Boot for the p2020rdb board ...
>>>> You got the idea ?
>>>> Hope I do not make Harald now headaches :-P
>>> Maybe I have added some headaches...
>> No worries!


>>>>>>>> - create a register dump file
>>>>>>>>      write register content into a register dump file
>>>>>>>> - do register checks
>>>>>>>>      open register dump file and check if register content
>>>>>>>>      is the same as in the file
>>>>>>>> - convert all DUTS testcases
>>>>>>>>      http://git.denx.de/?p=duts.git
>>>>>>> I do not thing this is a great idea. This "duts" is obsolete and I
>>>>>>> think
>>>>>>> we have now a more generic and better concept with tbot. I think we
>>>>>>> should just have a list of test cases and then translate them in
>>>>>>> @tbot.testcase, without looking at the past. IMHO duts is quite broken
>>>>>>> and we should not care of it, it can just confuse us and it could be a
>>>>>>> waste of time.
>>>>>> But there are a lot of valid tests!
>>>>> That is the reason I think we should have a list of testcases, and then
>>>>> implement them as @tbot.testcase
>>>> Yes!
>>>>>> It is just an idea ... I converted some of them (not finished all)
>>>>>> and made based on the results a U-Boot cmdline documentation, as
>>>>>> we had with the DULG.
>>>>> ok, I'll wait for ;-)
>>>> :-P
>>>> Not ready for the new tbot ... patches are welcome!
>>>>>>>>      goal, create at the end a u-boot commandline documentation
>>>>>>>> - call pytest from u-boot?
>>>>>>> Do we ?
>>>>>> I meant: call U-Boot testframework which is in "tools/py"
>>>>>> from tbot.
>>>>>>>> - if new u-boot does not boot, switch bootmode and unbreak it
>>>>>>> This is also very board specific and it does not always work. I prefer
>>>>>>> to start with a more generic approach.
>>>>>>> For example, start with testing network in U-Boot. How can I split
>>>>>>> between Lab setup and board setup ? Let's say the tftp server. I can
>>>>>>> set
>>>>>>> in the board file a "setenv serverip", but this is broken, because a
>>>>>>> board could belong to different Labs (I have a mira here and I have my
>>>>>>> own Lab setup). Is there a a way to do this ? Where should I look for
>>>>>>> such a cases ?
>>>>>> Than the serverip should be a lab specific variable.
>>>>> Should not be an attribute of UbootMachine class, that I can overwrite
>>>>> in my lab.py ?
>>>> Or better, it maybe is detectable through a testcase, executed on the
>>>> lab PC ?
>>>> The tftp serverip is configured somewhere on the lab PC ... so write a
>>>> testcase
>>> But a lab PC is not strictly required, and we have not. If you see the
>>> code, I manage to add my lab simply having my own board/lab.py instead
>>> of board/denx.py, and I inherit my board (mira) from it.
>> By default, TBot just uses your localhost as a "lab PC".
>> In the denx repo, board/denx.py is just a convenience to reduce code
>> duplication.
> I find it a nice convenience that I can reuse to redefine my internal
> "lab" commands

Yes, and I this is good!

>> If you want to add your own lab, you'd be better off creating
>> a labs/sbabic.py.
> Yes, this is what I did.

Me too :-P

>>   But I am not sure if that should reside in our
>> denx repo.
> No, it makes no sense - it is just a definition for my own lab.

No ... you should create your own repo ... May we have more than
one repo we use with tbot. Therefore "-T" parameter is... ?

I would not say we should have something like the meta layer construct
in yocto?

Again old tbot: There you just had to create a subdir in src/tc
for adding "private" testcases ... but forget this fast ;-)

>> If you want a board to work in multiple labs, you could make
>> the board-config check which lab was selected and use a different
>> base class depending on that:
>> 	if tbot.selectable.LabHost.name == "pollux":
>> 	    # Use pollux specific config
>> 	    import denx
>> 	    BoardBase = denx.DenxBoard
>> 	elif tbot.selectable.LabHost.name == "sbabic":
>> 	    # Use your personal lab config
>> 	    import sbabic
>> 	    BoardBase = sbabic.SbabicBoard
>> 	else:
>> 	    raise NotImplementedError("Board not available on this labhost!")
>> 	class Mira(BaseBoard):
>> 	    ...
> Good idea, thanks, I will do in this way.

I try it.

>>>> for it, which returns the ip ... and you do not need to configure it!
>> If `serverip` proves to be needed enough times, we could add it as an option
>> to the labhost:
>> 	class PolluxLab(lab.SSHLabHost):
>> 	    ...
>> 	    serverip = ""
> That means, if I use the default host, should I derive my own class
> (Let's say, SbabicLab) from LabHost, adding serverip or whatever I want,
> and then at the end LAB = SbabicLab ? Is this the correct way ?
>> and check it using
>> 	# Board config
>> 	tbot.selectable.LabHost.serverip
>> 	# or cleaner (maybe even with a default value?):
>> 	getattr(tbot.selectable.LabHost, "serverip", "")
> ok
>>>>>>>> Linux:
>>>>>>>> - get sources
>>>>>>>> - may apply patches to it
>>>>>>>> - install linux on the board
>>>>>>>> - check if booted version is the expected one
>>>>>>>> - create a register dump file
>>>>>>>>      write register content into a register dump file
>>>>>>>> - do register checks
>>>>>>> See above. I think this is useful during a porting, but it is less
>>>>>>> useful for a customer who wants to test functionalities. I would
>>>>>>> like to
>>>>>> I have here another opinion.
>>>>> Well, of course ;-). We should not always agree, we get more improvement
>>>>> when we discuss and have different opinions ! ;-)
>>>> Yep!
>>>> I like this discussion ... 4 years nearly nobody was interested in my
>>>> old tbot.
>>>> Ok, it was a big misuse of python ... but it worked ;-)
>>>> I could not say it to much ... many thanks to Harald!
>>>>>> This is also interesting for a customer.
>>>>>> Which customer does never change a DTS or does not try a linux update
>>>>>> on his own?
>>>>>> If he have an automated check, if all important registers are setup
>>>>>> as expected ... this is nice.
>>>>>> This testcase could be done very generic...
>>>>>>> have first a catalog of testcases with functionalities, like:
>>>>>>>       - is network working ?
>>>>>>>       - are peripherals working (SPI / I2C /....) ?
>>>>>> Yes. My hope is, that we get a lot of users, so we will get a lot of
>>>>>> testcases ;-)
>>>>> ok
>>>>>>> In the ideal case, DT is parsed to get a list of testcases...
>>>>>> Yes.
>>>>>>>>      open register dump file and check if register content
>>>>>>>>      is the same as in the file
>>>>>>>> - look if a list of string are in dmesg output
>>>>>>>> - look for example at the LTP project, what they test
>>>>>>> +1
>>>>>>> LTP contains a lot of useful testcases, but of course they are meant to
>>>>>>> run as scripts directly on the target / host. Anyway, they have
>>>>>>> testcases for a lot of things.
>>>>>> Yes, and we may can use this scripts! Start them and analyse the
>>>>>> results.
>>>>> ok, I let this for later, it is not clear to me how...
>>>> I am also just speculating. But executing a script on the board is easy...
>>> I see a lot of calls to something related to LTP (tst_res, tst_resm,..).
>>> Most testcases are simple, we could have most of them in tbot as own
>>> testcases as python code.
>>>>>>>> - check if ptest-runner is in the rootfs and call it
>>>>>>> ptest-runner means python. Do we have it on most projects ? Some yes,
>>>>>>> some not...
>>>>>> Therefore is "check if ptest-runner" exists ;-)
>>>>>>>> ...
>>>>>>>> yocto:
>>>>>>>> - get the sources
>>>>>>>> - configure
>>>>>>>> - bake
>>>>>>>> - check if files you are interested in are created
>>>>>>>> - install new images
>>>>>>>> - boot them
>>>>>>>> - check if rootfsversion is correct
>>>>>>> See above - IMHO it is better to split between functional tests on
>>>>>>> target and build, and to start with the functional tests.
>>>>>> Of course. Both parts can be done independently
>>>>> Sure !
> Stefano

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de

More information about the tbot mailing list