[tbot] [DENX] tbot: board hangs if no autoload

[Stefano Babic]

Hi Heiko,

On 15/11/18 11:44, Heiko Schocher wrote:
> Hello Stefano,
> 
> Am 15.11.2018 um 11:23 schrieb Stefano Babic:
>> Hi Heiko, Harald,
>>
>> On 15/11/18 10:49, Heiko Schocher wrote:
>>> Hello Harald,
>>>
>>> Am 15.11.2018 um 10:28 schrieb Harald Seiler:
>>>> On Thu, 2018-11-15 at 10:23 +0100, Lukasz Majewski wrote:
>>>>> On Thu, 15 Nov 2018 10:19:17 +0100
>>>>> Harald Seiler <hws at denx.de> wrote:
>>>>>
>>>>>> On Thu, 2018-11-15 at 10:10 +0100, Stefano Babic wrote:
>>>>>>> Hi Harald,
>>>>>>>
>>>>>>> On 15/11/18 09:36, Harald Seiler wrote:
>>>>>>>> Hi Stefano!
>>>>>>>>
>>>>>>>> Yes, TBot waits for an autoboot prompt.  You can disable this by
>>>>>>>> setting `autoboot_prompt` in your UBootMachine to the U-Boot
>>>>>>>> prompt.
>>>>>>>>
>>>>>>>>      class MyUBoot(board.UBootMachine):
>>>>>>>>          prompt = "=> "
>>>>>>>>          autoboot_prompt = "=> "
>>>>>>>>
>>>>>>>> I know this is more of a hack
>>>>>>>
>>>>>>> Yes, because it is not a fix property of the board. It depends if
>>>>>>> autoload is active and bootcmd is set on the board. The "mira"
>>>>>>> board has not "bootcmd" set in default environment, and behavior is
>>>>>>> different just after setting bootcmd.
>>>>>>
>>>>>> Hmm, good point, I will think about it ...
>>>>>>
>>>>>>>   
>>>>>>>> and I will add a proper way to do this in
>>>>>>>> a future release
>>>>>>>
>>>>>>> Nice !
>>>>>>>   
>>>>>>>> (keep an eye on the CHANGELOG, there will be a lot of
>>>>>>>> small convenience features like this in the next versions!)
>>>>>>>
>>>>>>> Do we have a repo for testcases with the new format ? I really
>>>>>>> appreciate that new tbot has a clean split between software (tbot),
>>>>>>> setup (boards and lab) and testcases. We have repos for the first
>>>>>>> two cases, I cannot find a set of common tc. I mean, tc that are
>>>>>>> already converted for @tbot.testcase - if not, I would start to
>>>>>>> write myself, but I do not want to reinvent the wheel (and of
>>>>>>> course, I will do more mistakes..)
>>>>>>
>>>>>> That is what my `tbot-denx` repo is supposed to be (DENX-Internal
>>>>>> only):
>>>>>>
>>>>>>      https://gitlab.denx.de/HaraldSeiler/tbot-denx
>>>>>>
>>>>>> However, it doesn't contain any testcases yet.
>>>>>
>>>>> Is there a way to convert (or directly re-use) Heiko's test cases?
>>>>
>>>> Unfortunately not, there is absolutely no compatibility between the two
>>>> versions ... So it needs a human to do it.
>>>>
>>>> I guess, at the moment I am in the best position to do so, I just need
>>>> input about which testcases have the highest priority to you.
>>>
>>> Instead it may makes more sense to discuss what we want to test and
>>> than how to write the testcases?
>>
>> Right, let's see.
>>
>>>
>>> Proposal what I have already:
>>>
>>> U-Boot
>>>
>>> - get sources
>>> - may apply patches to it
>>>    list of patches in a directory ?
>>>    get a list of patches from patchwork todo list?
>>>
>>> - install u-boot on the board
>>> - check if really the new version of u-boot boots
>>
>> I admit that I am not very interested how to build and to push software
>> on the board. There are a lot of different paths, and this is not what
>> the customers ask. They have their own buildserver (Yocto / buildroot /
>> Jenkins / custom / whatever...) and the process to get and applied
>> patches is more interesting for a U-Boot maintainer as for a customer.
> 
> Yes ... why should developers not use tbot??

Well, developers have his own way and each of us do things in a
different and preferred way. It is just to start a flame to use just Vim
instead of Emacs...

But what developers surely need (and this is why I put functional tests
on top of priorities) is a way to validate what they did and to have
regression tests without a lot of effort. And in both of them, tbot excels.

I would not say that there won't be a customer who wants to have this,
but as far as I know, most customers rely on already known way to build
software (Jenkins / bitbake /..) and I guess that building from U-Boot
is not the first priority for them. But testing that the build works is
on the top of the list.

> And if I have one command for doing all the boring stuff from
> scratch ... this is nice. Also if you get at the end a documentation
> with all the steps for the customer, how to reproduce this.
> 
>> If we start to convert how to install software on the board, we start
>> with a lot of single different cases, because this is absolutely board
>> specific.
> 
> Yes ... so write for the board specific part a board specific testcase
> which is called from a generic part ...

I am just looking to the current status and what we have available. To
do this, I am expecting that class Board has additional methods like
"install_uboot" and/or "install_linux" near poweron / poweroff/.., see
machine/board/board.py. So I guess we are not ready for it and it is
better to start with testcases that do not imply to have a very specific
setup for each board.

> 
>> My vote goes to start with the more general cases, that is: Software is
>> on the board, does the board work as expected ? Things like:
>>
>> - U-Boot:
>>     - does network work ?
>>     - does storage work ?
>>     - do other u-boot peripherals work ?
> 
> Of course also a valid starting point!
> 
> But also you must define a way, how to find out, what devices are
> oon the board... I did for example "help date" and if this is
> successfull, I can test the date command ...

I think this can be ok to put into the board configuration file. It is a
static configuration and does not depend on the runtime.

> 
> Or parse help output and decide then?

Also a good idea, too.

> Parse U-Boots config and/or DTS ?
> 
>> Such cases - they are unaware of which board is running, and we can at
>> the early beginning have more general test cases. Same thing for linux,
>> but of course we can have much more.
> 
> see above.
> 
> Also as you can call testcases from another testcase, you can write
> a board specific testcase, in which you (as board maintainer) should
> now, which generic testcases you can call ...

That is nice ! I wait for tomorrow when testcases will be put into
tbot-denx. It will help me to understand better.

> 
>>> - create a register dump file
>>>    write register content into a register dump file
>>> - do register checks
>>>    open register dump file and check if register content
>>>    is the same as in the file
>>> - convert all DUTS testcases
>>>    http://git.denx.de/?p=duts.git
>>
>> I do not thing this is a great idea. This "duts" is obsolete and I think
>> we have now a more generic and better concept with tbot. I think we
>> should just have a list of test cases and then translate them in
>> @tbot.testcase, without looking at the past. IMHO duts is quite broken
>> and we should not care of it, it can just confuse us and it could be a
>> waste of time.
> 
> But there are a lot of valid tests!

That is the reason I think we should have a list of testcases, and then
implement them as @tbot.testcase

> 
> It is just an idea ... I converted some of them (not finished all)
> and made based on the results a U-Boot cmdline documentation, as
> we had with the DULG.

ok, I'll wait for ;-)

> 
>>>    goal, create at the end a u-boot commandline documentation
>>>
>>> - call pytest from u-boot?
>>
>> Do we ?
> 
> I meant: call U-Boot testframework which is in "tools/py"
> from tbot.
> 
>>> - if new u-boot does not boot, switch bootmode and unbreak it
>>
>> This is also very board specific and it does not always work. I prefer
>> to start with a more generic approach.
>>
>> For example, start with testing network in U-Boot. How can I split
>> between Lab setup and board setup ? Let's say the tftp server. I can set
>> in the board file a "setenv serverip", but this is broken, because a
>> board could belong to different Labs (I have a mira here and I have my
>> own Lab setup). Is there a a way to do this ? Where should I look for
>> such a cases ?
> 
> Than the serverip should be a lab specific variable.

Should not be an attribute of UbootMachine class, that I can overwrite
in my lab.py ?

> 
>>>
>>> Linux:
>>>
>>> - get sources
>>> - may apply patches to it
>>> - install linux on the board
>>> - check if booted version is the expected one
>>> - create a register dump file
>>>    write register content into a register dump file
>>> - do register checks
>>
>> See above. I think this is useful during a porting, but it is less
>> useful for a customer who wants to test functionalities. I would like to
> 
> I have here another opinion.

Well, of course ;-). We should not always agree, we get more improvement
when we discuss and have different opinions ! ;-)

> 
> This is also interesting for a customer.
> 
> Which customer does never change a DTS or does not try a linux update
> on his own?
> 
> If he have an automated check, if all important registers are setup
> as expected ... this is nice.
> 
> This testcase could be done very generic...
> 
>> have first a catalog of testcases with functionalities, like:
>>
>>     - is network working ?
>>     - are peripherals working (SPI / I2C /....) ?
> 
> Yes. My hope is, that we get a lot of users, so we will get a lot of
> testcases ;-)

ok

> 
>> In the ideal case, DT is parsed to get a list of testcases...
> 
> Yes.
> 
>>>    open register dump file and check if register content
>>>    is the same as in the file
>>> - look if a list of string are in dmesg output
>>>
>>> - look for example at the LTP project, what they test
>>>
>>
>> +1
>>
>> LTP contains a lot of useful testcases, but of course they are meant to
>> run as scripts directly on the target / host. Anyway, they have
>> testcases for a lot of things.
> 
> Yes, and we may can use this scripts! Start them and analyse the results.
> 

ok, I let this for later, it is not clear to me how...

>>> - check if ptest-runner is in the rootfs and call it
>>
>> ptest-runner means python. Do we have it on most projects ? Some yes,
>> some not...
> 
> Therefore is "check if ptest-runner" exists ;-)
> 
>>> ...
>>>
>>> yocto:
>>> - get the sources
>>> - configure
>>> - bake
>>> - check if files you are interested in are created
>>> - install new images
>>> - boot them
>>> - check if rootfsversion is correct
>>
>> See above - IMHO it is better to split between functional tests on
>> target and build, and to start with the functional tests.
> 
> Of course. Both parts can be done independently

Sure !

Stefano

-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de
=====================================================================