[U-Boot] [RFC PATCH] test: py: Disable sleep test for qemu targets

[Michal Simek]

Hi,

On 1.12.2017 16:06, Heinrich Schuchardt wrote:
> 
> 
> On 12/01/2017 03:46 PM, Michal Simek wrote:
>> Qemu for arm32/arm64 has a problem with time setup.
> 
> Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

> 
>> That's why sleep test is failing. Add boardidentity marker to remove
>> specific boards from running that test.
> 
> Isn't this what 'boardspec' is used for?

There are two things here.

./test/py/test.py --bd zynq_zc706 --build  --board-identity zc706  -s

boardspec is zynq_z706 -> u-boot config
and I use board identity (zc706) as actual HW I have here

And I use this
./test/py/test.py --bd zynq_zc706 --build  --board-identity qemu -s
which means run the same config for zc706 but on qemu

etc.

> 
> test/py/pytest.ini:
> boardspec: U-Boot: Describes the set of boards a test can/can't run on.

as above. This file should be also update if this "feature" is fine.


>>
>> Signed-off-by: Michal Simek <michal.simek at xilinx.com>
>> ---
>>
>> If you want to add this checking in one patch and then put it to one
>> test then it is fine for me.
>>
>> ---
>>   test/py/conftest.py         | 28 ++++++++++++++++++++++++++++
>>   test/py/pytest.ini          |  1 +
>>   test/py/tests/test_sleep.py |  1 +
>>   3 files changed, 30 insertions(+)
>>
>> diff --git a/test/py/conftest.py b/test/py/conftest.py
>> index 6e66a48c15fd..1812ff340e6a 100644
>> --- a/test/py/conftest.py
>> +++ b/test/py/conftest.py
>> @@ -436,6 +436,33 @@ def setup_boardspec(item):
>>       if required_boards and ubconfig.board_type not in required_boards:
>>           pytest.skip('board "%s" not supported' % ubconfig.board_type)
>>   +def setup_boardidentity(item):
>> +    """Process any 'boardidentity' marker for a test.
>> +
>> +    Such a marker lists the set of board identity that a test
>> does/doesn't
>> +    support. If tests are being executed on an unsupported board, the
>> test is
>> +    marked to be skipped.
>> +
>> +    Args:
>> +        item: The pytest test item.
>> +
>> +    Returns:
>> +        Nothing.
>> +    """
>> +    mark = item.get_marker('boardidentity')
>> +    if not mark:
>> +        return
>> +    required_boards = []
>> +    for board in mark.args:
>> +        if board.startswith('!'):
>> +            if ubconfig.board_identity == board[1:]:
>> +                pytest.skip('board identity not supported')
>> +                return
>> +        else:
>> +            required_boards.append(board)
>> +    if required_boards and ubconfig.board_identity not in
>> required_boards:
>> +        pytest.skip('board identity not supported')
>> +
>>   def setup_buildconfigspec(item):
>>       """Process any 'buildconfigspec' marker for a test.
>>   @@ -503,6 +530,7 @@ def pytest_runtest_setup(item):
>>         start_test_section(item)
>>       setup_boardspec(item)
>> +    setup_boardidentity(item)
>>       setup_buildconfigspec(item)
>>       setup_requiredtool(item)
>>   diff --git a/test/py/pytest.ini b/test/py/pytest.ini
>> index 67e514f42058..9d64671814de 100644
>> --- a/test/py/pytest.ini
>> +++ b/test/py/pytest.ini
>> @@ -8,4 +8,5 @@
>>   [pytest]
>>   markers =
>>       boardspec: U-Boot: Describes the set of boards a test can/can't
>> run on.
>> +    boardidentity: U-Boot: Describes the board identity a test
>> can/can't run on.
>>       buildconfigspec: U-Boot: Describes Kconfig/config-header
>> constraints.
>> diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py
>> index 64e057132622..02a8a85b0e22 100644
>> --- a/test/py/tests/test_sleep.py
>> +++ b/test/py/tests/test_sleep.py
>> @@ -5,6 +5,7 @@
>>   import pytest
>>   import time
>>   + at pytest.mark.boardidentity("!qemu")
> 
> According to your commit message you don't want to exclude qemu-x86 here.

It is really just qemu board identity not qemu target.

The issue with this patch is that everybody can select whatever they
want to for board identification.

Maybe it would be much simpler to create variable and check it in that
test and skip that test.

Anyway this is what I have in the tree and it is time to find proper
solution that's why RFC.

Thanks,
Michal