[PATCH] test/py: tpm2: Skip tpm pytest based on env variable

[Michal Simek]

Hi

On 9/18/21 11:48 AM, Ilias Apalodimas wrote:
> Hi Tom,
> 
> On Fri, 17 Sept 2021 at 15:08, Tom Rini <trini at konsulko.com> wrote:
>>
>> On Fri, Sep 17, 2021 at 11:49:51AM +0000, T Karthik Reddy wrote:
>>> Hi Tom,
>>>
>>>> -----Original Message-----
>>>> From: Tom Rini <trini at konsulko.com>
>>>> Sent: Thursday, August 5, 2021 6:53 PM
>>>> To: T Karthik Reddy <tkarthik at xilinx.com>
>>>> Cc: Ashok Reddy Soma <ashokred at xilinx.com>; u-boot at lists.denx.de;
>>>> miquel.raynal at bootlin.com; git <git at xilinx.com>
>>>> Subject: Re: [PATCH] test/py: tpm2: Skip tpm pytest based on env variable
>>>>
>>>> On Thu, Aug 05, 2021 at 05:56:50AM +0000, T Karthik Reddy wrote:
>>>>> Hello Tom Rini,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Tom Rini <trini at konsulko.com>
>>>>>> Sent: Wednesday, August 4, 2021 8:52 PM
>>>>>> To: Ashok Reddy Soma <ashokred at xilinx.com>
>>>>>> Cc: u-boot at lists.denx.de; miquel.raynal at bootlin.com; git
>>>>>> <git at xilinx.com>; T Karthik Reddy <tkarthik at xilinx.com>
>>>>>> Subject: Re: [PATCH] test/py: tpm2: Skip tpm pytest based on env
>>>>>> variable
>>>>>>
>>>>>> On Fri, Jul 23, 2021 at 06:18:26AM -0600, Ashok Reddy Soma wrote:
>>>>>>
>>>>>>> From: T Karthik Reddy <t.karthik.reddy at xilinx.com>
>>>>>>>
>>>>>>> Tpm test cases relies on tpm device setup. Provide an environment
>>>>>>> variable "env__tpm_device_test_skip = True" to skip the test case
>>>>>>> if tpm device is not present.
>>>>>>> Only needed will have to add variable to the py-test framework.
>>>>>>> Test runs successfully even this variable is absent.
>>>>>>>
>>>>>>> Signed-off-by: T Karthik Reddy <t.karthik.reddy at xilinx.com>
>>>>>>> Signed-off-by: Ashok Reddy Soma <ashok.reddy.soma at xilinx.com>
>>>>>>
>>>>>> Why is this not using boardenv_* like other tests do to disable running?
>>>>>
>>>>> What I meant here was this test doesn't depend on configuration value
>>>>> from boardenv_*. But it uses env__tpm_device_test_skip variable from
>>>> the configuration file.
>>>>> I referred test/py/tests/test_sleep.py for writing this test case, hence
>>>> copied comments as well.
>>>>> Should I change this NOTE in comment section?
>>>>
>>>> Thanks for explaining.  This got me to re-read the existing tests and while
>>>> none of it is as clear as I would like, your example here is just as clear as the
>>>> rest, so it's fine and I'll pick it up soon.
>>>
>>> I see this patch is not yet applied on the repo, could you please let me know when this will be applied?
>>
>> Ilias, since you've said you'd review TPM related patches, does this:
>> https://patchwork.ozlabs.org/project/uboot/patch/1624340962-91190-1-git-send-email-t.karthik.reddy@xilinx.com/
>> work for you?  Thanks.
>>
> 
> I am not too familiar with the pytest framework we have, but couldn't
> we do this automatically?
> IOW instead of explicitly adding a variable, do a 'tpm info' and if
> that doesn't return a valid device, skip the tests.

Unfortunately not. Because imagine how this will work if you have tpm on
board but there is bug in the driver that it is not probed. Then tpm
info will return no device and all tests will be silently skipped.

It would be much better it the test itself was designed from the
beginning that you need to state that board has TPM. But it wasn't
that's why we designed this patch in a way that you declare that you
want to skip these tests which is OK.

It doesn't make any sense to go back and start to have multiple
defconfigs for different configurations where all boards works quite
well with one defconfig. That's why we enable TPM because it on SOM but
we need to be able to disable these tests on others.

Thanks,
Michal