[U-Boot] [RFC PATCH] test: py: Disable sleep test for qemu targets

Tue Dec 5 20:46:01 UTC 2017

On 05.12.2017 19:38, Tom Rini wrote:
> On Tue, Dec 05, 2017 at 11:20:57AM -0700, Stephen Warren wrote:
>> On 12/04/2017 04:21 PM, Tom Rini wrote:
>>> On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote:
>>>> On 12/04/2017 08:30 AM, Tom Rini wrote:
>>>>> On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote:
>>>>>> On 4.12.2017 15:03, Tom Rini wrote:
>>>>>>> On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote:
>>>>>>>> On 1.12.2017 23:44, Tom Rini wrote:
>>>>>>>>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote:
>>>>>>>>>> On 12/01/2017 08:19 AM, Michal Simek wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote:
>>>>>>>>>>>>> Qemu for arm32/arm64 has a problem with time setup.
>>>>>>>>>>>>
>>>>>>>>>>>> Wouldn't it be preferable to fix the root cause?
>>>>>>>>>>>
>>>>>>>>>>> Definitely that would be the best and IIRC I have tried to convince our
>>>>>>>>>>> qemu guy to do that but they have never done that.
>>>>>>>>>>
>>>>>>>>>> What is the exact failure condition? Is it simply that the test is still
>>>>>>>>>> slightly too strict about which delays it accepts, or is sleep outright
>>>>>>>>>> broken?
>>>>>>>>>>
>>>>>>>>>> You can use command-line option -k to avoid some tests. For example "-k not
>>>>>>>>>> sleep". That way, we don't have to hard-code the dependency into the test
>>>>>>>>>> source. Depending on the root cause (issue in U-Boot, or issue in just your
>>>>>>>>>> local version of qemu, or something that will never work) this might be
>>>>>>>>>> better?
>>>>>>>>>
>>>>>>>>> Even with the most recent relaxing of the sleep test requirements, I can
>>>>>>>>> still (depending on overall system load) have 'sleep' take too long, on
>>>>>>>>> QEMU.  I think it might have been half a second slow, but I don't have
>>>>>>>>> the log handy anymore.  Both locally and in travis we -k not sleep all
>>>>>>>>> of the qemu instances.
>>>>>>>>
>>>>>>>> ok. By locally do you mean just using -k not sleep?
>>>>>>>
>>>>>>> Yes, I have that in my CI scripts and similar.
>>>>>>
>>>>>> Wouldn't be easier to keep this in uboot-test-hooks repo with other
>>>>>> target setting?
>>>>>
>>>>> Or do as you did did and mark the tests as not allowed for qemu, yes.
>>>>>
>>>>>> What we are trying to do is that our testing group will run these tests
>>>>>> for me that's why it is just easier for me to change local
>>>>>> uboot-test-hooks repo instead of communicate with them what -k not XXX
>>>>>> parameters to add to certain scripts.
>>>>>>
>>>>>> It means in loop they will just run all tests on qemu, local targets and
>>>>>> in boardfarm. It is probably not big deal to tell them to add -k not
>>>>>> sleep for all qemu runs but I know that for some i2c testing qemu
>>>>>> doesn't emulate these devices that's why these tests fails. And the
>>>>>> amount of tests which we shouldn't run on qemu will probably grow.
>>>>>
>>>>> Well, I'm still open to possibly tweaking the allowed variance in the
>>>>> sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to
>>>>> "sleep should be pretty darn accurate on HW" for the test too, and
>>>>> perhaps that's best.
>>>>
>>>> The fundamental problem of "over-sleeping" due to host system load/.. exists
>>>> with all boards. There's nothing specific to qemu here except that running
>>>> U-Boot on qemu on the host rather than on separate HW might more easily
>>>> trigger the "high load on the host" condition; I see the issue now and then
>>>> and manually retry that test, although that is a bit annoying.
>>>>
>>>> The original test was mostly intended to make sure that e U-Boot clock
>>>> didn't run at a significantly different rate to the host, since I had seen
>>>> that issue during development of some board support or as a regression
>>>> sometime. Perhaps the definition of "significantly different" should be more
>>>> like "1/2 rate or twice rate or more" rather than "off by a small fraction
>>>> of a second". That might avoid so many false positives.
>>>
>>> I've pushed this up to 10 seconds and 0.5s worth of overrun and on
>>> qemu-mips here I see a 13.2s sleep.  That's pretty close to 1/3rd fast
>>> and to me a wrong-clocking value, yes?
>>
>> For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same
>> qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5
>> seconds (including my reaction time), so ~13.2 does sound like it's probably
>> a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with
>> real-time? I'd hope not for something simple like this, assuming you're
>> using a recent CPU, but maybe.
> 
> Yeah, I can do x86, ARM and PowerPC but it fails on MIPS.  And my build
> box isn't super new but an 8core/16thread E5-2670 should be good enough
> :)

the problem with MIPS is that the timer frequency is hard-coded with
config option CONFIG_SYS_MIPS_TIMER_FREQ in all older boards and the
boards supported by Qemu (qemu_mips, malta, boston). The reason is that
the timer frequency can't be derived from some hardware register. This
is only possible with modern MIPS SoCs, where one can derive the timer
frequency directly (or indirectly via the CPU frequency) from a PLL.

Also I can't see if or how the relevant Qemu timers for MIPS
(mips_gictimer.c, i8254.c) are throttled to a deterministic value.
Therefore I assume the timers are running as fast as possible. Thus
there always will be a discrepancy between the hard-coded U-Boot timer
frequency and the Qemu timer frequency dependent on the host system
where Qemu runs. Due to this it doesn't make sense to enable the sleep
test on MIPS.

I can think of following options to fix this:

1) U-Boot gets the possibilty to read the actual timer frequency somehow
from Qemu

2) estimate the timer frequency by reading the CPU counter register at
different times. But than we need some reference time or emulated RTC.
At least for the Malta board the kernel does this.

3) the Qemu timer frequency can be throttled to a deterministic value
(maybe configurable via Qemu command line argument).

4) keep the sleep test disabled

Or does someone know if and how the QEMU_CLOCK_VIRTUAL frequency can be
configured?

> 
> Hey Daniel, do you have test.py running on real MIPS hardware?  Thanks!
> 

unfortunately not. I don't have any of the boards supported in mainline.
And the boards I have aren't in mainline yet ;)

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171205/ed9f04ac/attachment.sig>