gtlab slow?

[Claudius Heine]

Hi everyone,

On 11/12/2019 09.25, Heiko Schocher wrote:
> Hello Harald,
> 
> Am 10.12.2019 um 18:18 schrieb Harald Seiler:
>> Hi Simon,
>>
>> On Tue, 2019-12-10 at 05:48 -0700, Simon Glass wrote:
>>> Hi,
>>>
>>> I sometimes find that gitlab.denx.de is very slow. Just now is is
>>> worse than usual - it has taken almost 10 minutes to push a branch and
>>> still not finished. I am getting a 503 error on the web site.
>>>
>>> Any ideas?
>>
>> Yes, we noticed that as well but so far have failed to find out the
>> exact reason.  Server-side we see horrible load spikes while the CPU is
>> essentially in idle and no notable IO is happening.  I personally
>> believe it might have something to do with the virtualization technology
>> but we don't have a certain answer yet.
>>
>> I added Claudius in CC, he might know more ...
> 
> May dummy thought ... could the gitlab runners be the reason? If we
> push to a u-boot repo on gitlab, we automatically start now a gitlab
> runner... so if there "a lot of" pushs may a lot of jobs are pending?

No probably not. I don't think the problem is anything under our control.

A bit of background information: Our gitlab is hosted on a virtual
server from the provider Strato. Strato uses Virtuozzo/OpenVZ as a
virtualization platform, which is sort of container based with a kernel
that is shared by all guests. So as a customer we have access to the
root file system, but most privileged access to the running kernel is
prohibited. (no dmesg, etc.)

With our monitoring tools, we can see the load sometimes spikes go over
100 while we have about 10GiB free memory and not much going on disk or
network. The cpu mostly idles but there are a lot of tasks in D state.
When the load is high it also seems to be that the cache/buffer of the
kernel is at about 200MiB. When the load is low then the cache/buffer is
about 5GiB. So there seems to be something off with the cache and/or IO,
which both is not under our control. (We don't even have access to block
level statistics)

Strato apparently took our gitlab instance at about 23:52 to 01:04 CET
offline last night and the load now seems more stable. Hopefully they
resolved this issue now. *knocking on wood*

regards,
Claudius

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20191211/2f9eea28/attachment.sig>