Slow mount of PVC for che workspaces - can prevent workspaces from starting.

[rhopp]

gitlab issue - https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/2169
Mounting PVC which has lots of files is very slow and it can even prevent che workspaces from starting.

To put it into perspective - It's enough to have 3 workspaces with Angular [1] example to get into the state, where workspaces fail to start.

Right now, this may not be such a big issue (but we had people in the past who were affected by this), but this won't scale in the future... Even if we raise the timeout for workspaces to start, it would be unbearalbe to wait for startup for ~10 minutes or more.

Steps to reproduce

Basically fill up claim-che-workspace PVC with lots of files and try to start workspace - observe that it takes either too long, or fails after ~10 minutes.

OR use "more real world" approach:

• Create workspace from factory: https://che.openshift.io/f?id=factory3db802wkfbg8wox5
  This link will spawn up workspace with Angular quickstart and immediately starts build - this generates (downloads) few thousands files.
• Stop that workspace
• Create new workspace with the same (or any other quickstart), do build (a.k.a. generate lots of files)
• Stop the second workspace
• Create third workspace - When I used 2 workspaces with Angular quickstart, this third one fails to start for me.

[1] - Created from factory - https://www.eclipse.org/che/getting-started/cloud-osio/

[rhopp]

@ScrewTSW also created job, which is monitoring mount times of PVC. This PVC was pre-populated with filex equivalent to two built angular quickstarts (~20000 files if I remember that correctly). @ScrewTSW Could you please provide link to zabbix?

[ScrewTSW]

Here are the job results with two angular projects installed.

Number of files in project:

Events from openshift console:

Link to Zabbix graph with data colelcted:
https://zabbix.devshift.net:9443/zabbix/charts.php?ddreset=1

[rhopp]

This have one more implication...

Normally, when workspace is deleted, the rm-<workspace-name> pod is spawned. It's purpose is to delete workspace files from the PVC.

But when the PVC has lots of files, the mount fails (or takes longer, then is the timeout for startup of the rm- pod) and user is basically stuck in situation when he cannot start any workspace AND he is not able to remove the data from PVC (well.. technically he can deploy some own pod with mounted PVC and delete it manually from terminal, but this will be disabled, when we revoke admin rights from users -che namespaces)

WDYT @davidfestal ?

[slemeur]

@rhopp : https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/2169

[ibuziuk]

@rhopp could you please test how it works on prod-preview where deployments are used instead of bare pods - eclipse-che/che#10021 (comment) ?

[rhopp]

@ibuziuk On prod-preview, when the failure happens, it happens with weird message (timeout 0 miliseconds, even though it fails after few minutes):

Error when starting agent
Unable to start workspace agent. Error when trying to start the workspace agent: Timed out waiting for [0] milliseconds for [Deployment] with name:[workspaceekokh2u0tab43lgn.dockerimage] in namespace [rhopp-preview-che].

[slemeur]

That's the one I hit last week indeed.

[ibuziuk]

@rhopp @slemeur but the message in openshift events is still about the Fail Mount problem, right ?
I believe the message exposed to che would be better once the following regression would be fixed - eclipse-che/che#10559

[sbose78]

Could you please tell me if there's been progress on resolving this?

[ScrewTSW]

@sbose78 Hello.
I've tested the new update on the free-stg cluster. I really tried to break it, but I was unable to :D
I've mounted a simple pod with the PVC attached, created 30 000 artificial files + 6 large ones besides those that already existed from the workspaces previously run with that account (around 8000 more real files)
I tried mounting the volume and creating new workspaces back and forth and I didn't see any issues with mount or increase in the time it took to mount the volume and start the pod/workspace.
I even tried a https://che.prod-preview.openshift.io/dashboard/#/load-factory?id=factoryqlvwzsnbpcnvwxbo factory. The start of workspace and cloning of the project worked without any issues.
The build is where the factory failed, but that's already a known bug.
The cluster seems to be stable. 👍 for promoting to prod-preview

[sbose78]

@ScrewTSW Fantastic! Thank you.

The build is where the factory failed, but that's already a known bug.

yeah, this should be OK.

[ibuziuk]

@ScrewTSW could you please verify and close this issue if all the starter clusters have been updated and the issue is not reproducible anymore ? cc: @rhopp

[ibuziuk]

All starter clusters have been updated this week to openshift 3.11. However, yet another requirement for fixing this slow volume mount issue is having gluster-subvol driver fix [1] deployed to the prod clusters. I will provide an update here when the fix would be available on prod clusters and qa verification could be done.

[1] gluster/gluster-subvol#24

[ibuziuk]

@rhopp @ScrewTSW @Katka92 the gluster subvol fix is applied on 1a cluster. Will you be able to verify that volume mount problem is not reproducible there and +1 for applying the same fix to all the other starter clusters?

[ppitonak]

I can confirm that e2e tests started passing on this cluster. 12 of 12 runs successful so far.

https://ci.centos.org/job/devtools-test-e2e-openshift.io-smoketest-us-east-1a-released/
https://ci.centos.org/job/devtools-test-e2e-openshift.io-smoketest-us-east-1a-beta/

[ibuziuk]

@ppitonak are we positive that the fact that tests started to pass related to the gluster subvol fix (are those were failing due to slow volume mount before?) and not fix for #4626 ?

[ppitonak]

I'm not sure but

you commented in this issue on Dec 12, 18:16 UTC
fix for #4626 got into production on Dec 12, 19:13 UTC
first successful build started on Dec 12 18:35 UTC

[ibuziuk]

@rhopp could someone from QA team with the account provisioned against 1a make sure that it is not possible to reproduce the problem by following the steps to reproduce from the description ?

[rhopp]

PVC mount issue seems to be fixed on 1a cluster. Workspace container starts quickly even with lots of files inside.

I've encountered another (not so critical issue) while testing that... I'll report that as separate issue.

[dak1n1]

The update has now been applied to the rest of OSIO.

[ibuziuk]

@rhopp closing this issue