3.7.0 cluster can't run pods: MountVolume.SetUp failed for volume...

[tnozicka]

Cluster was created by oc cluster up --version=v3.7.0

$ oc version
oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

All pod (including deployers for registry and router) are stuck on ContainerCreating (https://travis-ci.org/tnozicka/openshift-acme/builds/312537376#L1389)

This error is repeating very frequently:

E1206 17:38:26.399937    9919 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/439509e0-daac-11e7-8160-42010a1400bd-deployer-token-kbtwg\" (\"439509e0-daac-11e7-8160-42010a1400bd\")" failed. No retries permitted until 2017-12-06 17:38:26.89990613 +0000 UTC (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "deployer-token-kbtwg" (UniqueName: "kubernetes.io/secret/439509e0-daac-11e7-8160-42010a1400bd-deployer-token-kbtwg") pod "router-1-deploy" (UID: "439509e0-daac-11e7-8160-42010a1400bd") : exit status 1

master log is here: https://travis-ci.org/tnozicka/openshift-acme/builds/312537376#L2196

[tnozicka]

(3.6.1 works fine)

[sjenning]

When I try, I get this error event on all pods:

Error: failed to start container "deployment": Error response from daemon: {"message":"oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:364: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/origin/openshift.local.volumes/pods/63c18b21-def4-11e7-a418-1c1b0deeddfa/volumes/kubernetes.io~secret/deployer-token-vz4gl\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/65e637731cffd516b61a69293b3e354f3d3a4472bdf2bd57723dd469b477d043/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/65e637731cffd516b61a69293b3e354f3d3a4472bdf2bd57723dd469b477d043/merged/run/secrets/kubernetes.io/serviceaccount\\\\\\\" caused \\\\\\\"mkdir /var/lib/docker/overlay2/65e637731cffd516b61a69293b3e354f3d3a4472bdf2bd57723dd469b477d043/merged/run/secrets/kubernetes.io: read-only file system\\\\\\\"\\\"\"\n"}

@jsafrane likely a problem with containerized node mounting volume. Any help?

[jsafrane]

Containerized mount is alpha (i.e. behind a feature gate) in 1.9, mount propagation is alpha in 1.8, still behind a gate. This must be something else.

[sjenning]

@joelsmith PTAL

[tnozicka]

oc cluster up for 3.7.1 seems borked as well #18262