Cluster provision fails - registry deployment

[bparees]

TASK [openshift-registry : Deploy latest configuration of registry DC] *********
Monday 12 June 2017  11:59:39 +0000 (0:00:00.577)       0:18:41.437 *********** 
fatal: [ci-prtest-5a37c28-3006-ig-m-5g51]: FAILED! => {"changed": true, "cmd": ["oc", "deploy", "docker-registry", "--latest"], "delta": "0:00:00.271225", "end": "2017-06-12 07:59:39.671711", "failed": true, "rc": 1, "start": "2017-06-12 07:59:39.400486", "stderr": "Flag --latest has been deprecated, use 'oc rollout latest' instead\nerror: #1 is already in progress (Running).\nOptionally, you can cancel this deployment using 'oc rollout cancel dc/docker-registry'.", "stderr_lines": ["Flag --latest has been deprecated, use 'oc rollout latest' instead", "error: #1 is already in progress (Running).", "Optionally, you can cancel this deployment using 'oc rollout cancel dc/docker-registry'."], "stdout": "", "stdout_lines": []}

Failure summary:

  1. Host:     ci-prtest-5a37c28-3006-ig-m-5g51
     Play:     primary_master
     Task:     openshift-registry : Deploy latest configuration of registry DC
     Message:  ???

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/3006/consoleFull

[stevekuznetsov]

Could this be #13995?

/cc @sdodson

[0xmichalis]

Could this be #13995?

Or maybe #14589

[0xmichalis]

Or maybe #14589

Sorry, wrong issue. The present issue seems similar to #14603

[sdodson]

Is this a change in how deployments are managed? I believe we've always modified the deployment potentially before it finished but we've never observed an error before.

We should definitely fix this regardless.

[sdodson]

This code is actually in origin-gce. @smarterclayton Can we not just use openshift-ansible's registry deployment?

[smarterclayton]

It might be, the ref arch was doing something extra that made it work on GCE, before it was supported in openshift-ansible.

[0xmichalis]

Cluster provisioning in GCE is still an issue: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/batch/test_pull_request_origin_extended_conformance_gce/15850/

[0xmichalis]

Raising priority since this is happening frequently today
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18437/test_pull_request_origin_extended_conformance_gce/15858/
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/13701/test_pull_request_origin_extended_conformance_gce/15857/
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18478/test_pull_request_origin_extended_conformance_gce/15855//test_pull_request_origin_extended_conformance_gce/15851/
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/batch/test_pull_request_origin_extended_conformance_gce/15850/
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18476/test_pull_request_origin_extended_conformance_gce/15849/

/priority P0
/remove-priority P1

[0xmichalis]

The api server seems to never turn ready.

[0xmichalis]

@openshift/sig-master

[mfojtik]

Inspected the API logs, there is no error there. From the error in ansible log it seems like ansible can't reach the DNS: https://internal-api.prtest-5a37c28-15849.origin-ci-int-gce.dev.rhcloud.com:8443/healthz/ready as the "curl" command exit with 6 code which means CURLE_COULDNT_RESOLVE_HOST.

[mfojtik]

Moving back to @sdodson as this is unlikely API server issue.

[0xmichalis]

Opened a separate issue: #18525

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[stevekuznetsov]

/close