DeploymentConfig behaviour change from 3.6 to 3.7 #18406

mojsha · 2018-02-02T09:24:18Z

In 3.6, using F-M-P which updates the both the container image reference and the image trigger reference, there was no automatic re-deployment when trigger is set to automatic false.

In 3.7, it seems like an update to the container image reference automatically re-deploys.

This breaks our workflow.

Version

oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth

openshift v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62

Steps To Reproduce

3.7:

Create a DC and deploy it using the rollout command.
Update the image container reference and redeploy.

3.6:

Create a DC and deploy it using the rollout command.
Update the image container reference and redeploy.

Current Result

3.7:
It will trigger a deployment.
3.6:
Nothing will happen.

Expected Result

3.7:
Nothing should happen (as in 3.6) until a manual rollout command has been executed.

@soltysh @mfojtik @bparees

bparees · 2018-02-02T14:58:52Z

starting as a p1 because it's a regression/change in behavior (possibly intentional and justifiable)

bparees · 2018-02-02T14:59:28Z

(if anything it sounds like the 3.6 behavior was broken, so perhaps this was a bug fix in 3.7)

mojsha · 2018-02-05T08:19:50Z

@bparees From my perspective, a change in the DC - any change - should only be triggered if there is a ConfigChange trigger. That is, after all, why we have the ConfigChange trigger.

If anything else in the DC is changed, it does not re-deploy. So if this is intentional behaviour, it would effectively amount to an implicit/hidden trigger on the container image reference being introduced.

@LarsMilland

mojsha · 2018-02-05T09:10:55Z

Ok, after further analysis I can see that F-M-P is doing a PUT which triggers the re-deploy.

If I do a PATCH with the same content and change the container image reference, it does not seem to trigger a re-deploy.

0xmichalis · 2018-02-05T10:37:44Z

What is FMP?

mojsha · 2018-02-05T10:39:34Z

@Kargakis Sorry, I thought it was a known abbreviation/acronym... it stands for Fabric8 Maven Plugin.

0xmichalis · 2018-02-05T10:44:04Z

Can you post the DC before and after doing a PUT and before and after doing a PATCH?

cc @tnozicka

tnozicka · 2018-02-05T10:49:34Z

That what I was going to ask, please always provide the DC's YAML and exact steps to reproduce (ideally commands) - it will save us time when trying to understand and resolve the issue.

mojsha · 2018-02-05T11:35:31Z

Ok, I think I was wrong about the PUT vs PATCH thing. The behaviour is the same regardless.

But I will try to provide all the steps with a higher level of detail:
I use a disabled (automatic = false) ImageTrigger in order to be able to refer to images using the ImageStream tags, so the "before" is going to be the state after which it has been rolled out and the container image resolved.

Before PUT / PATCH:

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: 172.30.251.31:5000/fabric8test/demo@sha256:b49793e3c286cbd491a4790bca06c4ea8a805568df1e75b8bd075cd3e298fe5c
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.37-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After PUT (using F-M-P)

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After PATCH

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

mojsha · 2018-02-05T11:44:27Z

And finally, the reason I was wrong about the difference between PUT / PATCH:

When you PUT / PATCH an undeployed (i.e. not yet had 'oc rollout' executed against it) application, it seems to work fine, i.e. it does not trigger a deployment. I had executed the PATCH against an undeployed application.

Here is the DC before and after an undeployed app has been patched:

Before

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.40-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

tnozicka · 2018-02-05T12:11:13Z

Thanks for the info I think I now now what's wrong. The trigger controller won't inject an image because it's paused, but since you set it "manually" the DC controller thinks there has been an image injected and considers that an image change.

I think that should be fixable by the DC controller also checking if the trigger isn't paused.

There should be an easy way to around it in the mean time: If you are using ImageChangeTrigger always set container's image to " ". Actually that applies in general - if there is a configChangeTrigger never set image to anything else than " ". That way even if you need to do something like PUT or oc apply won't mess it up.

mojsha · 2018-02-06T11:24:00Z

@tnozicka Thank you for this suggestion - it worked!

EDIT: @tnozicka My earlier experiment was unfortunately not entire correctly placed. It does work - the first time. Any subsequent rollout leads to a problem with the RC complaining about a missing image reference and seems to be an issue you are well aware of: #16728.

Can you tell me how I should proceed? This is a huge blocker for my organization - we cannot proceed with upgrade to 3.7+ due to this, as all our projects are dependent on this build and deployment mechanism.

Can you backport the fix to a 3.7.x-version, or provide an alternative way short of waiting for 3.9 to be released? Much appreciated.

tnozicka · 2018-02-07T13:58:48Z

I'll check and see how much conflict there will be in release-3.7.

tnozicka · 2018-02-07T13:59:41Z

Not sure whether there will be another release of 3.7.x though.

tnozicka · 2018-02-08T11:57:05Z

backport to 3.7 #18524

jperville · 2018-02-08T17:39:56Z

Hello @tnozicka , I try to apply your workaround from #18406 (comment) which is to set the DC's container.image to " " (string containing a space).

Using OSE v3.7.0, I am unable to deploy the DC with container.image set to " " and automatic imageChangeTrigger, the following messages appears in Events "Couldn't deploy version 5: ReplicationController "xxxx-5" is invalid: spec.template.spec.containers[0].image: Required value".

tnozicka · 2018-02-12T12:42:06Z

Hi @jperville, that backport is not merged yet (hitting unrelated flake)

Using OSE v3.7.0, I am unable to deploy the DC with container.image set to " " and automatic imageChangeTrigger, the following messages appears in Events "Couldn't deploy version 5: ReplicationController "xxxx-5" is invalid: spec.template.spec.containers[0].image: Required value".

OSE backport will follow once that PR makes it through the queue in Origin ;)

I am unable to deploy the DC with container.image set to " "

That is strange as this is just short time failure as the image in DC will get eventually rewritten by image trigger causing a new rollout with valid image.

tnozicka · 2018-02-12T12:45:42Z

maybe it doesn't get synced; that part is being fixed in that backport as well

tnozicka · 2018-04-25T07:12:43Z

Both backports are merged now.

bparees assigned mfojtik Feb 2, 2018

bparees added priority/P1 component/apps kind/bug Categorizes issue or PR as related to a bug. labels Feb 2, 2018

tnozicka assigned tnozicka and unassigned mfojtik Feb 5, 2018

0xmichalis added the sig/master label Feb 6, 2018

rohanKanojia mentioned this issue Feb 8, 2018

application redeployment is getting failed for OpenShift v3.7.0 fabric8io/fabric8-maven-plugin#1130

Closed

galderz mentioned this issue Feb 11, 2018

OpenShift unable to complete deployment #18303

Closed

piyush-garg mentioned this issue Mar 26, 2018

application redeployment is getting failed for OpenShift v3.7/minishift v1.7.0+1549135 minishift/minishift#1821

Closed

piyush-garg mentioned this issue Apr 13, 2018

Regression Tests are not working properly on Openshift 3.7.0 and 3.7.1 fabric8io/fabric8-maven-plugin#1272

Closed

tnozicka closed this as completed Apr 25, 2018

DeploymentConfig behaviour change from 3.6 to 3.7 #18406

DeploymentConfig behaviour change from 3.6 to 3.7 #18406

Comments

mojsha commented Feb 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Version

Steps To Reproduce

Current Result

Expected Result

bparees commented Feb 2, 2018

Uh oh!

bparees commented Feb 2, 2018

Uh oh!

mojsha commented Feb 5, 2018

Uh oh!

mojsha commented Feb 5, 2018

Uh oh!

0xmichalis commented Feb 5, 2018

Uh oh!

mojsha commented Feb 5, 2018

Uh oh!

0xmichalis commented Feb 5, 2018

Uh oh!

tnozicka commented Feb 5, 2018

Uh oh!

mojsha commented Feb 5, 2018

Uh oh!

mojsha commented Feb 5, 2018

Uh oh!

tnozicka commented Feb 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mojsha commented Feb 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnozicka commented Feb 7, 2018

Uh oh!

tnozicka commented Feb 7, 2018

Uh oh!

tnozicka commented Feb 8, 2018

Uh oh!

jperville commented Feb 8, 2018

Uh oh!

tnozicka commented Feb 12, 2018

Uh oh!

tnozicka commented Feb 12, 2018

Uh oh!

tnozicka commented Apr 25, 2018

Uh oh!

mojsha commented Feb 2, 2018 •

edited

Loading

tnozicka commented Feb 5, 2018 •

edited

Loading

mojsha commented Feb 6, 2018 •

edited

Loading