Proposal: Clouformation wait for signal

[gianrubio]

In the current cloudformation template, kube-aws use PauseTime flag to wait for create/update resources.

    UpdatePolicy" : {
        "AutoScalingRollingUpdate" : {
          "MinInstancesInService" : "{{.ControllerCount}}",
          "MaxBatchSize" : "1",
          "PauseTime" : "PT2M"
        }
      }

My proposal is to use cfn-signal to control the resources [1][2] and only send the signal after the last service was started. I can implement this proposal and I'm here to listen for suggestions.

[Unit]
Description=Cloudformation Signal Ready
After=docker.service
Requires=docker.service
After=etcd2.service
Requires=etcd2.service

[Install]
WantedBy=multi-user.target

[Service]
Type=oneshot
EnvironmentFile=/etc/environment
ExecStart=/bin/bash -c '\
eval $(docker run crewjam/ec2cluster); \
docker run --rm crewjam/awscli cfn-signal \
    --resource MasterAutoscale --stack $TAG_AWS_CLOUDFORMATION_STACK_ID \
    --region $REGION; \
'

[1] https://aws.amazon.com/blogs/devops/use-a-creationpolicy-to-wait-for-on-instance-configurations/
[2] http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-creationpolicy.html

[pieterlange]

Big fan of this idea but i'm seeing some issues here:

• Running containers straight from docker hub outside of the CoreOS management sphere.
• We obviously would need to do this for both the AutoScaleController and AutoScaleWorker groups.

We don't actually need crewjam/ec2cluster here as all we need to set is $TAG_AWS_CLOUDFORMATION_STACK_ID and $REGION. We could set both environment variables using the feature described here: #29

There is an image available from CoreOS to replace the crewjam/awscli docker command in order to run cfn-signal: https://quay.io/repository/coreos/awscli but it could use an update (@colhom).

(Full disclosure: @gianrubio is a colleague :-) )

[iwarp]

Also big fan of this, In the UpdatePolicy you will have to include WaitOnResourceSignals=true and set the PauseTime higher in order to give time for the system to come up, otherwise cloudformation will rollback the deploy.

[mumoshu]

@gianrubio Also big fan of this!

I'd like to add that, if we're going to apply this to anything other than cloud-config-etcd, we have to workaround the known systemd issue that prevents services from restarting on startup failures when they're required via Requires.

// It'd been very tricky to handle so please let me continue with longer explanation..

Well, in such case, I believe we should at least replace Requires with Wants and optionally wait for wanted services to be up with Restart=on-failure + ExecStartPre=/usr/bin/systemctl is-active <wanted service>.

Also note that Restart=on-failure doesn't work with Type=oneshot. So in many cases we'll probably end up translating it to Type=simple, or e.g. ExecStartPre of a Type=simple unit.

You can check out dc7aafa to see how I've been doing this.

I'm saying "anything other than cloud-config-etcd" here because I believe etcd nodes don't have the startup instability issue like worker/controller thus no fixes like that applied to etcd nodes.
So you can probably apply your proposed implementation as is to cloud-config-etcd, but not to cloud-config-worker and cloud-config-controller.

See

• aws: workaround for systemd #1312 coreos/coreos-kubernetes#697
• kube-aws: decrypt-tls-assets.service failed in a controller node coreos/coreos-kubernetes#675
• and another issues referenced from these
  for related discussions.

[pieterlange]

We'll make sure to do proper testing on this taking special care of the systemd trickyness. I believe @gianrubio simply copied an excerpt from crewjam-etcd for demonstration purposes only.

This wait-for-signal PR wouldn't apply to the kube-aws embedded etcd nodes as they're:

• locked resources
• not part of an autoscalinggroup but in fact separately booted instances (fixed)

[mumoshu]

Ah, thanks for the correction!
I'd been forgetting about those two things and blindly mentioned about cloud-config-etcd.
Certainly, it won't apply to etcd nodes. Only controllers/workers which are "updatable" in the first place would be addressed in the PR.

[gianrubio]

Sorry if my first idea was not so clear, @pieterlange already detailed.

@mumoshu I think we should always use signals for cloud formation bootstrapping, it's the only way to guarantee that the cluster was successfully created/updated, we could apply this for etcd (create) and for workers/controlleres (create/update)

[mumoshu]

@gianrubio Thanks for your clarification. I totally agree to you.
The distinction between create and update is nice 👍

Let me also add that, from an user's perspective, cfn signals for updates would be more critical than for creations, because there's already an possible issue related to it which incurs downtime in certain case #50

Implementing signals for creations would also be nice to have because it will allow me to snip the wait until the k8s api server become responsible part of my automations script like this.