[Proposal] Supporting active-standby applications in Kubernetes

[XiaoningDing]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

This is a feature request.

What you expected to happen:

Propose to have a built-in support for active-standby applications in Kubernetes.

Active-Standby applications are applications that have multiple running instances, but only one instance is active and serving workloads at one time. All other instances are idle (standby). If the active instance is down, one of the standby instances will be promoted to active immediately.

Lots of enterprise legacy applications are such applications. In fact, Kubernetes scheduler and controller manager are also examples. They run in active-standby mode in a HA deployment of Kubernetes.

These applications have similar requirements in their development and deployment, like:

• Role assignment: one of the instances should be active and all others be standby. Each instance needs to be aware of its current role.
• Health monitoring and role switching: when the active instance is down, one of the standby instances needs to be promoted to active. When a standby instance is down, a new standby instance should be added.
• Traffic routing: a single service is exposed, but traffic should always be routed to the current active instance.
• Controlled update: when update such a collection of pod instances, the standby instances should be updated before the active instance is updated.

These requirements cannot be handled by current StatefulSet object. Statefulset object is a good abstraction for clustered stateful applications, however it seems there are many differences between clustered model and active-standby model. In the former model instances are aware of each other and have their own storage, but in the latter model instances are not aware of each other and states need to be synced.

Now developers and operators need to implement above mentioned requirements again and again for all such active-standby applications. I'm wondering if we can make active-standby model a native supported model in Kubernetes so applications can focus on their own business logic.

[philips]

I think in these cases you should have an external health mechanism from Kubernetes and then a tool that adds/removes labels from the active/passive pods to update the service.

I would suggest that either the external health mechanism either use etcd for the leader election or use locking on a ConfigMap or something.

[smarterclayton]

We talked about this in the early stages of StatefulSet design. I do think that @bprashanth and I wanted to see something like this exist.

[smarterclayton]

@kubernetes/sig-apps-feature-requests

[XiaoningDing]

My thought is to introduce a system label like alpha.kubernetes.io/role-active to mark which instance is the current active role. Only one instance will have this label at any given time. This label can be assigned by a controller or elected through other ways. And Kubernetes will update the associated service to include this additional label selector, so traffic routing will be done automatically and transparently.

I tried this approach and it works fine. After role switching happens, traffic will be redirected to the new active instance immediately.

But there is a problem. Application pod needs to be aware of its current role. My initial thought is to use Downward API. So pods can mount metadata.labels as a file ( or an environment variable ) and watch this file to know its current role. But during my test, I found there is a several-minute latency between labels get updated and file contents get refreshed. This latency is not acceptable to our products. I haven’t dug into Downward API volume codes to see if it’s an implementation issue, by design, or due to some cache inside the container.

Do you guys have good ideas on how to let application be aware of its current role with a low latency? Not sure if it’s worth introducing a new pod lifecycle hook.

@smarterclayton @philips @bprashanth

[timothysc]

Side-car with leader-election code is the simplest answer.

I've done it a number of times, or you could startup wrap as well if you don't want a side-car. Building it into the primitives is thorny b/c there is no 1st class api object... (cough cough, snipe snipe)

/cc @mikedanese

[XiaoningDing]

@timothysc Side-car is only one option for lead election. Some applications probably will prefer a more lightweight approach, like a central controller assigning roles. Also there are other requirements like traffic routing and controller update.

Considering active-standby is a common application model, probably it's worthwhile to build a new first-class api object (standbySet?) to cover all these requirements?

[timothysc]

There would need to re-dig-up an api-object for locking, and add the capability to other api-objects.

/cc @davidopp @rrati

[davidopp]

I don't mean to throw a wet blanket on this proposal, and it's definitely not my decision, but this seems outside the scope of Kubernetes to me.

[bgrant0607]

I agree with @davidopp. Build an operator:

https://coreos.com/blog/introducing-operators.html

Use ThirdPartyResource for locking, if necessary.

[XiaoningDing]

I will check with our internal customers to see if an operator can fully meet their requirements. Will update this later.