Allow workers to sit inside private subnets within an existing VPC

[cknowles]

Use case proposal:

• Use an existing VPC, e.g. has an existing RDS deployment inside
• Private subnets used for at least workers, perhaps controllers too if we use a bastion (first use case we want public controllers so we can SSH into the cluster)
• Use existing NAT Gateway per AZ for HA with the appropriate route tables used per worker AZ

Similar to kubernetes/kops#428.

It was part of what coreos/coreos-kubernetes#716 was intended to cover. Pre-filing this issue so we can move discussion out of #35.

[cknowles]

@pieterlange @mumoshu the original pull request does two things:

1. Allows existing route tables to be associated with kube-aws created subnets
2. Separates out subnets of the controller(s) so their route tables can be different

The reason using the existing routeTableId is less useable for this case is because unless we are very careful with the route table we defeat the HA aspects as we need to route traffic on very specific CIDRs to the right AZs.

[cknowles]

I have a feeling we should separate out the various use cases as I've kind of mixed them together since the change was relatively small to cope with all of this.

[pieterlange]

The reason using the existing routeTableId is less useable for this case is because unless we are very careful with the route table we defeat the HA aspects as we need to route traffic on very specific CIDRs to the right AZs.

Are you referring to the route to 0.0.0.0/0 via NAT gateway here? As the NAT gateway can only live in on AZ at a time, there is an inherent dependency here. There is more than one way to skin this cat.

• Maintain multiple route tables, one per AZ (your suggestion)
• Keep using one routetable and manually / lamba failover the NAT gateway to a different NAT gateway on AZ downtime

I believe that ultimately this problem can be solved without constraining operator choices by implementing node pools. The controllers would be in a different pool. Workers in each AZ could be in different pools, one pool per AZ. Or they can be in one pool and use one routetable.

[cknowles]

Yeah, our private subnet route tables look something like:

10.0.0.0/16 -> local
0.0.0.0/0 - > nat-UNIQUEID

Having looked at it again just now I think there is no way to switch traffic based on the source CIDR in a single route table. So in other words, we can't use a single route table while still routing the worker traffic from each individual AZ to the appropriate NAT Gateway in the same AZ.

I think generally we'd want to have a route table per AZ to mirror a typical multi-AZ NAT Gateway setup.

The original pull was a simple version on the basis of a whole AZ going down, however if we are talking about service level outages per AZ then if a single NAT Gateway is down I'm not sure of how the failover could/should work. Probably something along the lines of what you mention with lambda/manual.

[pieterlange]

if a single NAT Gateway is down

The NAT gateways themselves are HA (or so amazon says). The issue currently is if the gateway lives in AZ-a and that entire AZ goes down, AZ-b and AZ-c lose connectivity to the internet until a new NAT gateway is booted in one of those AZ's and the route table is updated.

I think generally we'd want to have a route table per AZ to mirror a typical multi-AZ NAT Gateway setup.

👍

[cknowles]

Yep agreed. From the docs:
NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture.

Do you have more details on the node pools? I'd like to have a look over it.

[pieterlange]

@mumoshu wrote something about this in #46 just now :)

[mumoshu]

@c-knowles I've announced my POC for node pools in #46 (comment) yesterday!

Could it be something that can be a foundation to address this issue?

With the poc, we now have separate subnet(s) for each node pool(=a set of worker nodes).
Is that enough or do we need ways to differentiate route table per subnet/AZ as you've discussed so far?

P.S. I don't recommend reading each commit in the node-pools branch for now, mainly because those are really dirty and not ready for reviews 😆

[cknowles]

Cool, thanks. I was going to review it soon for this use case. Prior to that, what I would say is the current setup we are using from code at coreos/coreos-kubernetes#716 uses a different subnet per AZ because each AZ has it's own NAT Gateway. I think control over different AZs within a tool like kube-aws will usually come down wanting control over what gets set per AZs (with sensible defaults).

[cknowles]

@mumoshu After reviewing node pools further, I think this use case should be a matter of:

• For workers, specify the appropriate routeTableId in each AZ via a node pool for each AZ which allows us to then use the autoscaler as per Feature: Node pools #46.
• For controllers, specify the appropriate routeTableId depending on whether public/private controllers are needed.

I think that's it. I'm going to try it out tomorrow. My only reservation would be that it's a quite complex setup for what I think is a very common scenario. I also could not see a way to disable the new separate etcd node creation, I don't really want/need one in the dev cluster but I guess that's not supported any more?

[mumoshu]

@c-knowles Really looking forward to your results!

Regarding the separate etcd nodes: Yes, collocating etcd and control plane in a single node is not supported.

I also believe that the uniform architectures for the dev and the prod envs would be good in several points including e.g. dev-prod parity, less user confusion/support, less code complexity.
I'm a bit afraid of being obtrusive here but basically,I recommend adapting to etcd node separation, regardless of dev/prod.

If you're suffering from cost, I guess you can use smaller instance types for etcd nodes in a dev env and then wasted cost would be minimum.

[cknowles]

I tend to agree to a certain level. Our dev cluster is where we development cluster changes/upgrades and we run automated tests against it for our apps and deployment mechanisms. We may even have a few of those at any time. We actually have a staging cluster after dev as well. The cost currently went up 50% minimum for any of those clusters (min was 2 nodes, now it's 3). I'd like to support external etcd as well such as compose.io.

Updates/feedback so far:

• The model above only really works if we can create the initial stack with zero workers. We want the initial stack to be only controllers with a chosen route table. Currently not possible due to hard coded min/max as count-1 and count+1:
  CREATE_FAILED AWS::AutoScaling::AutoScalingGroup AutoScaleWorker Min bound, -1, must be non-negative Allow more control over ASG definition #142
• When creating a node-pool, it may be nice to also copy across the values for worker config like workerInstanceType, workerRootVolumeSize etc from the main cluster.yaml. stackTags also. Easier to configure, easier to version-control, more tightly integrated node pools #315
• Node pool default cluster.yaml differences from main cluster.yaml - useCalico is commented out for main but explicit in node pool (my preference is go with all explicit defaults in the cluster.yaml and remove defaults from the go code). Same with amiId, node pool default has empty YAML property. Easier to configure, easier to version-control, more tightly integrated node pools #315
• Given the auto scaler can only work in a single AZ, should we allow node pools to be multi AZ? Currently the node pool cluster.yaml contains subnets commented out (I didn't touch it though). Setup would be more complex for some use cases but it could cut what we need to support if we provide standard examples in the docs. Easier to configure, easier to version-control, more tightly integrated node pools #315 covers making node pools generally easier to configure, multi-AZ node pools are still allowed but perhaps there are uses cases for that too.
• Some inconsistency in command naming. kube-aws says node-poolsas does init output but render output says nodepool. It also mentions --pool-name instead of --node-pool-name and actually we cannot validate yet? Correct node pool command inconsistencies #174
• How are we going to manage upgrades to the main stack with the node pools in place? I'm thinking about if I have to go and rename the stacks each time or whether we can prefix the node pool stack names with the main cluster name when we render/up. Node Pool Nested Stack Name #187
• How to update node pool stacks? The Stack Policy is set to {} so updates are currently disallowed. Make kube-aws node-pools update not to fail #140
• Definitely need more instructions for node pools! Happy to contribute to this based on my usage. Documentation for v0.9.2 #162
• Prefix node pool subnets with node pool name - Name the node pool subnets after the node pool #129