[openstack-cloud-controller-manager] Shared tenancy OpenStack project service type LoadBalancer collision

[scrothers]

/kind bug

What happened:
Two Kubernetes clusters in the same project were created. The following steps were taken along with their timeline.

• Two different Kubernetes clusters were created using RKE2 in the same OpenStack project.
• Both Kubernetes clusters came online and were fully functional.
• Kubernetes cluster A had the default:kubernetes service changed from ClusterIP to LoadBalancer.
• A load balancer was created and the Kubernetes service was made public successfully.
• Kubernetes cluster B had the default:kubernetes service changed from ClusterIP to LoadBalancer.
• OCCM identified the first load balancer in Octavia as it's own because the name was kube_service_kubernetes_default_kubernetes and rebuilt it to point to Kubernetes cluster B.
• Kubernetes cluster A is now desynchronized and has a LoadBalancer entry which no longer exists.

What you expected to happen:
Ideally, I would like to see the load balancer be created with a random suffix in OpenStack (IE: kube_service_kubernetes_default_kubernetes_vxwmx). We use suffixes in other places in Kubernetes, makes sense to use them here too. Further, if a load balancer collision were to exist, instead of deleting/recreating a load balancer of the same name, perhaps an error event should be fired instead if the suffix option is unavailable?

Also, if the occm is going to delete an existing load balancer, there should be a more obvious log entry for this. Currently it just transitions from ACTIVE to PENDING_CREATE in the logs. It's very ambiguous and should have more detailed output.

How to reproduce it:

• Create two Kubernetes clusters in OpenStack in the same project, change the default:kubernetes service on each one from ClusterIP to LoadBalancer with a 5 minute gap between the first cluster being fully provisioned.
• Check the Octavia API, only one load balancer will exist pointed to the second cluster.

Anything else we need to know?:

• Kubernetes cluster A and Kubernetes cluster B have two different Octavia UUIDs, so the LoadBalancer was deleted/created, not simply modified.
• Container image: docker.io/k8scloudprovider/openstack-cloud-controller-manager:latest

Environment:

• openstack-cloud-controller-manager version: latest
• OpenStack version: Zed
• Others:

[jichenjc]

Ideally, I would like to see the load balancer be created with a random suffix in OpenStack (IE: kube_service_kubernetes_default_kubernetes_vxwmx). We use suffixes in other places in Kubernetes, makes sense to use them here too. Further,

make sense and I think we should tolerate multiple cluster creation, create a suffix also make sense

if a load balancer collision were to exist, instead of deleting/recreating a load balancer of the same name, perhaps an error event should be fired instead if the suffix option is unavailable?

you mean, we should set a option to enable suffix or not ? this might be complicated ..

@dulek @zetaab FYI

[dulek]

@scrothers: I think our assumption was to use clusterName instead of a random prefix/suffix to distinguish LBs from different clusters. See¹.

This would mean both of your clusters are named "kubernetes" and that's the root of the issue.

The problem is I just checked some OpenShift test runs and for some reason clusterName is "kubernetes" there too. So I'm not sure how to set that clusterName and if it's even possible.

Footnotes

1. https://github.com/kubernetes/cloud-provider-openstack/blob/bd767ba5d4493d2aa7965d57286f1e6460e9dba9/pkg/openstack/loadbalancer.go#L599 ↩

[zetaab]

the problem is that quite many cluster still uses default clusterName which is kubernetes. It is also difficult to migrate away from it.

[dulek]

the problem is that quite many cluster still uses default clusterName which is kubernetes. It is also difficult to migrate away from it.

Okay. And also problem will happen only if you have two services named the same in your clusters, so some folks might be unaware of that. I would think that putting some cluster identifier into the tags could help too. And that is an additive change, so fairly easy to implement for the clusters already running too.

@zetaab: Do you know where do I set that clusterName thing?

[zetaab]

Afaik its startup flag for the occm binary

[jichenjc]

https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml#L45

https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml#L66

I think @zetaab is right, seems we should ask for input instead of default one ?
or create a suffix and overtime we can handle this kind of issue ?

[zetaab]

I think we should add warning at least if using default value kubernetes

[armagankaratosun]

Hello all,

I also had the same issue, using RKE2 and Openstack CCM within the same project that I have in Openstack. I think the issue here is with the rancher-provided charts, especially rke2-ingress-nginx because if you don't overwrite the name, it will always use the same values for the resources that it creates, ending up with the same service names in all clusters.

rke2-ingress-nginx-controller
rke2-ingress-nginx-admission
rke2-ingress-defaultbackend
.
.
.

And as far as I understood from the behavior of CCM, it uses the notation kube_service_<clusterName>_<serviceName> which in this case will end up with a load balancer (assuming that you are exposing your ingress via an Octavia load balancer);

kube_service_kubernetes_kube-system_rke2-ingress-nginx-controller

and if you have two clusters in the same project, and you want to do the same thing on both of them, then the load balancers will overwrite each other, because they both have the same service name, triggering the reported bug.

One workaround might be overriding the name of the chart, hence changing the service name (serviceName) instead of the cluster (clusterName), because it is simpler, and can be done at any given time.

rke2-ingress-nginx:
    nameOverride: my-different-cluster-ope
    controller:
#     ## Set dnsPolicy to ClusterFirstWithHostNet if you want to set hostNetwork: true and hostPort.enabled: true
      dnsPolicy: ClusterFirstWithHostNet 
      hostNetwork: false
#     ## Use host ports 80 and 443
      hostPort:
        enabled: false
#       ports:
#         http: 80
#         https: 443

which will create (a service in Kubernetes and) a load balancer named kube_service_kubernetes_kube-system_rke2-ingress-nginx-my-different-cluster-ope-controller in Openstack, which is unique.

You might argue, "But Armagan, what about the other services that I have with type: Loadbalancer?". You are right, but you can always set your service names unique, to avoid overwriting your load balancers in Openstack, for example;

apiVersion: v1
kind: Service
metadata:
  name: unique-service-name
spec:
  selector:
    active 'true'
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  sessionAffinity: None
  type: LoadBalancer

will create a load balancer named kube_service_kubernetes_unique-service-name in Openstack. I know this is not a fix, but a workaround, but it lets things rolling at least.

TL, DR; Just make sure that your service names are unique and you will be fine. Hope it helps!

[scrothers]

TL, DR; Just make sure that your service names are unique and you will be fine. Hope it helps!

@armagankaratosun, absolutely the worst piece of advice. In a world where upstream deploys via helm, and where not all helm charts have that ability, your solution is entirely unfeasible.

Further, if you install something like RKE2 as you imply, greater than 2 times in a single OpenStack project, by default it'll be broken until you remediate by hand afterwards.

I don't think you understand the point of Kubernetes if you believe that manual remediation is an appropriate suggestion to even add to the discussion.

[armagankaratosun]

I don't think you understand the point of Kubernetes if you believe that manual remediation is an appropriate suggestion to even add to the discussion

@scrothers Let's take a step back before judging my understanding of Kubernetes shall we?

Fist of all, where did you read that I suggest manual remediation? As far as I can see, you are jumping into conclusions.

@armagankaratosun, absolutely the worst piece of advice. In a world where upstream deploys via helm, and where not all helm charts have that ability, your solution is entirely unfeasible.

You say that in a world we deploy everything via helm etc, how come I suggest something that not all of the helm charts have. But the point is, both nameOverride and fullnameOverride is something you have almost in all helm charts ( if not all of them, it is in the template after all). So what I suggest is applicable for almost all charts that you can deploy, therefore, it is feasible.

Further, if you install something like RKE2 as you imply, greater than 2 times in a single OpenStack project, by default it'll be broken until you remediate by hand afterwards.
.

Again, you are jumping into conclusions. Yes, I do have in fact more than 2 clusters in a single Openstack project. I do not make any manual changes and the clusters are running just fine. You know how? Via changing the chart names with nameOverride like I suggest initially.

Ok this is more about Rancher than OCCM now but you know the rke2 provided add-ons are helm charts and you can overwrite the chart values for those charts via providing yaml manifests, right? And you can customize this before you deploy your cluster, therefore, you will not end up with broken clusters that you need to remediate by hand.

So again, what I am suggesting here is very straightforward. Just ensure that you will have unique service names so that OCCM can create the load balancers without any issue. And I do this via customizing the nameOverride value in the charts (especially the rke2 charts), giving them unique names, adding suffixes like you initially suggested, but not as part of OCCM. Of course it might be better if OCCM can handle this without any intervention, but hey, this is life.

I also said that this is not a solution but a workaround. I hope I explained myself more clearly this time.

[dulek]

This is still a very valid issue though, we cannot expect users to differentiate service names, user might not even know that there are multiple Kubernetes instances running in the same OpenStack project.

The fact that fixing it is non-trivial is a different story.

[jichenjc]

do we know any other Cloud provider such as AWS handle such situation?

even though there are migration issue I Think we can focus on new clusters

how about we add clustername and suffix both so even someone didn't pay attention to the cluster name but we still handle this in a reasonable way?

[zetaab]

AWS uses tags in loadbalancers, so in case of openstack we could use metadata/tags to separate these with clustername.

[jichenjc]

so in case of openstack we could use metadata/tags to separate these with clustername.

but this still need user input .. as we need distinguish multiple k8s in one openstack.. so maybe go with suffix as this makes cluster name trasparent?