keda-operator throttling on the Kubernetes API server

[ingmars1709]

In the logs of the keda-operator we see PATCH calls throttled to the Kubernetes API server on the ScaledObjects.
We have around 400 ScaledObjects divided between 17 namespaces.

Waited for 1.320310071s due to client-side throttling, not priority and fairness,
request: PATCH:https://10.24.0.1:443/apis/keda.sh/v1alpha1/namespaces/mynamespace/scaledobjects/scaled-service-foo/status

It looks like we are being throttled on our Kubernetes API server.

Are there any settings on the Keda chart that can tweak the amount of /status PATCH calls done through the K8s API-server?

We are using the Keda Chart version 2.17.0

[alvidofaisal]

I've been looking into this issue.

Based on my review of the codebase and the issue description, here are a couple of thoughts and a contribution I've made:

• Operator Configuration: It seems the KEDA operator has flags like --kube-api-qps, --kube-api-burst, and --keda-scaledobject-ctrl-max-reconciles. For users experiencing throttling with many ScaledObjects (like the ~400 mentioned), adjusting these might offer some relief by allowing the operator to handle more requests or process more ScaledObjects concurrently. It would be great if you could experiment with these settings. For example, values like --kube-api-qps=40, --kube-api-burst=60 could be a starting point. (I've put together some more detailed notes on this in a local markdown file, keda_api_throttling.md, which basically summarizes this point and how to apply them via deployment args).

• Code Contribution - Reducing Status Updates: I noticed that the ScaledObject controller updates its status conditions, which involves a PATCH call. To help reduce the number of these calls, especially when the object's state isn't changing, I've submitted a pull request (Optimize scaledobject status updates #6817) that makes these status updates more selective. It now checks if the conditions have actually changed before making the PATCH call. This should hopefully reduce some load on the API server.

I hope these suggestions and the PR are helpful in addressing the reported throttling.