add horizontalpodautoscaler

[coolbry95]

Proposed changes

Adds the horizontalpodautoscaler for the ingress controller

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[coolbry95]

Broke the last branch because of merges. I will rebase my branch back on to master. I squash the commits when doing the rebasing.

[shaun-nx]

I'm including a link to the old (closed) PR of this so we don't lose the discussions: #3249

[shaun-nx]

@coolbry95 Can you follow-up this discussion thread? #3249 (comment)

[brianehlert]

I want everyone to be mindful here.
I see that HPA will be removing a version with 1.26, however that will pull in the supported version of K8s to 1.23 https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#removal-of-the-v2beta2-horizontalpodautoscaler-api

Which I think is too close to the current release of K8s for many of our customers.

Pulling in the current supported K8s version to 1.21 is as close as I want to go.
And that is already happening due to EndpointSlices.

[lucacome]

@brianehlert since autoscaling/v2 is available starting from 1.23 I think it makes sense to use that instead of autoscaling/v2beta2, which will be deprecated when 1.26 comes out in a couple of weeks. This just means that anybody on versions < 1.23 won't be able to use HPA.

[brianehlert]

@brianehlert since autoscaling/v2 is available starting from 1.23 I think it makes sense to use that instead of autoscaling/v2beta2, which will be deprecated when 1.26 comes out in a couple of weeks. This just means that anybody on versions < 1.23 won't be able to use HPA.

As long as that is somehow non-breaking for customers that might install the next version of NIC on 1.22 or 1.21
I don't want to bring the oldest supported version to 1.23 simply for the convenience of having HPA in the Helm chart. That is my point.

[coolbry95]

This won't pull the minimum version to 1.23. The HPA will only be available for clusters in 1.23 and above.

[shaun-nx]

Confirmed that the HPA resource doesn't get install on a server version less than 1.23 👍
The feature is behaving as expected on k8s server >= 1.23 as well. Looks all good to me, nice work!

[coolbry95]

Fixes #3174