Skip to content

HPA showing unkown cpu utilization #19941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kkorada opened this issue Jun 8, 2018 · 7 comments
Closed

HPA showing unkown cpu utilization #19941

kkorada opened this issue Jun 8, 2018 · 7 comments
Assignees
@sjenning
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/pod

Comments

@kkorada
Copy link

kkorada commented Jun 8, 2018 •
edited
Loading

I am unable to get cpu utilization from HPA. So as a result my pod is not getting auto scaled.

Version

v3.9

Steps To Reproduce
  1. Create a HPA for any project
  2. Try to List HPA
Current Result

C:\Users\kkorada>oc get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
lb DeploymentConfig/lb / 90% 1 10 0 1h

Expected Result

expected to see some current value

Additional Information

please find my heapster logs below
Starting Heapster with the following arguments: --source=kubernetes.summary_api:${MASTER_URL}?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --tls_cert=/heapster-certs/tls.crt --tls_key=/heapster-certs/tls.key --tls_client_ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --allowed_users=system:master-proxy --metric_resolution=30s --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-metrics-certs/tls.crt&user=hawkular&pass=$HEAPSTER_PASSWORD&filter=label(container_name:^system.slice.*|^user.slice)&concurrencyLimit=5
I0607 13:33:16.955917 1 heapster.go:72] heapster --source=kubernetes.summary_api:${MASTER_URL}?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --tls_cert=/heapster-certs/tls.crt --tls_key=/heapster-certs/tls.key --tls_client_ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt --allowed_users=system:master-proxy --metric_resolution=30s --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-metrics-certs/tls.crt&user=hawkular&pass=$HEAPSTER_PASSWORD&filter=label(container_name:^system.slice.*|^user.slice)&concurrencyLimit=5
I0607 13:33:16.955973 1 heapster.go:73] Heapster version v1.3.0
I0607 13:33:16.956264 1 configs.go:61] Using Kubernetes client with master "https://172.30.0.1:443" and version v1
I0607 13:33:16.956286 1 configs.go:62] Using kubelet port 10250
I0607 13:33:16.958417 1 driver.go:333] Initialised Hawkular Sink with parameters {_system https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-metrics-certs/tls.crt&user=hawkular&pass=qtT4osx3TQG2Cej&filter=label(container_name:^system.slice.*|^user.slice)&concurrencyLimit=5 0xc42006b4a0 5}
I0607 13:33:17.113188 1 heapster.go:196] Starting with Hawkular-Metrics Sink
I0607 13:33:17.113209 1 heapster.go:196] Starting with Metric Sink
I0607 13:33:17.127789 1 heapster.go:106] Starting heapster on port 8082
E0607 13:34:10.861236 1 client.go:203] Could not update tags: Hawkular returned status code 500, error message: Failed to perform operation due to an error: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.72.238:9042 (com.datastax.driver.core.exceptions.TransportException: [hawkular-cassandra/172.30.72.238:9042] Connection has been closed))
E0607 13:34:40.861831 1 client.go:203] Could not update tags: Put https://hawkular-metrics:443/hawkular/metrics/gauges/pod%2F5e704f17-6a57-11e8-bad3-064a71f820f2%2Fmemory%2Fworking_set/tags: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
W0607 13:46:49.231659 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 13:58:08.274885 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 14:15:53.375633 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 14:24:06.438062 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 14:39:48.553271 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 14:51:09.620549 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 15:05:34.696477 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 15:13:51.761871 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 15:29:51.888232 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 15:46:25.931221 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 15:54:03.044813 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:07:23.083153 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:15:54.170683 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:28:14.238711 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:35:46.338358 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:44:04.415185 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 16:58:52.447861 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 17:08:44.514351 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 17:25:42.607129 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 17:35:27.647258 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 17:48:19.685324 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 17:56:40.749860 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 18:07:16.836841 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 18:15:06.898832 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 18:30:01.996395 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 18:41:40.058908 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 18:49:47.105939 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 19:06:48.169654 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 19:21:54.239190 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 19:30:30.277364 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 19:41:24.384833 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 19:56:00.480791 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 20:11:48.534167 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 20:18:28.612066 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 20:29:49.635495 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 20:43:51.716404 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 21:00:41.740492 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 21:15:57.817178 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 21:28:29.853419 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 21:44:57.928577 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:00:08.966499 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:14:47.018403 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:31:41.060427 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:40:11.092227 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:48:40.151742 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 22:58:36.222307 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 23:11:38.287546 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 23:26:06.308557 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 23:35:57.395275 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0607 23:50:21.510789 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 00:00:12.613143 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 00:14:02.677296 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 00:32:02.798561 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 00:39:40.891163 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 00:49:39.989320 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 01:02:43.064665 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 01:11:00.156907 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 01:20:19.269757 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 01:28:46.307655 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 01:46:40.337167 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:00:23.457051 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:12:58.551847 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:19:17.583857 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:30:02.660599 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:44:59.720190 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 02:53:31.825307 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 03:05:46.854946 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 03:14:24.902870 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 03:28:01.014905 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 03:37:21.050710 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 03:45:40.148038 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion for the provided watch is too old.
W0608 04:01:12.208229 1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: watch of *v1.Namespace ended with: The resourceVersion fo
@kkorada
Copy link
Author

kkorada commented Jun 8, 2018

i am getting below warning in the web console
image

@jwforres
Copy link
Member

jwforres commented Jul 3, 2018

@openshift/sig-pod

the web console warning sounds similar to the known issues we had with the HPAs the web console was creating @DirectXMan12 is this the same issue?

Related: #18517 (comment)

@sjenning sjenning removed their assignment Jul 5, 2018
@sjenning
Copy link
Contributor

sjenning commented Jul 5, 2018

possibly related
#19045
#19437

@sjenning sjenning self-assigned this Jul 5, 2018
@openshift-bot
Copy link
Contributor

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 3, 2018
@openshift-bot
Copy link
Contributor

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 2, 2018
@openshift-bot
Copy link
Contributor

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci-robot
Copy link

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sjenning sjenning

Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/pod
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@DirectXMan12 @sjenning @jwforres @openshift-bot @kkorada @openshift-ci-robot