Skip subcontainer update on v2 calls #1722
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Nice dig |
sjenning
force-pushed
the
skip-subcontainer-update-v2
branch
from
223530f to
3ba4699
Compare
|
cc @tallclair |
|
That "Close and comment" button should be red! 😖 |
|
@dashpole @tallclair if I can get a review on this in short order, I'll update my kube bump PR kubernetes/kubernetes#50629 to include this change. |
|
ill take a look today later today. |
|
LGTM, will let @dashpole review before merging. Awesome investigation. Seeing the profile before and after is really helpful. |
This was referenced Aug 19, 2017
openshift-merge-robot
added a commit
to openshift/origin
that referenced
this pull request
Aug 22, 2017
Automatic merge from submit-queue (batch tested with PRs 15861, 15865, 15873, 15848, 15891) UPSTREAM: google/cadvisor: 1722: Skip subcontainer update on v2 calls google/cadvisor#1722 This change results in a significant CPU reduction on the node. @derekwaynecarr @eparis @smarterclayton
openshift-merge-robot
added a commit
to openshift/origin
that referenced
this pull request
Aug 24, 2017
Automatic merge from submit-queue UPSTREAM: google/cadvisor: 1722: Skip subcontainer update on v2 calls 3.6 pick of #15873 xref google/cadvisor#1722 @derekwaynecarr @eparis @smarterclayton
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently
GetInfo()updates both the spec and the subcontainers. The subcontainer walk is VERY expensive as it does a stat syscall on basically every directory and file in the cgroupfs tree viaioutil.Readdir(). This uses lots of CPU directly and also allocates a TON of objects that have to be GCed later, consuming even more CPU.AFAICT, the v2 api does not need to do this subcontainer update at all since the v2 ContainerInfo doesn't have a subcontainers field:
https://github.com/sjenning/cadvisor/blob/master/info/v1/container.go#L128-L139
https://github.com/sjenning/cadvisor/blob/master/info/v2/container.go#L59-L65
The code supports this conclusion as v2 go paths look like this
The
Subcontainersin thecontainerInfois discarded.This is a huge performance win. In Kubernetes,
GetContainerInfoV2()is called every 10s from the summaryProvider. It is currently one of the top ~5 hot paths in steady-state kubelet.@derekwaynecarr @smarterclayton @dashpole @vishh @tallclair