Skip to content

Optimize PrometheusCollector scrape efficiency. #2974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 24 commits into from
Closed

Optimize PrometheusCollector scrape efficiency. #2974

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6d0fb84
Added baseline benchmark for PrometheusCollector_Collect.
bwplotka Oct 22, 2021
3516134
Optimization attempt 1.
bwplotka Oct 23, 2021
44415ce
Optimization attempt 2.
bwplotka Oct 24, 2021
71765e0
Finalized new client golang abstraction and using it.
bwplotka Oct 24, 2021
2d3c554
Optimized the cached part.
bwplotka Oct 24, 2021
c40fe8c
Ultimate slice reuse.
bwplotka Oct 24, 2021
ab75edd
Updated to newer version of client_golang.
bwplotka Jan 26, 2022
df0736a
Fixed handler.
bwplotka Jan 27, 2022
b98e3bb
Added forgotten update.
bwplotka Feb 17, 2022
bcb7e26
Moved cached here from client_golang for tied integration.
bwplotka Feb 23, 2022
8de935d
Embedded cache, fixes and optimizations.
bwplotka Feb 23, 2022
66021d8
Optimized.
bwplotka Feb 23, 2022
d5a22e0
Optimization.
bwplotka Feb 23, 2022
51fa483
fix.
bwplotka Feb 23, 2022
c622498
tmp.
bwplotka Feb 23, 2022
d5e8761
Most optimized version.
bwplotka Feb 23, 2022
20dc825
Adjusted for new type.
bwplotka Feb 23, 2022
ecaea89
Adjusted tests.
bwplotka Feb 23, 2022
a880892
Fixed sort of expected thing (was wrong) and data errors.
bwplotka Feb 23, 2022
c6ed5c1
Simplified.
bwplotka Feb 23, 2022
de13093
Fixes.
bwplotka Feb 24, 2022
31697c9
cleaner syntax.
bwplotka Feb 24, 2022
a587512
Fix deps & uncomment cache tests
saswatamcode Mar 9, 2022
b2bec41
Add file boilerplate
saswatamcode Mar 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions cmd/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,23 @@ require (
github.com/Rican7/retry v0.1.1-0.20160712041035-272ad122d6e5
github.com/SeanDolphin/bqschema v0.0.0-20150424181127-f92a08f515e1
github.com/Shopify/sarama v1.19.0
github.com/Shopify/toxiproxy v2.1.4+incompatible // indirect
github.com/abbot/go-http-auth v0.0.0-20140618235127-c0ef4539dfab
github.com/eapache/go-resiliency v1.1.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/garyburd/redigo v0.0.0-20150301180006-535138d7bcd7
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect
github.com/influxdb/influxdb v0.9.6-0.20151125225445-9eab56311373
github.com/mesos/mesos-go v0.0.7-0.20180413204204-29de6ff97b48
github.com/onsi/ginkgo v1.7.0 // indirect
github.com/onsi/gomega v1.4.3 // indirect
github.com/pierrec/lz4 v2.0.5+incompatible // indirect
github.com/pquerna/ffjson v0.0.0-20171002144729-d49c2bc1aa13 // indirect
github.com/prometheus/client_golang v1.8.0
github.com/prometheus/client_golang v1.12.2-0.20220223112252-1f81b3e9130f
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a // indirect
github.com/stretchr/testify v1.7.0
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c
google.golang.org/api v0.34.0
gopkg.in/olivere/elastic.v2 v2.0.12
k8s.io/klog/v2 v2.4.0
Expand Down
231 changes: 31 additions & 200 deletions cmd/go.sum
View file
Open in desktop

Large diffs are not rendered by default.

43 changes: 28 additions & 15 deletions cmd/internal/http/handlers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,14 @@ import (
"github.com/google/cadvisor/cmd/internal/pages"
"github.com/google/cadvisor/cmd/internal/pages/static"
"github.com/google/cadvisor/container"
v2 "github.com/google/cadvisor/info/v2"
"github.com/google/cadvisor/manager"
"github.com/google/cadvisor/metrics"
"github.com/google/cadvisor/validate"

auth "github.com/abbot/go-http-auth"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/prometheus/client_golang/prometheus/promhttp"
"k8s.io/klog/v2"
"k8s.io/utils/clock"
Expand Down Expand Up @@ -92,31 +94,42 @@ func RegisterHandlers(mux httpmux.Mux, containerManager manager.Manager, httpAut
return nil
}

// RegisterPrometheusHandler creates a new PrometheusCollector and configures
// RegisterPrometheusHandler creates a new ContainerCollector and configures
// the provided HTTP mux to handle the given Prometheus endpoint.
func RegisterPrometheusHandler(mux httpmux.Mux, resourceManager manager.Manager, prometheusEndpoint string,
f metrics.ContainerLabelsFunc, includedMetrics container.MetricSet) {
goCollector := prometheus.NewGoCollector()
processCollector := prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{})
machineCollector := metrics.NewPrometheusMachineCollector(resourceManager, includedMetrics)

r := prometheus.NewRegistry()
r.MustRegister(
collectors.NewGoCollector(),
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
)
container := metrics.NewContainerCollector(resourceManager, f, includedMetrics, clock.RealClock{})
machine := metrics.NewMachineCollector(resourceManager, includedMetrics)

nameTypeCache := metrics.NewCachedGatherer(container, machine)
nameDockerCache := metrics.NewCachedGatherer(container, machine)
nameTypeGatherer := prometheus.NewMultiTRegistry(prometheus.ToTransactionalGatherer(r), nameTypeCache)
nameDockerGatherer := prometheus.NewMultiTRegistry(prometheus.ToTransactionalGatherer(r), nameDockerCache)

mux.Handle(prometheusEndpoint, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
// TODO(bwplotka): Why we ask for full object if we override half of fields?
opts, err := api.GetRequestOptions(req)
if err != nil {
http.Error(w, "No metrics gathered, last error:\n\n"+err.Error(), http.StatusInternalServerError)
return
}
opts.Count = 1 // we only want the latest datapoint
opts.Recursive = true // get all child containers

r := prometheus.NewRegistry()
r.MustRegister(
metrics.NewPrometheusCollector(resourceManager, f, includedMetrics, clock.RealClock{}, opts),
machineCollector,
goCollector,
processCollector,
)
promhttp.HandlerFor(r, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}).ServeHTTP(w, req)
opts.Count = 1 // We only want the latest datapoint.
opts.Recursive = true // Get all child containers.

// Present different metrics depending on option.
if opts.IdType == v2.TypeDocker {
nameDockerCache.UpdateOnMaxAge(opts)
promhttp.HandlerForTransactional(nameDockerGatherer, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}).ServeHTTP(w, req)
return
}
nameTypeCache.UpdateOnMaxAge(opts)
promhttp.HandlerForTransactional(nameTypeGatherer, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}).ServeHTTP(w, req)
}))
}

Expand Down
2 changes: 1 addition & 1 deletion container/libcontainer/handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func NewHandler(cgroupManager cgroups.Manager, rootFs string, pid int, includedM
}
}

// Get cgroup and networking stats of the specified container
// GetStats gets cgroup and networking stats of the specified container.
func (h *Handler) GetStats() (*info.ContainerStats, error) {
ignoreStatsError := false
if cgroups.IsCgroup2UnifiedMode() {
Expand Down
13 changes: 8 additions & 5 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@ module github.com/google/cadvisor
go 1.16

require (
cloud.google.com/go v0.54.0
cloud.google.com/go v0.65.0
github.com/Microsoft/go-winio v0.4.15
github.com/aws/aws-sdk-go v1.35.24
github.com/blang/semver v3.5.1+incompatible
github.com/cespare/xxhash/v2 v2.1.2
github.com/containerd/containerd v1.4.12
github.com/containerd/ttrpc v1.0.2
github.com/containerd/typeurl v1.0.2
Expand All @@ -16,6 +17,7 @@ require (
github.com/docker/go-units v0.4.0
github.com/euank/go-kmsg-parser v2.0.0+incompatible
github.com/gogo/protobuf v1.3.2
github.com/golang/protobuf v1.5.2
github.com/gorilla/mux v1.8.0 // indirect
github.com/karrick/godirwalk v1.16.1
github.com/mindprince/gonvml v0.0.0-20190828220739-9ebdce4bb989
Expand All @@ -28,13 +30,14 @@ require (
github.com/opencontainers/runc v1.1.0
github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.7.1
github.com/prometheus/client_golang v1.12.2-0.20220223112252-1f81b3e9130f
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.10.0
github.com/prometheus/common v0.32.1
github.com/stretchr/testify v1.7.0
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
golang.org/x/sys v0.0.0-20211116061358-0a5406a5449c
golang.org/x/net v0.0.0-20210525063256-abc453219eb5
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9
google.golang.org/grpc v1.33.2
google.golang.org/protobuf v1.27.1
k8s.io/klog/v2 v2.4.0
k8s.io/utils v0.0.0-20211116205334-6203023598ed
)
Loading