Excessive steady state memory allocation from authorizer in 3.7

[smarterclayton]

On v3.7.0-0.143.7 memory allocations in steady state from the cluster are mostly dominated by rbac. The current hot stop is the cluster role lister - implies there is a component that is calling into the internal version cluster role.go code more often than it should (29% of all allocations on the cluster). Historically this has been due to missing or avoiding a cache that we should have hit.

Roughly 4 billion allocations are coming from the clusterrole lister which is ~ 10% of total allocations, and 30% are coming from the child paths. Implies we aren't hitting a cache we should be hitting.

5665180875 17.44% 17.44% 5665180875 17.44%  runtime.rawstringtmp /usr/lib/golang/src/runtime/string.go
3984610750 12.27% 29.71% 3984610750 12.27%  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion.(*clusterRoleLister).Get /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion/clusterrole.go
1992612009  6.14% 35.85% 1992612009  6.14%  reflect.unsafe_New /usr/lib/golang/src/runtime/malloc.go
1864647115  5.74% 41.59% 2436501244  7.50%  runtime.mapassign /usr/lib/golang/src/runtime/hashmap.go
1859345748  5.73% 47.32% 1859345748  5.73%  runtime.makemap /usr/lib/golang/src/runtime/hashmap.go
1676510192  5.16% 52.48% 1676510192  5.16%  runtime.convT2E /usr/lib/golang/src/runtime/iface.go
1205914232  3.71% 56.19% 1205914232  3.71%  runtime.convT2I /usr/lib/golang/src/runtime/iface.go
 842609074  2.59% 58.79%  842609074  2.59%  reflect.(*structType).Field /usr/lib/golang/src/reflect/type.go
 623927158  1.92% 60.71% 5483081369 16.88%  github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*SubjectAccessEvaluator).AllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/subject_locator.go
 595830390  1.83% 62.54%  595830390  1.83%  strings.genSplit /usr/lib/golang/src/strings/strings.go
 557447565  1.72% 64.26%  557447565  1.72%  runtime.hashGrow /usr/lib/golang/src/runtime/hashmap.go
 454435068  1.40% 65.66%  454435068  1.40%  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion.(*clusterRoleBindingLister).List.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion/clusterrolebinding.go
 363047690  1.12% 66.78%  363047690  1.12%  strings.makeCutsetFunc /usr/lib/golang/src/strings/strings.go
 340411376  1.05% 67.83% 3609284145 11.11%  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion.(*Converter).doConversion /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion/converter.go

tree for clusterRoleLister

----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context
----------------------------------------------------------+-------------
                                        3984610750   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*ClusterRoleGetter).GetClusterRole /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
3984610750 12.27% 12.27% 3984610750 12.27%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion.(*clusterRoleLister).Get /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion/clusterrole.go
----------------------------------------------------------+-------------
                                          73192854   100% |   github.com/openshift/origin/pkg/authorization/authorizer/scope.(*scopeAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/scope/authorizer.go
         0     0% 12.27%   73192854  0.23%                | github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*RBACAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
----------------------------------------------------------+-------------
                                        3911412434   100% |   github.com/openshift/origin/pkg/project/auth.(*authorizerReviewer).Review /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/reviewer.go
         0     0% 12.27% 3911412434 12.04%                | github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftSubjectLocator).GetAllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
                                        3911412434   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*SubjectAccessEvaluator).AllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/subject_locator.go
----------------------------------------------------------+-------------
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union/union.go
         0     0% 12.27%   73192854  0.23%                | github.com/openshift/origin/pkg/authorization/authorizer/scope.(*scopeAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/scope/authorizer.go
                                          73192854   100% |   github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
----------------------------------------------------------+-------------
                                        3131057601 80.05% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeRoleBindings /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                         779677443 19.93% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeNamespaces /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27% 3911412434 12.04%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).(github.com/openshift/origin/pkg/project/auth.syncRequest)-fm /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                        3911412434   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).syncRequest /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).Run.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                        3911416277   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                        3911412434   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).(github.com/openshift/origin/pkg/project/auth.syncRequest)-fm /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27% 3911412434 12.04%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).syncRequest /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                        3911412434   100% |   github.com/openshift/origin/pkg/project/auth.(*authorizerReviewer).Review /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/reviewer.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).Run.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                        3131057601 80.05% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeRoleBindings /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                         779677443 19.93% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeNamespaces /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                         779677443   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27%  779677443  2.40%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeNamespaces /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                         779677443   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).(github.com/openshift/origin/pkg/project/auth.syncRequest)-fm /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                        3131057601   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27% 3131057601  9.64%                | github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).synchronizeRoleBindings /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
                                        3131057601   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).(github.com/openshift/origin/pkg/project/auth.syncRequest)-fm /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                        3911412434   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).syncRequest /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
         0     0% 12.27% 3911412434 12.04%                | github.com/openshift/origin/pkg/project/auth.(*authorizerReviewer).Review /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/reviewer.go
                                        3911412434   100% |   github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftSubjectLocator).GetAllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Forever /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
                                        3911416277   100% |   github.com/openshift/origin/pkg/project/auth.(*AuthorizationCache).Run.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/project/auth/cache.go
----------------------------------------------------------+-------------
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Forever /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
         0     0% 12.27% 3911416277 12.04%                | github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
                                        3911416277   100% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
----------------------------------------------------------+-------------
                                          19149051 95.84% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   19979401 0.062%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.(*chainAdmissionHandler).Admit <autogenerated>
                                          19979401   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.chainAdmissionHandler.Admit /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission/chain.go
----------------------------------------------------------+-------------
                                          19979401   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.(*chainAdmissionHandler).Admit <autogenerated>
         0     0% 12.27%   19979401 0.062%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.chainAdmissionHandler.Admit /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission/chain.go
                                          19979401   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.(*unionAuthzHandler).Authorize <autogenerated>
----------------------------------------------------------+-------------
                                          36815851 50.30% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go
                                          19979401 27.30% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.chainAdmissionHandler.Admit /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission/chain.go
                                          16397602 22.40% |   runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s
         0     0% 12.27%   73192854  0.23%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.(*unionAuthzHandler).Authorize <autogenerated>
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union/union.go
----------------------------------------------------------+-------------
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.(*unionAuthzHandler).Authorize <autogenerated>
         0     0% 12.27%   73192854  0.23%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union/union.go
                                          73192854   100% |   github.com/openshift/origin/pkg/authorization/authorizer/scope.(*scopeAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/scope/authorizer.go
----------------------------------------------------------+-------------
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   36815851  0.11%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          55964902   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   55964902  0.17%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.(*unionAuthzHandler).Authorize <autogenerated>
                                          19149051 34.22% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          55964902   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   55964902  0.17%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go
                                          55964902   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   36815851  0.11%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          35781556   100% |   runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s
         0     0% 12.27%   35781556  0.11%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go
                                          35781556   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   36815851  0.11%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          55964902   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   55964902  0.17%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go
                                          55964902   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
         0     0% 12.27%   36815851  0.11%                | github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go
                                          36815851   100% |   net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
----------------------------------------------------------+-------------
                                        3911412434 98.16% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*SubjectAccessEvaluator).AllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/subject_locator.go
                                          73198316  1.84% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).VisitRulesFor /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
         0     0% 12.27% 3984610750 12.27%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).GetRoleReferenceRules /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
                                        3984610750   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*ClusterRoleGetter).GetClusterRole /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
----------------------------------------------------------+-------------
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*RBACAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
         0     0% 12.27%   73198316  0.23%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).VisitRulesFor /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
                                          73198316   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).GetRoleReferenceRules /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
----------------------------------------------------------+-------------
                                        3984610750   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).GetRoleReferenceRules /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
         0     0% 12.27% 3984610750 12.27%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*ClusterRoleGetter).GetClusterRole /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
                                        3984610750   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion.(*clusterRoleLister).Get /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion/clusterrole.go
----------------------------------------------------------+-------------
                                          73192854   100% |   github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
         0     0% 12.27%   73192854  0.23%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*RBACAuthorizer).Authorize /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/rbac.go
                                          73192854   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).VisitRulesFor /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
----------------------------------------------------------+-------------
                                        3911412434   100% |   github.com/openshift/origin/pkg/authorization/authorizer.(*openshiftSubjectLocator).GetAllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/pkg/authorization/authorizer/authorizer.go
         0     0% 12.27% 3911412434 12.04%                | github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.(*SubjectAccessEvaluator).AllowedSubjects /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/subject_locator.go
                                        3911412434   100% |   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation.(*DefaultRuleResolver).GetRoleReferenceRules /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/registry/rbac/validation/rule.go
----------------------------------------------------------+-------------
                                          55964902   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go
                                          55964902   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go
                                          35781556 63.94% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go
                                          19149051 34.22% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go
         0     0% 12.27%   55964902  0.17%                | net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go
                                          55964902   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go
                                          55964902   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go
                                          55964902   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go
                                          36815851 65.78% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go
                                          19149051 34.22% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/admission.(*chainAdmissionHandler).Admit <autogenerated>
----------------------------------------------------------+-------------
         0     0% 12.27% 3964431247 12.21%                | runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s
                                        3911416277 98.66% |   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Forever /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go
                                          35781556   0.9% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1 /builddir/build/BUILD/atomic-openshift-git-0.ee33a8f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go
                                          16397602  0.41% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authorization/union.(*unionAuthzHandler).Authorize <autogenerated>
----------------------------------------------------------+-------------

[smarterclayton]

On large clusters subject access review is called frequently by external components (1/4 of total requests in some cases) so that's a possible source. Also, there are places we're making client calls instead of loopback, so we may have turned a code path that was previously fast into a slow path.

[smarterclayton]

CPU was ok, we don't have anything hot in the profiles so I'd say we're in pretty good shape there. Just the amount of garbage is causing us to fill up OS pages and get fragmentation - so our actual memory use is down, but we're using more OS memory. To get to the higher density levels we'll need to avoid this hotspot.

@openshift/team-performance

[tiran]

(*clusterRoleLister).Get() creates a temporary rbac.ClusterRole{} object to look up the actual role from the indexer cache. I have the theory Go may have decided to allocate the object on the heap instead of the stack because it is passed down to another function. This would explain the excessive amount of allocations. https://golang.org/doc/faq#stack_or_heap

func (s *clusterRoleLister) Get(name string) (*rbac.ClusterRole, error) {
	key := &rbac.ClusterRole{ObjectMeta: v1.ObjectMeta{Name: name}}
	obj, exists, err := s.indexer.Get(key)
        ...
```

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/client/listers/rbac/internalversion/clusterrole.go#L56-L67

[smarterclayton]

Yes, especially if passed in to an interface. go can only avoid heap allocations if you don't call interfaces anywhere down the stack.

[tiran]

If I understand the indexer, DeletionHandlingMetaNamespaceKeyFunc and MetaNamespaceKeyFunc correctly, the it should be possible to the temporary key object with a simple obj, exists, err := s.indexer.GetByKey(name).