test flake: TestImageStreamList fails with panic in healthz route handler

[miminar]

There's a panic in the healthz route handler. But I'm still unsure if it's a real cause or not:

(shortened)

failed  TestImageStreamList
=== RUN   TestImageStreamList
I0724 08:47:31.451851    5109 wrap.go:42] GET /healthz: (587.717µs) 500
goroutine 4921 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42dae9180, 0x1f4)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42dae9180, 0x1f4)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42db18c40, 0x1f4)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:185 +0xb5
net/http.Error(0x7ff2489ebee0, 0xc42705ad50, 0xc425b9d440, 0x22a, 0x1f4)
	/usr/local/go/src/net/http/server.go:1954 +0xda
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:110 +0x3cb
net/http.HandlerFunc.ServeHTTP(0xc42333d300, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc422cdb340, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x567
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc420a7ee70, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x56a5efd, 0xf, 0xc422a6c2d0, 0xc420a7ee70, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc424dce640, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	<autogenerated>:66 +0x86
github.com/openshift/origin/pkg/cmd/server/handlers.AuthorizationFilter.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/authorization.go:64 +0x113
net/http.HandlerFunc.ServeHTTP(0xc42566b3c0, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/pkg/cmd/server/handlers.ImpersonationFilter.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/impersonation.go:30 +0x312a
net/http.HandlerFunc.ServeHTTP(0xc4251dc460, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/pkg/cmd/server/handlers.AuthenticationHandlerFilter.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/authentication.go:32 +0x299
net/http.HandlerFunc.ServeHTTP(0xc42566b400, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/pkg/cmd/server/origin.namespacingFilter.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:97 +0xd2
net/http.HandlerFunc.ServeHTTP(0xc426903c50, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/pkg/cmd/server/origin.cacheControlFilter.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:65 +0xc2
net/http.HandlerFunc.ServeHTTP(0xc426903c80, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/vendor/github.com/gorilla/context.ClearHandler.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gorilla/context/context.go:141 +0x8b
net/http.HandlerFunc.ServeHTTP(0xc424dcee20, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4252802a0, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:2238 +0x130
net/http.(*ServeMux).ServeHTTP(0xc424d56ba0, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:2238 +0x130
github.com/openshift/origin/pkg/cmd/server/origin.WithPatternsHandler.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:427 +0xc5
net/http.HandlerFunc.ServeHTTP(0xc422c29bc0, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/pkg/cmd/server/origin.WithAssetServerRedirect.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:211 +0x87
net/http.HandlerFunc.ServeHTTP(0xc424d57950, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x189
net/http.HandlerFunc.ServeHTTP(0xc4231afb00, 0x7ff2489ebee0, 0xc42705ad50, 0xc425d7a300)
	/usr/local/go/src/net/http/server.go:1942 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc424e15c60, 0xcc145c0, 0xc42705ad50, 0xc425d7a300, 0xc425d96120)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x1c0

logging error output: "[+]ping ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/ca-registration failed: reason withheld\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\nhealthz check failed\n"
...
I0724 08:47:32.801001    5109 rest.go:349] Starting watch for /api/v1/namespaces/kube-system/secrets, rv=151 labels= fields=type=kubernetes.io/service-account-token timeout=1h39m14.156061774s
I0724 08:47:32.801569    5109 wrap.go:42] GET /api/v1/namespaces/kube-system/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=151&watch=true: (1.442452ms) 200 [[integration.test/v1.7.0+695f48a16f (linux/amd64) kubernetes/695b561] 127.0.0.1:52308]
F0724 08:47:32.804758    5109 client_builder.go:273] unable to get token for service account: error watching
Running TestImageStreamCreate...

See the whole log at: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_integration/4748/consoleFull

[miminar]

Happened again:
https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_check/4872/testReport/junit/github/com_openshift_origin_test_setup_start-master/hack_lib_start_sh_299__executing__oc_get___raw__healthz___config___openshifttmp_openshift_test_cmd_openshift_local_config_master_admin_kubeconfig___expecting_any_result_and_text__ok___re_trying_every_0_25s_until_completion_or_160_000s/

=== BEGIN TEST CASE ===
hack/lib/start.sh:299: executing 'oc get --raw /healthz --config='/openshifttmp/openshift/test-cmd/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 160.000s
FAILURE after 159.776s: hack/lib/start.sh:299: executing 'oc get --raw /healthz --config='/openshifttmp/openshift/test-cmd/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 160.000s: the command timed out
Standard output from the command:
Standard error from the command:
The connection to the server 172.17.0.2:28443 was refused - did you specify the right host or port?
... repeated 29 times
Error from server (Forbidden): User "system:admin" cannot "get" on "/healthz"
... repeated 2 times
Unable to connect to the server: read tcp 172.17.0.2:53456->172.17.0.2:28443: read: connection reset by peer
The connection to the server 172.17.0.2:28443 was refused - did you specify the right host or port?
... repeated 159 times
=== END TEST CASE ===

See the https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_check/4872/s3/download/scripts/test-cmd/logs/openshift.log for the panic in the healthz handler.

This time, it's cmd test.

[miminar]

Another occurence: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_check/4872/testReport/junit/github/com_openshift_origin_test_setup_start-master/hack_lib_start_sh_299__executing__oc_get___raw__healthz___config___openshifttmp_openshift_test_cmd_openshift_local_config_master_admin_kubeconfig___expecting_any_result_and_text__ok___re_trying_every_0_25s_until_completion_or_160_000s/

[jsafrane]

still flaking: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/15725/test_pull_request_origin_integration/7526/

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close