Skip to content

Multiple EgressNetworkPolicies in same net namespace with ovs-multitenant SDN #12038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aliscott opened this issue Nov 28, 2016 · 9 comments
Closed

Multiple EgressNetworkPolicies in same net namespace with ovs-multitenant SDN #12038

aliscott opened this issue Nov 28, 2016 · 9 comments
Assignees
@danwinship @knobunc
Labels
component/networking kind/bug Categorizes issue or PR as related to a bug. priority/P2

Comments

@aliscott
Copy link

Using the ovs-multitenant SDN plugin, when I add any EgressNetworkPolicy to a project I get the following error:

multiple EgressNetworkPolicies in same network namespace (sledgehammer:default, ali-test-1:default) is not allowed; dropping all traffic

But the network namespaces for these projects are different:

[root@master1 ~]# oc get netnamespaces
NAME               NETID
ali-test-1         11875203
default            0
kube-system        7799796
logging            2757896
management-infra   2434755
openshift          12423517
openshift-infra    2947485
sledgehammer       3912440

Any ideas what could be causing this issue?

Version
[root@master1 ~]# openshift version
openshift v1.4.0-alpha.1+5f9a5ec-217
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Additional Information
[root@master1 ~]# oadm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'default/dashboard-ahateam-internal:8443/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[network-diag-ns-uyl4a/dashboard-ahateam-internal:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'network-diag-ns-uyl4a/dashboard-ahateam-internal:8443/system:admin':
       The server URL is 'https://dashboard.ahateam.internal:8443'
       The user authentication is 'system:admin/dashboard-ahateam-internal:8443'
       The current project is 'network-diag-ns-uyl4a'
       Successfully requested project list; has access to project(s):
         [aha-internal alistair-test management-infra openshift-infra ali-test-1 default electune examples kube-system
logging ...]

[Note] Running diagnostic: ConfigContexts[default/dashboard-ahateam-com:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'default/dashboard-ahateam-com:8443/system:admin':
       The server URL is 'https://dashboard.ahateam.com:8443'
       The user authentication is 'system:admin/dashboard-ahateam-internal:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [default electune examples sledgehammer openshift openshift-infra aha-internal ali-test-1 alistair-test
kube-system ...]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

Info:  Output from the diagnostic pod (image openshift/origin-deployer:v1.4.0-alpha.1):
       [Note] Running diagnostic: PodCheckAuth
              Description: Check that service account credentials authenticate as expected

       Info:  Service account token successfully authenticated to master
       Info:  Service account token was authenticated by the integrated registry.

       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+f189ede):
       [Note] Completed with no errors or warnings seen.

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

Info:  Output from the network diagnostic pod on node "node-fbcb9413635c.eu-central-1a.ahateam.internal":
       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
       [Note] Completed with no errors or warnings seen.

Info:  Output from the network diagnostic pod on node "master1.eu-central-1a.ahateam.internal":
       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
       [Note] Completed with no errors or warnings seen.

Info:  Output from the network diagnostic pod on node "master3.eu-central-1a.ahateam.internal":
       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
       [Note] Completed with no errors or warnings seen.

Info:  Output from the network diagnostic pod on node "node-aa3991db123d.eu-central-1a.ahateam.internal":
       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
       [Note] Completed with no errors or warnings seen.

Info:  Output from the network diagnostic pod on node "master2.eu-central-1a.ahateam.internal":
       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
       [Note] Completed with no errors or warnings seen.

[Note] Running diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

Info:  The "docker-registry" service has multiple associated pods each mounted with
       ephemeral storage, but also has a custom config /etc/docker/registry/config.yml
       mounted; assuming storage config is as desired.

WARN:  [DClu1012 from diagnostic ClusterRegistry@openshift/origin/pkg/diagnostics/cluster/registry.go:300]
       The pod logs for the "docker-registry-10-aydb8" pod belonging to
       the "docker-registry" service indicated unknown errors.
       This could result in problems with builds or deployments.
       Please examine the log entries to determine if there might be
       any related problems:

       time="2016-11-27T22:34:20.559319265Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=f4d6acf1-eb37-4aaa-8962-3638832705f9
http.request.method=GET http.request.remoteaddr="10.130.2.1:41242" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:50:11.790562091Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=4e1981a1-2b9e-453a-b0a2-397256ba1b3c
http.request.method=GET http.request.remoteaddr="10.130.2.1:42180" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:50:12.602118527Z" level=error msg="response completed with error" err.code="blob unknown"
err.detail=sha256:fd7b0ddd0fdf6b4bffd80315a3e6529a861a764bafad6c12aa95ef7758c11e2c err.message="blob unknown to
registry" go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=371422a6-b1e8-4e78-a701-3f36a31f292d
http.request.method=HEAD http.request.remoteaddr="10.130.2.1:42208"
http.request.uri="/v2/examples/php-laravel-todo/blobs/sha256:fd7b0ddd0fdf6b4bffd80315a3e6529a861a764bafad6c12aa95ef7758c11e2c"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=79.239715ms
http.response.status=404 http.response.written=157 instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
vars.digest="sha256:fd7b0ddd0fdf6b4bffd80315a3e6529a861a764bafad6c12aa95ef7758c11e2c"
vars.name="examples/php-laravel-todo"
       time="2016-11-27T22:50:21.585277876Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=df8170a6-69bd-48d6-a764-6253f8bc1cf7
http.request.method=GET http.request.remoteaddr="10.130.2.1:42228" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:50:24.964098991Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=ab90643a-ca0d-4f2f-a16e-474f6d2d47eb
http.request.method=GET http.request.remoteaddr="10.130.2.1:42240" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:50:41.706713168Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=b8999f67-7a7e-4119-877e-86835af6f585
http.request.method=GET http.request.remoteaddr="10.130.2.1:42254" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:51:09.712403774Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=17ec053c-9208-42f3-ad96-c21fd5513c8f
http.request.method=GET http.request.remoteaddr="10.130.2.1:42280" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:51:54.70614442Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=826e123c-8121-490e-ad3d-e260e9e8d366
http.request.method=GET http.request.remoteaddr="10.130.2.1:42310" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:53:25.714171428Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=243e884b-3b8e-4e2f-a6f3-7b990a50ea7f
http.request.method=GET http.request.remoteaddr="10.130.2.1:42372" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T22:56:09.714395124Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=d4a23a12-30fa-4ad9-8020-e846ccafb00a
http.request.method=GET http.request.remoteaddr="10.130.2.1:42532" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T23:01:14.70867189Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=8065f182-afeb-4b97-ac04-73163a44323d
http.request.method=GET http.request.remoteaddr="10.130.2.1:42912" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T23:01:47.888096853Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=aa7f59ba-8aee-40d7-ad26-60475975ce9b
http.request.method=GET http.request.remoteaddr="10.130.2.1:42942" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
       time="2016-11-27T23:01:57.64095501Z" level=error msg="response completed with error" err.code="blob unknown"
err.detail=sha256:1e1f479df78611ae5dca307cb174f4939b753bffba0a0fcecd02a70309c5d1eb err.message="blob unknown to
registry" go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=6df0a06f-3084-4284-b150-981a93f4d3e7
http.request.method=HEAD http.request.remoteaddr="10.130.2.1:42974"
http.request.uri="/v2/examples/php-laravel-todo/blobs/sha256:1e1f479df78611ae5dca307cb174f4939b753bffba0a0fcecd02a70309c5d1eb"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=41.38504ms
http.response.status=404 http.response.written=157 instance.id=db6d28b8-1f64-4b2d-be55-567a705fcc36
vars.digest="sha256:1e1f479df78611ae5dca307cb174f4939b753bffba0a0fcecd02a70309c5d1eb"
vars.name="examples/php-laravel-todo"

WARN:  [DClu1012 from diagnostic ClusterRegistry@openshift/origin/pkg/diagnostics/cluster/registry.go:300]
       The pod logs for the "docker-registry-10-k3syp" pod belonging to
       the "docker-registry" service indicated unknown errors.
       This could result in problems with builds or deployments.
       Please examine the log entries to determine if there might be
       any related problems:

       time="2016-11-27T22:42:14.571280919Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=0a8e9823-5fc7-4cc2-b9a5-8ee67a2519c4
http.request.method=GET http.request.remoteaddr="10.130.2.1:41822" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=9b7e6c0b-ce37-432d-b480-d46e5a8b8718
       time="2016-11-27T22:42:22.888729705Z" level=error msg="response completed with error" err.code="blob unknown"
err.detail=sha256:9f578a5ee75d25986442aa6ba2421fbf06f50f136d89d0fd03117ca0f1638df9 err.message="blob unknown to
registry" go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=3197a123-2bf8-426a-95d3-5831cbd1299e
http.request.method=HEAD http.request.remoteaddr="10.130.2.1:41858"
http.request.uri="/v2/examples/php-laravel-todo/blobs/sha256:9f578a5ee75d25986442aa6ba2421fbf06f50f136d89d0fd03117ca0f1638df9"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=41.533763ms
http.response.status=404 http.response.written=157 instance.id=9b7e6c0b-ce37-432d-b480-d46e5a8b8718
vars.digest="sha256:9f578a5ee75d25986442aa6ba2421fbf06f50f136d89d0fd03117ca0f1638df9"
vars.name="examples/php-laravel-todo"
       time="2016-11-27T22:42:35.098251816Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=40df3ab2-10a1-4dcf-877e-48761fa9f8ee
http.request.method=GET http.request.remoteaddr="10.130.0.1:40504" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=9b7e6c0b-ce37-432d-b480-d46e5a8b8718
       time="2016-11-27T23:02:05.979205333Z" level=error msg="error authorizing context: authorization header required"
go.version=go1.6 http.request.host="172.30.150.55:5000" http.request.id=31a984c7-9812-4cac-8e95-941c0da8aac0
http.request.method=GET http.request.remoteaddr="10.130.0.1:41718" http.request.uri="/v2/"
http.request.useragent="docker/1.10.3 go/go1.6.3 git-commit/cb079f6-unsupported kernel/3.10.0-327.36.3.el7.x86_64
os/linux arch/amd64" instance.id=9b7e6c0b-ce37-432d-b480-d46e5a8b8718

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra
subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount management-infra management-admin    }.

Info:  clusterrolebinding/self-provisioners has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra
subjects.

Info:  clusterrolebinding/self-provisioners has extra subject {ServiceAccount management-infra management-admin    }.

WARN:  [CRBD1003 from diagnostic ClusterRoleBindings@openshift/origin/pkg/diagnostics/cluster/rolebindings.go:87]
       clusterrolebinding/system:build-strategy-jenkinspipeline-binding is missing expected subjects.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to include expected
subjects.

Info:  clusterrolebinding/system:build-strategy-jenkinspipeline-binding is missing subject {SystemGroup
system:authenticated    }.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

[Note] Running diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)

WARN:  [DClu3004 from diagnostic MasterNode@openshift/origin/pkg/diagnostics/cluster/master_node.go:164]
       Unable to find a node matching the cluster server IP.
       This may indicate the master is not also running a node, and is unable
       to proxy to pods over the Open vSwitch SDN.

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node node-8ca6f70f2de6.eu-central-1a.ahateam.internal is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node node-8ca6f70f2de6.eu-central-1a.ahateam.internal --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

[Note] Running diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config

[Note] Running diagnostic: AnalyzeLogs
       Description: Check for recent problems in systemd service logs

Info:  Checking journalctl logs for 'origin-node' service
Info:  Checking journalctl logs for 'docker' service

[Note] Running diagnostic: MasterConfigCheck
       Description: Check the master config file

ERROR: [DH0004 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:45]
       Validation of master config file '/etc/origin/master/master-config.yaml' failed:
       oauthConfig.identityProvider[1].provider.clientSecret: Required value
       oauthConfig.identityProvider[2].provider.clientSecret: Required value

WARN:  [DH0005 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:52]
       Validation of master config file '/etc/origin/master/master-config.yaml' warned:
       assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console
       assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console
       assetConfig.servingInfo: Invalid value: "\u003cnot displayed\u003e": changes to assetConfig certificate
configuration are not used when colocated with master API
       auditConfig.auditFilePath: Required value: audit can now be logged to a separate file

[Note] Running diagnostic: NodeConfigCheck
       Description: Check the node config file

Info:  Found a node config file: /etc/origin/node/node-config.yaml

[Note] Running diagnostic: UnitStatus
       Description: Check status for related systemd units

[Note] Summary of diagnostics execution (version v1.4.0-alpha.1+5f9a5ec-217):
[Note] Warnings seen: 6
[Note] Errors seen: 1
@pweil- pweil- added component/networking kind/bug Categorizes issue or PR as related to a bug. priority/P2 labels Nov 28, 2016
@danwinship
Copy link
Contributor

have you merged netnamespaces (oadm pod-networks join-networks ...) in the past?

can you attach the output of oc get netnamespaces -o json?

@aliscott
Copy link
Author

have you merged netnamespaces (oadm pod-networks join-networks ...) in the past?

No

can you attach the output of oc get netnamespaces -o json?

[root@master1 ~]# oc get netnamespaces -o json
{
    "kind": "List",
    "apiVersion": "v1",
    "metadata": {},
    "items": [
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "aha-internal",
                "selfLink": "/oapi/v1/netnamespaces/aha-internal",
                "uid": "d9fc20c3-b255-11e6-950b-00163e000872",
                "resourceVersion": "9597",
                "creationTimestamp": "2016-11-24T14:54:08Z"
            },
            "netname": "aha-internal",
            "netid": 9574629
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "ali-test-1",
                "selfLink": "/oapi/v1/netnamespaces/ali-test-1",
                "uid": "891a0f6c-b2fd-11e6-9f08-00163e00029a",
                "resourceVersion": "78917",
                "creationTimestamp": "2016-11-25T10:54:27Z"
            },
            "netname": "ali-test-1",
            "netid": 11875203
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "default",
                "selfLink": "/oapi/v1/netnamespaces/default",
                "uid": "182f8cf2-b245-11e6-91f4-00163e00029a",
                "resourceVersion": "451",
                "creationTimestamp": "2016-11-24T12:54:10Z"
            },
            "netname": "default",
            "netid": 0
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "kube-system",
                "selfLink": "/oapi/v1/netnamespaces/kube-system",
                "uid": "18305554-b245-11e6-91f4-00163e00029a",
                "resourceVersion": "452",
                "creationTimestamp": "2016-11-24T12:54:10Z"
            },
            "netname": "kube-system",
            "netid": 7799796
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "logging",
                "selfLink": "/oapi/v1/netnamespaces/logging",
                "uid": "436b6ac7-b246-11e6-9f20-00163e0009d8",
                "resourceVersion": "902",
                "creationTimestamp": "2016-11-24T13:02:32Z"
            },
            "netname": "logging",
            "netid": 2757896
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "management-infra",
                "selfLink": "/oapi/v1/netnamespaces/management-infra",
                "uid": "3ae99c3b-b245-11e6-9f20-00163e0009d8",
                "resourceVersion": "511",
                "creationTimestamp": "2016-11-24T12:55:09Z"
            },
            "netname": "management-infra",
            "netid": 2434755
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "openshift",
                "selfLink": "/oapi/v1/netnamespaces/openshift",
                "uid": "1833e565-b245-11e6-91f4-00163e00029a",
                "resourceVersion": "454",
                "creationTimestamp": "2016-11-24T12:54:10Z"
            },
            "netname": "openshift",
            "netid": 12423517
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "openshift-infra",
                "selfLink": "/oapi/v1/netnamespaces/openshift-infra",
                "uid": "18331a60-b245-11e6-91f4-00163e00029a",
                "resourceVersion": "453",
                "creationTimestamp": "2016-11-24T12:54:10Z"
            },
            "netname": "openshift-infra",
            "netid": 2947485
        },
        {
            "kind": "NetNamespace",
            "apiVersion": "v1",
            "metadata": {
                "name": "sledgehammer",
                "selfLink": "/oapi/v1/netnamespaces/sledgehammer",
                "uid": "e05b877c-b2ed-11e6-9f08-00163e00029a",
                "resourceVersion": "70246",
                "creationTimestamp": "2016-11-25T09:02:22Z"
            },
            "netname": "sledgehammer",
            "netid": 3912440
        }
    ]
}

@danwinship danwinship mentioned this issue Nov 28, 2016
Merged
@danwinship
Copy link
Contributor

So this is fixed in git now and is also on the release-1.4 branch so it should eventually end up in a patch release if you can't build from git. Unfortunately there's no way to work around the bug until then.

@aliscott
Copy link
Author

aliscott commented Dec 3, 2016

Thanks @danwinship 😄

@aliscott
Copy link
Author

@danwinship I have started seeing this issue again in OpenShift 1.4.1

The difference this time is it is detecting the same single EgressNetworkPolicy multiple times:

multiple EgressNetworkPolicies in same network namespace (ali-test-1:default, ali-test-1:default) is not allowed; dropping all traffic

Version

[root@master3 ~]# oc version
oc v1.4.1
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dashboard.abarcloud.internal:8443
openshift v1.4.1
kubernetes v1.4.0+776c994

@danwinship
Copy link
Contributor

Is the error only happening after restarting a node? Or just randomly?

@aliscott
Copy link
Author

@danwinship It's happening randomly. The issue goes away when I restart origin-node.

@aliscott
Copy link
Author

@danwinship should I open a new issue for this?

@danwinship
Copy link
Contributor

yeah, I guess so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danwinship danwinship

@knobunc knobunc

Labels
component/networking kind/bug Categorizes issue or PR as related to a bug. priority/P2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@danwinship @aliscott @pweil- @knobunc