Authentication Unknown state starting cluster on bare metal

[youssefbouzaine]

Hello

Actually i stated okd cluster version 4.5 after 4 hours of booting i. m still facing an unknown state on Authentication operator

`NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                                                             Unknown     Unknown       True       4h13m
cloud-credential                           4.5.0-0.okd-2020-10-15-235428   True        False         False      4h18m
cluster-autoscaler                         4.5.0-0.okd-2020-10-15-235428   True        False         False      4h7m
config-operator                            4.5.0-0.okd-2020-10-15-235428   True        False         False      4h6m
console                                    4.5.0-0.okd-2020-10-15-235428   False       True          True       3h59m
csi-snapshot-controller                    4.5.0-0.okd-2020-10-15-235428   True        False         False      4h8m
dns                                        4.5.0-0.okd-2020-10-15-235428   True        False         False      4h11m
etcd                                       4.5.0-0.okd-2020-10-15-235428   True        False         False      4h11m
image-registry                             4.5.0-0.okd-2020-10-15-235428   True        False         False      4h8m
ingress                                    4.5.0-0.okd-2020-10-15-235428   True        False         False      3h59m
insights                                   4.5.0-0.okd-2020-10-15-235428   True        False         False      4h8m
kube-apiserver                             4.5.0-0.okd-2020-10-15-235428   True        False         False      4h11m
kube-controller-manager                    4.5.0-0.okd-2020-10-15-235428   True        False         False      4h11m
kube-scheduler                             4.5.0-0.okd-2020-10-15-235428   True        False         False      4h6m
kube-storage-version-migrator              4.5.0-0.okd-2020-10-15-235428   True        False         False      3h59m
machine-api                                4.5.0-0.okd-2020-10-15-235428   True        False         False      4h8m
machine-approver                           4.5.0-0.okd-2020-10-15-235428   True        False         False      4h10m
machine-config                             4.5.0-0.okd-2020-10-15-235428   True        False         False      4h11m
marketplace                                4.5.0-0.okd-2020-10-15-235428   True        False         False      4h7m
monitoring                                 4.5.0-0.okd-2020-10-15-235428   True        False         False      3h58m
network                                    4.5.0-0.okd-2020-10-15-235428   True        False         False      4h13m
node-tuning                                4.5.0-0.okd-2020-10-15-235428   True        False         False      4h13m
openshift-apiserver                        4.5.0-0.okd-2020-10-15-235428   True        False         False      3h59m
openshift-controller-manager               4.5.0-0.okd-2020-10-15-235428   True        False         False      4h7m
openshift-samples                          4.5.0-0.okd-2020-10-15-235428   True        True          True       4h4m
operator-lifecycle-manager                 4.5.0-0.okd-2020-10-15-235428   True        False         False      4h12m
operator-lifecycle-manager-catalog         4.5.0-0.okd-2020-10-15-235428   True        False         False      4h12m
operator-lifecycle-manager-packageserver   4.5.0-0.okd-2020-10-15-235428   True        False         False      3h12m
service-ca                                 4.5.0-0.okd-2020-10-15-235428   True        False         False      4h13m
storage                                    4.5.0-0.okd-2020-10-15-235428   True        False         False      4h8m`

i m seeing also that console i degraded

you can check below logs of authentication operator

oc get co authentication -o json | jq '.status.conditions[0]'

{
  "lastTransitionTime": "2021-07-12T11:24:13Z",
  "message": "RouteHealthDegraded: failed to GET route: Service Unavailable",
  "reason": "RouteHealth_FailedGet",
  "status": "True",
  "type": "Degraded"
}

check below log of console operator

oc -n openshift-console get event
LAST SEEN   TYPE      REASON                 OBJECT                        MESSAGE
146m        Warning   Unhealthy              pod/console-889d4f799-txkmh   Readiness probe failed: Get https://10.129.0.35:8443/health: dial tcp 10.129.0.35:8443: connect: connection refused
151m        Normal    Pulled                 pod/console-889d4f799-txkmh   Container image "quay.io/openshift/okd-content@sha256:f4756951cf7f923b049f860310f38b8acd95ad6c6024e41b93f3e5e121b89734" already present on machine
141m        Warning   BackOff                pod/console-889d4f799-txkmh   Back-off restarting failed container
114m        Normal    Pulled                 pod/console-889d4f799-txkmh   Container image "quay.io/openshift/okd-content@sha256:f4756951cf7f923b049f860310f38b8acd95ad6c6024e41b93f3e5e121b89734" already present on machine
139m        Normal    Created                pod/console-889d4f799-txkmh   Created container console
139m        Normal    Started                pod/console-889d4f799-txkmh   Started container console
139m        Normal    TaintManagerEviction   pod/console-889d4f799-txkmh   Cancelling deletion of Pod openshift-console/console-889d4f799-txkmh
4m34s       Warning   Unhealthy              pod/console-889d4f799-txkmh   Readiness probe failed: Get https://10.129.0.35:8443/health: dial tcp 10.129.0.35:8443: connect: connection refused
136m        Warning   Unhealthy              pod/console-889d4f799-txkmh   Liveness probe failed: Get https://10.129.0.35:8443/health: dial tcp 10.129.0.35:8443: connect: connection refused
136m        Normal    Killing                pod/console-889d4f799-txkmh   Container console failed liveness probe, will be restarted
9m25s       Warning   BackOff                pod/console-889d4f799-txkmh   Back-off restarting failed container
4m38s       Warning   Unhealthy              pod/console-889d4f799-vnvhl   Readiness probe failed: Get https://10.128.0.43:8443/health: dial tcp 10.128.0.43:8443: connect: connection refused
49m         Warning   Unhealthy              pod/console-889d4f799-vnvhl   Liveness probe failed: Get https://10.128.0.43:8443/health: dial tcp 10.128.0.43:8443: connect: connection refused
99m         Normal    Pulled                 pod/console-889d4f799-vnvhl   Container image "quay.io/openshift/okd-content@sha256:f4756951cf7f923b049f860310f38b8acd95ad6c6024e41b93f3e5e121b89734" already present on machine
9m46s       Warning   BackOff                pod/console-889d4f799-vnvhl   Back-off restarting failed container

any help will be appreciated

[youssefbouzaine]

@vrutkovs this is not an okd issue ?

my cluster is not getting up after 4 hours of booting

[vrutkovs]

This is not an actionable item we can start fixing right away. If we'll confirm this to be something we need to fix we'll convert it back to the issue again.

[vrutkovs]

Also, the bug is filed for OKD 4.5. Is it reproducible in the latest stable or OKD 4.8?

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[youssefbouzaine]

you can find below the output of muust-gather command

oc adm must-gather
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/okd-content@sha256:5a9b4b4cdd15e8587d02490cf753c385213e6a97e5c2bf7babf416585e42fd70
[must-gather      ] OUT namespace/openshift-must-gather-w4vtc created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-tg9mx created
[must-gather      ] OUT pod for plug-in image quay.io/openshift/okd-content@sha256:5a9b4b4cdd15e8587d02490cf753c385213e6a97e5c2bf7babf416585e42fd70 created
[must-gather-x475m] OUT gather logs unavailable: Get https://192.168.10.135:10250/containerLogs/openshift-must-gather-w4vtc/must-gather-x475m/gather?follow=true: Forbidden
[must-gather-x475m] OUT waiting for gather to complete
[must-gather-x475m] OUT gather never finished: timed out waiting for the condition
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-tg9mx deleted
[must-gather      ] OUT namespace/openshift-must-gather-w4vtc deleted
error: gather never finished for pod must-gather-x475m: timed out waiting for the condition