kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory

[tanmoydutta]

Describe the bug
OKD bootstrap node initialization is not successful.
Error in the system log -

Sep 01 18:08:01 bootstrap-test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 01 18:08:01 bootstrap-test kernel: audit: type=1130 audit(1630519681.676:17996): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 01 18:08:11 bootstrap-test systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2323.
Sep 01 18:08:11 bootstrap-test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test systemd[1]: Stopped Kubernetes Kubelet.
Sep 01 18:08:11 bootstrap-test systemd[1]: Starting Kubernetes Kubelet...
Sep 01 18:08:11 bootstrap-test audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test kernel: audit: type=1130 audit(1630519691.789:17997): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test kernel: audit: type=1131 audit(1630519691.789:17998): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test podman[302173]: 2021-09-01 18:08:11.997598591 +0000 UTC m=+0.056148228 container create b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: Started libcrun container.
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.039244503 +0000 UTC m=+0.097794028 container init b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b5>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.044910865 +0000 UTC m=+0.103460388 container start b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.044988448 +0000 UTC m=+0.103537973 container attach b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: libpod-b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733.scope: Deactivated successfully.
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.10129015 +0000 UTC m=+0.159839685 container died b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.119485253 +0000 UTC m=+0.178034778 container remove b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Found left-over process 302210 (conmon) in control group while starting unit. Ignoring.
Sep 01 18:08:12 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 01 18:08:12 bootstrap-test systemd[302257]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Sep 01 18:08:12 bootstrap-test systemd[302257]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 01 18:08:12 bootstrap-test systemd[1]: Failed to start Kubernetes Kubelet.

Version
OKD version: 4.7.0-0.okd-2021-03-28-152009
FCOS version: Fedora CoreOS 34.20210626.3.1
Attempting UPI install for the cluster on IBM Cloud VPC

How reproducible
100% reproducible

Log bundle
openshift_install.log

[vrutkovs]

Please attach log bundle

[tanmoydutta]

ctr.log.txt.gz
Log generation fails. Manually generated the following -

[core@bootstrap-test ~]$ journalctl -b -f -u bootkube.service
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 01 11:21:36 bootstrap-test systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Sep 01 11:21:36 bootstrap-test systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.

[core@bootstrap-test ~]$ journalctl -b -f -u kubelet.service -u crio.service
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 02 09:15:35 bootstrap-test podman[974405]: 2021-09-02 09:15:35.103862505 +0000 UTC m=+0.162732690 container remove b0e6f30c71d1a8df5beea7418faf66889a5249c1b0e7bbc0aa956676599bbe74 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da, name=elated_solomon, io.openshift.release=4.7.0-0.okd-2021-03-28-152009, io.openshift.release.base-image-digest=sha256:69707c5f0c5285128fd7649e2d262fca8b0855dfcd7b3c108e5436290330d4df)
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Found left-over process 974442 (conmon) in control group while starting unit. Ignoring.
Sep 02 09:15:35 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Found left-over process 974454 (podman) in control group while starting unit. Ignoring.
Sep 02 09:15:35 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 02 09:15:35 bootstrap-test systemd[974489]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Sep 02 09:15:35 bootstrap-test systemd[974489]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 02 09:15:35 bootstrap-test systemd[1]: Failed to start Kubernetes Kubelet.

[vrutkovs]

Check release-image and release-image-download systemd service logs - if your machine doesn't have kubelet binary it means it didn't pivot itself in OKD ostree commit

[tanmoydutta]

[core@bootstrap-test ~]$ journalctl -b -f -u release-image
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 01 11:21:32 bootstrap-test systemd[1]: Starting Download the OpenShift Release Image...
Sep 01 11:21:32 bootstrap-test release-image-download.sh[764]: Pulling quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da...
Sep 01 11:21:34 bootstrap-test podman[769]: 2021-09-01 11:21:34.087209148 +0000 UTC m=+1.571760235 system refresh
Sep 01 11:21:34 bootstrap-test podman[769]: 2021-09-01 11:21:34.087947075 +0000 UTC m=+1.572498150 image pull quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da
Sep 01 11:21:35 bootstrap-test release-image-download.sh[769]: 3b229e39bc869ea76bc4493ec494610fd132259b2f98fab86e6fc9ae895c9a21
Sep 01 11:21:35 bootstrap-test systemd[1]: Finished Download the OpenShift Release Image.

[core@bootstrap-test ~]$ journalctl -b -f -u release-image-download
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --

[vrutkovs]

Are you sure you're using OKD installer and not one from OCP?

[tanmoydutta]

Yes, this is the version output of the installer -

% ./openshift-install version
./openshift-install 4.7.0-0.okd-2021-03-28-152009
built from commit cf35dc04aedf7a80bea5470358025cbe91d232c9
release image quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da

Ignition config was created using the same.

[vrutkovs]

Check bootstrap.ign - it must have release-image-download service

[tanmoydutta]

Yes I see two occurrences of that. e.g. -
{"overwrite":true,"path":"/usr/local/bin/release-image-download.sh","user":{"name":"root"},"contents":{"source":"data:text/plain;charset=utf-8...

and another -

ExecStart=/usr/local/bin/release-image-download.sh

[tanmoydutta]

Boot.log
Here are some more info from boot for the bootstrap node (See attached file)

[vrutkovs]

• error copying from container: error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/var/lib/containers/storage/overlay/7a128f642baf45e650e6e1cad241b864bffd15fefc8>

Is this a libvirt install?

[tanmoydutta]

No, VSI provisioning on IBM Cloud using FCOS qcow2 image

[tanmoydutta]

Any update on this ?

[wupeng606]

I'm facing this error in bootstrap node of OKD 4.9.
ENV:
vSphere 7.0.2,
Restricted network with Private registry Harbor,
FCOS 35.20220213.3.0

[pkqsun]

Facing same issue of using:

openshift-install 4.10.0-0.okd-2022-07-09-073606
built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
release image quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d
release architecture amd64
fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2

Jul 22 16:24:58 okd410-6jfbz-bootstrap podman[30484]: 2022-07-22 16:24:58.719643717 +0000 UTC m=+0.263650635 container died 508e873e676695e66890a75b9e1978df00733776b95438a2334d89cd0258acca (image=quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d, name=vibrant_cerf)
Jul 22 16:24:58 okd410-6jfbz-bootstrap podman[30484]: 2022-07-22 16:24:58.782347878 +0000 UTC m=+0.326354796 container remove 508e873e676695e66890a75b9e1978df00733776b95438a2334d89cd0258acca (image=quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d, name=vibrant_cerf, io.openshift.release.base-image-digest=sha256:6c1c967d70be9cb1f03ac612bbe0c0a564302e9d0ae16f857a33a298ca55c94a, io.openshift.release=4.10.0-0.okd-2022-07-09-073606)
Jul 22 16:24:58 okd410-6jfbz-bootstrap kubelet.sh[30592]: /usr/local/bin/kubelet.sh: line 7: /usr/bin/hyperkube: No such file or directory
Jul 22 16:24:58 okd410-6jfbz-bootstrap systemd[1]: kubelet.service: Main process exited, code=exited, status=127/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit kubelet.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 127.
Jul 22 16:24:58 okd410-nuke-6jfbz-bootstrap systemd[1]: kubelet.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
Jul 22 16:24:58 okd410-nuke-6jfbz-bootstrap systemd[1]: Failed to start Kubernetes Kubelet.
░░ Subject: A start job for unit kubelet.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[yaroze]

/usr/local/bin/kubelet.sh: line 7: /usr/bin/hyperkube: No such file or directory

Getting exactly the same.

openshift-install 4.12.0-0.okd-2023-03-05-022504
built from commit 7c2530226516a12c37f10bc14e070f66c0f27930
release image quay.io/openshift/okd@sha256:fef2d0803e6ee6ec36d1979a0b847441580fbd9ee3ad583c97edb7ff811ee3b6
☝️ I might be looking at the wrong place but I saw that image doesn't have /usr/bin/hyperkube
release architecture amd64
fedora-coreos-37.20230218.3.0-live.x86_64.iso

Should I try a different build?

[rbuzatu90]

@tanmoydutta did you manage to have it working ?

[tandutta]

@rbuzatu90 We did not solve it in the end as we also could not find any solution nor support. We just moved to another hyperscaler.

[rbuzatu90]

I'm also having this exact issue on 4.10. Anyone have found the fix ?