Unable to upgrade, etcd backup pod fails

[bianchi2]

When trying to upgrade, etc-packup pod errors out with:

+ '[' -n /etc/kubernetes/cluster-backup ']'
+ echo 'removing all backups in /etc/kubernetes/cluster-backup'
+ rm -rf /etc/kubernetes/cluster-backup
removing all backups in /etc/kubernetes/cluster-backup
rm: cannot remove '/etc/kubernetes/cluster-backup': Device or resource busy

OKD is installed on AWS.

Version
4.14.0-0.okd-2024-01-06-084517

[marlonfrank30]

getting the same issue however on esxi vmware... also followed the documentation available here https://docs.okd.io/4.15/updating/updating_a_cluster/updating-cluster-web-console.html#updating-cluster-web-console ...have you got any feedback?

[JaimeMagiera]

Hi,

We are not working on FCOS builds of OKD any more. Please see these documents...

https://okd.io/blog/2024/06/01/okd-future-statement
https://okd.io/blog/2024/07/30/okd-pre-release-testing

We will be providing documentation on upgrading clusters from 4.15 FCOS to 4.16 SCOS. In terms of clusters that are older, you may be able to get help from community members. I'll convert this to a discussion to facilitate that.

Many thanks,

Jaime

[EyalPazz]

Any solution?

[JaimeMagiera]

Closing this since 4.14 is not on anyone's radar.

[oncach]

Hi,
I did hit this error while updating from 4.14.0-0.okd-2024-01-06-084517 to 4.15.0-0.okd-2024-02-10-035534 on vmw esxi,
oc get clusterversion returned that update isn't progressing and oc describe clusterversion returned condition with message:

'Preconditions failed for payload loaded version="4.15.0-0.okd-2024-02-10-035534"
 image="registry.ci.openshift.org/origin/release@sha256:e13d5f24fc0347b51f1144e7f5bb03b48a99b43d3de0a260af171bf7e39a30f7":
 Precondition "EtcdRecentBackup" failed because of "UpgradeBackupFailed": RecentBackup: pod failed within retry duration: delete skipped'

As described in above posts, problem is with cluster-backup pod in openshift-etcd namespace - oc get pod/cluster-backup -n openshift-etcd -o wide --watch did show that pod is repeatedly run on one of the master nodes, ocassionally changing the node where it gets scheduled and according to its logs pod always finished with error - "rm: cannot remove '/etc/kubernetes/cluster-backup': Device or resource busy". The reason why this happened is that pods container 'cluster-backup' is run with env variable DELETE_BACKUP_DIR set to "/etc/kubernetes/cluster-backup" and runs sequence of shell commands one of whitch is rm -rf "${DELETE_BACKUP_DIR}", at the same time this directory is a mount point over which is mounted host's directory with the same path - mounted over mount point can't be deleted, hence that error.

To solve this i came with a dirty hack, cause i didn't see how to solve this otherwise (suppose there's other simplier solution, just couldn't find it on net), should you use it, use it at your own risk (i do not state that following steps are complete and without errors and can not cause any harm, you have been warned, it just worked for me;).
I found that cluster-backup pod container is using the same image as running etcd pods containers, i used this stable etcd pod to find the extracted layers of the image on one of the master nodes, then within one of this images extracted layers containing rm command i renamed the rm command to 'rm.orig' and created bash scrit named 'rm' which calls original 'rm.orig' and always exits with success return code 0. After this modification I waited till in a while cluster-backup pod gets scheduled on this node with modified image - this time the pod finished without errors and OCP update to 4.15. started to proceed, then i renamed rm.orig back to rm to revert extracted image modification.

Step-by-step on my master node ocp-abcde-master-0 (the following commands options values and outputs are from my environment, adjust to fit yours):

1 )
take a note of the image name of the cluster-backup container of the cluster-backup pod in openshift-etcd namespace:
oc get pod/cluster-backup -n openshift-etcd -o jsonpath='{.spec.containers[0].image}'
in my case it was: quay.io/openshift/okd-content@sha256:258e66d455b627549e156489886e544b77467e8e8e1ac6a359735ae7c5c33c6c

2 )
ssh or oc debug node to one of master nodes and switch to host filesystem and to root user, following steps marked with .master are to be run on this master node

3.master)
get image id:
crictl img quay.io/openshift/okd-content@sha256:258e66d455b627549e156489886e544b77467e8e8e1ac6a359735ae7c5c33c6c

IMAGE TAG IMAGE ID SIZE
quay.io/openshift/okd-content 6a5a18fc59e7b 543MB

4.master)
list containers/pods using this image:
crictl ps --image 6a5a18fc59e7b

CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
d8d9143a3a5db 6a5a18fc59e7b2bc22e1c340f1a286ee75d5f290ab8eba81976e0305a2199ac9 19 hours ago Running etcd-metrics 0 b9bffdc05a568 etcd-ocp-abcde-master-0
118e0f578986c 6a5a18fc59e7b2bc22e1c340f1a286ee75d5f290ab8eba81976e0305a2199ac9 19 hours ago Running etcd 0 b9bffdc05a568 etcd-ocp-abcde-master-0
fc7447d3f2b2c 6a5a18fc59e7b2bc22e1c340f1a286ee75d5f290ab8eba81976e0305a2199ac9 19 hours ago Running etcdctl 0 b9bffdc05a568 etcd-ocp-abcde-master-0

5.master)
use one of the containers - in my case etcd with container id 118e0f578986c and find where is its root directory mounted on host
crictl inspect 118e0f578986c | jq .info.runtimeSpec.root

{ "path": "/var/lib/containers/storage/overlay/9e131d182c94659dc23e2f03b666cb1920483801daa1d54d31dd49edf22e6b59/merged" }

6.master)
find the extracted layers of the image from whitch is the root fs of the container constructed:
mount | grep "/var/lib/containers/storage/overlay/9e131d182c94659dc23e2f03b666cb1920483801daa1d54d31dd49edf22e6b59/merged"

overlay on /var/lib/containers/storage/overlay/9e131d182c94659dc23e2f03b666cb1920483801daa1d54d31dd49edf22e6b59/merged type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c697,c920",
  lowerdir=/var/lib/containers/storage/overlay/l/IGMCYTCKPR25REHZS6UYYGH7BP:
           /var/lib/containers/storage/overlay/l/DKK4PCL7L4H4YCGY5W3Q3D6MPH:
           /var/lib/containers/storage/overlay/l/WANL6U67B5ZKQFLIQPE5ENE3UT:
           /var/lib/containers/storage/overlay/l/26KLGRFCQ76OXHHQN52UEW6AAB:
           /var/lib/containers/storage/overlay/l/3BIPZY3R7ER73WICSH66GK56G6:
           /var/lib/containers/storage/overlay/l/N5A3GZG4I37DW4DPGS5L3GELSC,
  upperdir=.....

(output was modified to make it more readable)

7.master)
find the rm binary in one of the lowerdir directories from the output of the previous step command:
ls -l /var/lib/containers/storage/overlay/l/N5A3GZG4I37DW4DPGS5L3GELSC/usr/bin/rm

-r-xr-xr-x. 1 root root 49 Jul 9 2021 /var/lib/containers/storage/overlay/l/N5A3GZG4I37DW4DPGS5L3GELSC/usr/bin/rm

8 ) run the following command to see where the pod cluster-backup gets scheduled and what is its status (let it run and continue with other steps)
oc get pod/cluster-backup -n openshift-etcd -o wide --watch

9.master)
rename rm to rm.orig and create 'allways successfull' rm script:

cd /var/lib/containers/storage/overlay/l/N5A3GZG4I37DW4DPGS5L3GELSC/usr/bin/
mv rm rm.orig
cat <<_EOT_ > rm
#!/bin/bash
/usr/bin/rm.orig "\$@"
exit 0
_EOT_
chmod a+x rm

10 )
check that rm change went well - check /usr/bin/rm* commands in the filesystem of the running etcd pod on the master node
oc exec pod/etcd-ocp-abcde-master-0 -n openshift-etcd -c etcd -ti -- sh -c 'ls -l /usr/bin/rm*'

-rw-r--r--. 1 root root 41 Apr 28 14:54 /usr/bin/rm                    
-r-xr-xr-x. 1 root root 49 Jul  9  2021 /usr/bin/rm.orig
-r-xr-xr-x. 1 root root 52 Jul  9  2021 /usr/bin/rmdir

11 )
wait till the cluster-backup pod gets scheduled on master node where changes to rm were made -

• watch the output of the command from the step 8 )

oc get pod/cluster-backup -n openshift-etcd --watch -o wide

NAME             READY   STATUS   RESTARTS   AGE   IP             NODE                     NOMINATED NODE   READINESS GATES
cluster-backup   0/1     Error    0          10s   1.2.3.20   ocp-abcde-master-1   <none>           <none>
cluster-backup   0/1     Terminating   0          56s   1.2.3.20   ocp-abcde-master-1   <none>           <none>
cluster-backup   0/1     Terminating   0          56s   1.2.3.20   ocp-abcde-master-1   <none>           <none>

cluster-backup   0/1     Pending       0          0s    <none>         ocp-abcde-master-0   <none>           <none>
cluster-backup   0/1     Init:0/1      0          0s    1.2.3.10   ocp-abcde-master-0   <none>           <none>
cluster-backup   0/1     Init:0/1      0          2s    1.2.3.10   ocp-abcde-master-0   <none>           <none>
cluster-backup   0/1     PodInitializing   0          3s    1.2.3.10   ocp-abcde-master-0   <none>           <none>
cluster-backup   1/1     Running           0          4s    1.2.3.10   ocp-abcde-master-0   <none>           <none>
cluster-backup   0/1     Completed         0          11s   1.2.3.10   ocp-abcde-master-0   <none>           <none>
cluster-backup   0/1     Completed         0          13s   1.2.3.10   ocp-abcde-master-0   <none>           <none>

Wait for status 'Completed'.

12 )
Check that update started to progress:
oc get clusterversion

13.master)
Replace modified rm with rm.orig:

cd /var/lib/containers/storage/overlay/l/N5A3GZG4I37DW4DPGS5L3GELSC/usr/bin/
mv rm.orig rm

That's it, my cluster was successfully updated to 4.15, perhaps someone updating old cluster like me finds this useful..
Regards,o.