4.18-4.19 update stuck at crd consoleplugins.console.openshift.io

[Link1J]

Describe the bug

Update to 4.19.0-okd-scos.0 from 4.18.0-okd-scos.9, with error message Could not update customresourcedefinition "consoleplugins.console.openshift.io" (595 of 917).

The logs for cluster-version-operator have the following in it.

I0518 20:57:37.285445       1 sync_worker.go:1041] Running sync for customresourcedefinition "consoleplugins.console.openshift.io" (595 of 917)
E0518 20:57:37.389133       1 task.go:128] "Unhandled Error" err="error running apply for customresourcedefinition \"consoleplugins.console.openshift.io\" (595 of 917): CustomResourceDefinition.apiextensions.k8s.io \"consoleplugins.console.openshift.io\" is invalid: status.storedVersions[0]: Invalid value: \"v1alpha1\": must appear in spec.versions" logger="UnhandledError"
E0518 20:57:47.419632       1 task.go:128] "Unhandled Error" err="error running apply for customresourcedefinition \"consoleplugins.console.openshift.io\" (595 of 917): CustomResourceDefinition.apiextensions.k8s.io \"consoleplugins.console.openshift.io\" is invalid: status.storedVersions[0]: Invalid value: \"v1alpha1\": must appear in spec.versions" logger="UnhandledError"

This issue seems to have been found by Redhat (here). My cluster started at 4.13.0-0.okd-2023-07-09-062029 which is earlier then what was reported in that issue.

Version

4.18.0-okd-scos.9

Reproducibility

Unsure (Seems perfectly reproducible)

Log Bundle

Unable to collect because of errors running oc adm must-gather. Will upload if it becomes possible to collect.

[niktsl]

A workaround to continue with the upgrade

1. Pause the update

2. Check that served and storage are both false
   kubectl get crd consoleplugins.console.openshift.io -o jsonpath='{range .spec.versions[?(@.name=="v1alpha1")]}served: {.served} storage: {.storage}{end}'

3. Lists all existing resources using the v1alpha1 version of the consoleplugins CRD across all namespaces.
   oc get consoleplugins.v1alpha1.console.openshift.io --all-namespaces

If served and storage are false and no existing resources are using v1alpha1, you can safely proceed with the following steps.

4. Export the CRD: kubectl get crd consoleplugins.console.openshift.io -o yaml > crd-full.yaml

5. Edit: crd-full.yaml -> remove v1alpha1 and status

6. Force replace the CRD : kubectl replace --force -f crd-full.yaml

7. Resume upgrade