RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks #834

andyatmiami · 2025-01-12T19:07:57Z

Description

Our testing framework had a fundamental issue in that it would, for certain images, naively run the test suite of parent images for a child image. In the case of tensorflow images, this caused problems because the (child) tensorflow image would purposefully downgraded a package version given compatability issues. Furthermore, when attempting to verify the changes to address this core issue, numerous other bugs were encountered that required to be fixed.

Significant changes introduced in this commit:

logic of the test-% Makefile target was moved into a shell script named test_jupyter_with_papermill.sh
test resources required to be copied to pod are now retrieved via the locally checked out repo
- previously there were pulled from a remote branch via wget (defaulting to main branch)
- this ensure our PR checks are now always leveraging any updated files
test_notebook.ipynb files now expect an expected_versions.json file to exist within the same directory.
- expected_versions.json file is derived from the relevant ...-notebook-imagestream.yaml manifest and leveraged when asserting on dependency versions
- admittedly the duplication of various helper functions across all the notebook files is annoying - but helps to keep the .ipynb files self-contained
...-notebook-imagestream.yaml manifest had annotations updated to include any package that had a unit test in test_notebook.ipynb asserting versions
CUDA tensorflow unit test that converts to ONNX was updated to be actually functional

Minor changes introduced in this commit:

use ifdef OS in Makefile to avoid warnings about undefined variable
all test_notebook.ipynb files:
- have an id attribute defined in metadata
- specify the nbformat as 4.5
the more compute-intensive notebooks had readinessProbe and livenessProbe settings updated to be less aggressive
- was observing liveness checks sporadically failing while the notebook was being tested - and this update seemed to help
trustyai notebook now runs the minimal and datascience papermill tests (similar to tensorflow and pytorch) instead of including the test code within its own test_noteook.ipynb file
various "quality of life" improvements where introduced into test_jupyter_with_papermill.sh
- properly invoke tests for any valid/supported target
  - previously certain test targets required manual intervention in order to run end to end
- improved logging (particularly when running with the -x flag)
- more modular structure to hopefully improve readability
- script now honors proper shell return code semantics (i.e. returns 0 on success)

It should also be noted that while most intel notebooks are now passing the papermill tests - there are issues with the intel tensorflow unit tests still. More detail is captured in the JIRA ticket related to this commit. However, we are imminently removing intel logic from the notebooks repo entirely... so I didn't wanna burn any more time trying to get that last notebook to pass as it will be removed shortly!

Further details on expected_versions.json:

yq (which is installed via the Makefile is used to:
- query the relevant imagestream manifest to parse out the ["opendatahub.io/notebook-software"] and ["opendatahub.io/notebook-python-dependencies"] annotations from the first element of the spec.tags attribute
- inject name/version elements for nbdime and nbgitpuller (which are asserted against in minimal notebook tests)
- convert this yaml list to a JSON object of the form: {<package>: <version>}
this JSON object is then copied into the running notebook workload in the same directly that test_notebook.ipynb resides
each test_notebook.ipynb has a couple helper functions defined to then interact with this file:
- def load_expected_versions() -> dict:
- def get_expected_version(dependency_name: str) -> str:
  - The argument provided to the get_expected_version function should match the name attribute of the JSON structure defined in the imagestream manifest

Related-to: https://issues.redhat.com/browse/RHOAIENG-16587

How Has This Been Tested?

Testing can be performed simply by invoking make with a target that matches the test-% pattern-matching rule.

ex: gmake test-jupyter-pytorch-ubi9-python-3.11 -e NOTEBOOK_REPO_BRANCH_BASE=https://raw.githubusercontent.com/andyatmiami/notebooks/fix-e2e-asserts for basic sanity checking. However, all known/supported/expected test-% patterns should be verified.

I have also created a utility in https://gitlab.cee.redhat.com/astonebe/notebook-utils that can be used to more efficiently test notebooks "in bulk". While it still runs sequentially, it handles deploying, testing, and undeploying a given list of targets.

I used this utility script to test all notebooks that have a corresponding imagestream manifest.

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

openshift-ci · 2025-01-12T19:08:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jiridanek · 2025-01-12T19:51:29Z

Alternative proposal

The package versions we should have in our images are the ones given in the docs, which is at https://access.redhat.com/articles/rhoai-supported-configs (section "Supported workbench images"). This table is generated (or written manually, before red-hat-data-services#476), based on our ImageStream manifests at https://github.com/opendatahub-io/notebooks/tree/main/manifests/base (and https://github.com/red-hat-data-services/notebooks/tree/main/manifests/base for RHOAI)

notebooks/manifests/base/jupyter-rocm-pytorch-notebook-imagestream.yaml

Lines 27 to 33 in 8a24d01

    
                   opendatahub.io/notebook-python-dependencies: | 
        
                     [ 
        
                       {"name": "ROCm-PyTorch", "version": "2.4"}, 
        
                       {"name": "Tensorboard", "version": "2.16"}, 
        
                       {"name": "Kafka-Python-ng", "version": "2.2"}, 
        
                       {"name": "Matplotlib", "version": "3.9"}, 
        
                       {"name": "Numpy", "version": "2.1"},

We should not duplicate that information in the test ipython notebook files. The tests should check that what's in the image corresponds to what's given in the manifests.

andyatmiami · 2025-01-12T22:15:31Z

So, given today in our tests we are duplicating this information...

do you want me to address your suggestion in this PR or a follow up PR?

For emphasis - I wholly agree with your proposal - I've just scope-creeped the heck outta this present PR already 😎

jiridanek · 2025-01-13T05:44:10Z

do you want me to address your suggestion in this PR or a follow up PR?

We did not do enough PRs of this kind for me to discern a pattern, but on my previous team, there was preference for moving in small steps, e.g. first move script embedded in Makefile to it's own file unchanged, then modify said script.

Guess this is largely a mater of personal taste. And also of review speed, it's hard to split things up when reviews for the parts are not forthcoming.

andyatmiami · 2025-01-14T17:31:23Z

/test all

jstourac · 2025-01-15T12:53:49Z

I got to this for a brief review. I agree with Jiri to re-use what we define in our manifest files in imagestreams for the testing purposes instead of duplicating - but yeah, let's do it step by step as you wrote above. So, I am happy to see these tests being extracted from the Makefile making it a bit more readable. What I'm not a big fan here is that we're creating yet another bash script for this. I think we should utilize the existing pytest framework with the testcontainers for as much tests as possible so that we can run these in a unified environment.

Anyway, since this aims only to extract these out of Makefile and to keep this compatible with the existing approach, I'm not against this in general right now. But to the future, I think we should think to migrate this again - this time to the pytest where most of our tests will be, hopefully.

jiridanek · 2025-01-15T12:55:39Z

I'm with Jan, pulling the bash code out of Makefile is a good improvement in itself.

andyatmiami · 2025-01-17T22:32:37Z

/test notebooks-ubi9-e2e-tests

(weird GCP errors on building the cluster - hopefully transient 🤞 )

andyatmiami · 2025-01-17T23:17:22Z

PASSING TESTS!

andyatmiami · 2025-01-17T23:17:52Z

/test notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

jupyter/intel/ml/ubi9-python-3.11/kustomize/base/statefulset.yaml

scripts/test_jupyter_with_papermill.sh

andyatmiami · 2025-01-19T21:26:42Z

/test notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

jiridanek · 2025-01-20T07:55:02Z

.gitignore

@@ -128,6 +128,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+.DS_store


jiridanek · 2025-01-20T09:36:51Z

/lgtm

this is a huge improvement over what we had before, so yeah, I'm happy to have it merged

…notebooks Our testing framework had a fundamental issue in that it would, for certain images, naively run the test suite of parent images for a child image. In the case of `tensorflow` images, this caused problems because the (child) `tensorflow` image would **purposefully** _downgraded_ a package version given compatability issues. Furthermore, when attempting to verify the changes to address this core issue, numerous other bugs were encountered that required to be fixed. Significant changes introduced in this commit: - logic of the `test-%` Makefile target was moved into a shell script named `test_jupyter_with_papermill.sh` - test resources required to be copied to pod are now retrieved via the locally checked out repo - previously there were pulled from a remote branch via `wget` (defaulting to `main` branch) - this ensure our PR checks are now always leveraging any updated files - test_notebook.ipynb files now expect an `expected_versions.json` file to exist within the same directory. - `expected_versions.json` file is derived from the relevant `...-notebook-imagestream.yaml` manifest and leveraged when asserting on dependency versions - admittedly the duplication of various helper functions across all the notebook files is annoying - but helps to keep the `.ipynb` files self-contained - `...-notebook-imagestream.yaml` manifest had annotations updated to include any package that had a unit test in `test_notebook.ipynb` asserting versions - CUDA tensorflow unit test that converts to ONNX was updated to be actually functional Minor changes introduced in this commit: - use `ifdef OS` in Makefile to avoid warnings about undefined variable - all `test_notebook.ipynb` files: - have an `id` attribute defined in metadata - specify the `nbformat` as `4.5` - the more compute-intensive notebooks had `readinessProbe` and `livenessProbe` settings updated to be less aggressive - was observing liveness checks sporadically failing while the notebook was being tested - and this update seemed to help - `trustyai` notebook now runs the minimal and datascience papermill tests (similar to `tensorflow` and `pytorch`) instead of including the test code within its own `test_noteook.ipynb` file - various "quality of life" improvements where introduced into `test_jupyter_with_papermill.sh` - properly invoke tests for any valid/supported target - previously certain test targets required manual intervention in order to run end to end - improved logging (particularly when running with the `-x` flag) - more modular structure to hopefully improve readability - script now honors proper shell return code semantics (i.e. returns `0` on success) It should also be noted that while most `intel` notebooks are now passing the papermill tests - there are issues with the `intel` `tensorflow` unit tests still. More detail is captured in the JIRA ticket related to this commit. However, we are imminently removing `intel` logic from the `notebooks` repo entirely... so I didn't wanna burn any more time trying to get that last notebook to pass as it will be removed shortly! Further details on `expected_versions.json`: - `yq` (which is installed via the `Makefile` is used to: - query the relevant imagestream manifest to parse out the `["opendatahub.io/notebook-software"]` and `["opendatahub.io/notebook-python-dependencies"]` annotations from the first element of the `spec.tags` attribute - inject name/version elements for `nbdime` and `nbgitpuller` (which are asserted against in `minimal` notebook tests) - convert this `yaml` list to a JSON object of the form: `{<package>: <version>}` - this JSON object is then copied into the running notebook workload in the same directly that `test_notebook.ipynb` resides - each `test_notebook.ipynb` has a couple helper functions defined to then interact with this file: - `def load_expected_versions() -> dict:` - `def get_expected_version(dependency_name: str) -> str: - The argument provided to the `get_expected_version` function should match the `name` attribute of the JSON structure defined in the imagestream manifest Related-to: https://issues.redhat.com/browse/RHOAIENG-16587

jiridanek · 2025-01-20T15:47:00Z

/lgtm

atheo89 · 2025-01-20T15:57:00Z

/lgtm

andyatmiami · 2025-01-20T22:22:10Z

/override ci/prow/intel-notebooks-e2e-tests ci/prow/notebooks-ubi9-e2e-tests ci/prow/rocm-notebooks-e2e-tests

openshift-ci · 2025-01-20T22:22:19Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/intel-notebooks-e2e-tests, ci/prow/notebooks-ubi9-e2e-tests, ci/prow/rocm-notebooks-e2e-tests

In response to this:

/override ci/prow/intel-notebooks-e2e-tests ci/prow/notebooks-ubi9-e2e-tests ci/prow/rocm-notebooks-e2e-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:24:07Z

/override ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

openshift-ci · 2025-01-20T22:24:13Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:25:52Z

/override ci/prow/notebook-jupyter-intel-tf-ubi9-python-3-11-pr-image-mirror

(same old quay 502 issues and retries not handling it)

error: unable to copy layer sha256:45d40005ddd1bccd29a7b1abc2a88c08732ab10f9904c997cc0c322f77319bce to quay.io/opendatahub/workbench-images: received unexpected HTTP status: 502 Bad Gateway
info: Mirroring completed in 33.41s (4.343MB/s)
error: one or more errors occurred while uploading images
2025-01-20T17:08:23+00:00 ERROR Unable to mirror image
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2025-01-20T17:08:23Z"}
error: failed to execute wrapped command: exit status 1

openshift-ci · 2025-01-20T22:25:58Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/notebook-jupyter-intel-tf-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-jupyter-intel-tf-ubi9-python-3-11-pr-image-mirror

(same old quay 502 issues and retries not handling it)

error: unable to copy layer sha256:45d40005ddd1bccd29a7b1abc2a88c08732ab10f9904c997cc0c322f77319bce to quay.io/opendatahub/workbench-images: received unexpected HTTP status: 502 Bad Gateway
info: Mirroring completed in 33.41s (4.343MB/s)
error: one or more errors occurred while uploading images
2025-01-20T17:08:23+00:00 ERROR Unable to mirror image
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2025-01-20T17:08:23Z"}
error: failed to execute wrapped command: exit status 1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:26:49Z

/override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

openshift-ci · 2025-01-20T22:26:55Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:27:29Z

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

openshift-ci · 2025-01-20T22:27:35Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:28:10Z

/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

openshift-ci · 2025-01-20T22:28:15Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror

(same old build timeout BS.. i'm working on getting to the bottom of that!)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:28:44Z

/override ci/prow/images

(same old build timeout BS.. i'm working on getting to the bottom of that!)

openshift-ci · 2025-01-20T22:28:49Z

@andyatmiami: Overrode contexts on behalf of andyatmiami: ci/prow/images

In response to this:

/override ci/prow/images

(same old build timeout BS.. i'm working on getting to the bottom of that!)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andyatmiami · 2025-01-20T22:28:59Z

/approve

openshift-ci · 2025-01-20T22:29:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyatmiami

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andyatmiami]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks

…lux/component-updates/component-update-odh-workbench-jupyter-pytorch-rocm-py311-ubi9-n-v2-22 Update odh-workbench-jupyter-pytorch-rocm-py311-ubi9-n-v2-22 to 181b0a5

openshift-ci bot added the do-not-merge/work-in-progress label Jan 12, 2025

andyatmiami force-pushed the fix-e2e-asserts branch from b4be87f to f69e4f5 Compare January 13, 2025 15:32

andyatmiami mentioned this pull request Jan 17, 2025

RHOAIENG-8388: rm(intel): Intel tensorflow notebook failed to get tested on OCP-CI so broken intel notebooks should be deleted #843

Merged

3 tasks

andyatmiami force-pushed the fix-e2e-asserts branch 2 times, most recently from ee21ef4 to 5dd5618 Compare January 17, 2025 19:51

andyatmiami changed the title ~~fix(test): provide ability to override versions used in asserts during papermill tests~~ fix(test): ensure papermill tests run successfully for all supported notebooks Jan 17, 2025

andyatmiami changed the title ~~fix(test): ensure papermill tests run successfully for all supported notebooks~~ RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks Jan 17, 2025

andyatmiami force-pushed the fix-e2e-asserts branch from 5dd5618 to 7388f35 Compare January 17, 2025 19:57

andyatmiami marked this pull request as ready for review January 17, 2025 19:57

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 17, 2025

openshift-ci bot requested review from dibryant and jiridanek January 17, 2025 19:58

opendatahub-io deleted a comment from openshift-ci bot Jan 17, 2025

jiridanek reviewed Jan 18, 2025

View reviewed changes

jupyter/intel/ml/ubi9-python-3.11/kustomize/base/statefulset.yaml Show resolved Hide resolved

jiridanek reviewed Jan 18, 2025

View reviewed changes

scripts/test_jupyter_with_papermill.sh Outdated Show resolved Hide resolved

andyatmiami force-pushed the fix-e2e-asserts branch from 7388f35 to 325c5df Compare January 19, 2025 19:34

jiridanek reviewed Jan 20, 2025

View reviewed changes

.gitignore

@@ -128,6 +128,7 @@ venv/

ENV/

env.bak/

venv.bak/

.DS_store

Copy link

Member

jiridanek Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍎

openshift-ci bot assigned jstourac Jan 20, 2025

andyatmiami force-pushed the fix-e2e-asserts branch from 325c5df to 3924fe7 Compare January 20, 2025 15:43

openshift-ci bot removed the lgtm label Jan 20, 2025

openshift-ci bot added the lgtm label Jan 20, 2025

openshift-ci bot assigned atheo89 Jan 20, 2025

opendatahub-io deleted a comment from openshift-ci bot Jan 20, 2025

openshift-ci bot added the approved label Jan 20, 2025

openshift-merge-bot bot merged commit 5c34753 into opendatahub-io:main Jan 20, 2025
29 checks passed

jiridanek pushed a commit to rhoai-ide-konflux/notebooks that referenced this pull request Mar 30, 2025

Merge pull request opendatahub-io#834 from andyatmiami/fix-e2e-asserts

a5a971f

RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks

atheo89 mentioned this pull request Jun 24, 2025

RHOAIENG-26921: Create Jupyter Tensorflow Python 3.12 Image #1151

Open

3 tasks

RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks #834

RHOAIENG-16587: fix(test): ensure papermill tests run successfully for all supported notebooks #834

Uh oh!

Conversation

andyatmiami commented Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Merge criteria:

Uh oh!

openshift-ci bot commented Jan 12, 2025

Uh oh!

jiridanek commented Jan 12, 2025

Alternative proposal

Uh oh!

andyatmiami commented Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiridanek commented Jan 13, 2025

Uh oh!

andyatmiami commented Jan 14, 2025

Uh oh!

jstourac commented Jan 15, 2025

Uh oh!

jiridanek commented Jan 15, 2025

Uh oh!

andyatmiami commented Jan 17, 2025

Uh oh!

andyatmiami commented Jan 17, 2025

Uh oh!

andyatmiami commented Jan 17, 2025

Uh oh!

Uh oh!

Uh oh!

andyatmiami commented Jan 19, 2025

Uh oh!

jiridanek Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

jiridanek commented Jan 20, 2025

Uh oh!

jiridanek commented Jan 20, 2025

Uh oh!

atheo89 commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

andyatmiami commented Jan 20, 2025

Uh oh!

openshift-ci bot commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

andyatmiami commented Jan 12, 2025 •

edited

Loading

andyatmiami commented Jan 12, 2025 •

edited

Loading