NO-JIRA: We should always run tests where we have no opinion on their gate

[JoelSpeed]

When a test is added, that contains [OCPFeatureGate:, we expect that the gate already exists in openshift/api and we therefore have an opinion about the gate as to whether it is enabled or disabled.

In all cases, we render every known gate as either enabled or disabled, so every gate has an opinion.

However, when we remove gates, we no longer have an opinion, and the current code disables the tests.

Generally, removing the gate means the feature was promoted to default in a previous release, and therefore the tests should continue to run.

In some cases, the gate may be removed without ever graduating. This is rare, and in these cases we would start running their tests with the feature removed, which would fail, and we would find out pretty quickly.

This seems like the lesser of two evils, the alternative is we just stop running tests and are oblivious to this

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JoelSpeed
Once this PR has been reviewed and has the lgtm label, please assign dgoodwin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@JoelSpeed: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn-upgrade	89e5a3a	link	false	/test e2e-azure-ovn-upgrade
ci/prow/e2e-gcp-csi	89e5a3a	link	false	/test e2e-gcp-csi
ci/prow/e2e-aws-ovn-kube-apiserver-rollout	89e5a3a	link	false	/test e2e-aws-ovn-kube-apiserver-rollout
ci/prow/e2e-azure-ovn-etcd-scaling	89e5a3a	link	false	/test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-edge-zones	89e5a3a	link	true	/test e2e-aws-ovn-edge-zones
ci/prow/e2e-metal-ipi-virtualmedia	89e5a3a	link	false	/test e2e-metal-ipi-virtualmedia
ci/prow/e2e-metal-ipi-ovn-dualstack-local-gateway	89e5a3a	link	false	/test e2e-metal-ipi-ovn-dualstack-local-gateway
ci/prow/e2e-vsphere-ovn-etcd-scaling	89e5a3a	link	false	/test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-azure	89e5a3a	link	false	/test e2e-azure
ci/prow/e2e-aws-ovn-upgrade	89e5a3a	link	false	/test e2e-aws-ovn-upgrade
ci/prow/e2e-metal-ipi-ovn-dualstack	89e5a3a	link	false	/test e2e-metal-ipi-ovn-dualstack
ci/prow/e2e-aws-ovn	89e5a3a	link	false	/test e2e-aws-ovn
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	89e5a3a	link	false	/test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/e2e-aws-ovn-single-node-serial	89e5a3a	link	false	/test e2e-aws-ovn-single-node-serial
ci/prow/verify-deps	89e5a3a	link	true	/test verify-deps
ci/prow/4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback	89e5a3a	link	false	/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
ci/prow/e2e-aws-disruptive	89e5a3a	link	false	/test e2e-aws-disruptive
ci/prow/e2e-aws-ovn-etcd-scaling	89e5a3a	link	false	/test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-serial-publicnet-1of2	89e5a3a	link	false	/test e2e-aws-ovn-serial-publicnet-1of2
ci/prow/e2e-aws-ovn-single-node-upgrade	89e5a3a	link	false	/test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@JoelSpeed: This pull request explicitly references no jira issue.

In response to this:

When a test is added, that contains [OCPFeatureGate:, we expect that the gate already exists in openshift/api and we therefore have an opinion about the gate as to whether it is enabled or disabled.

In all cases, we render every known gate as either enabled or disabled, so every gate has an opinion.

However, when we remove gates, we no longer have an opinion, and the current code disables the tests.

Generally, removing the gate means the feature was promoted to default in a previous release, and therefore the tests should continue to run.

In some cases, the gate may be removed without ever graduating. This is rare, and in these cases we would start running their tests with the feature removed, which would fail, and we would find out pretty quickly.

This seems like the lesser of two evils, the alternative is we just stop running tests and are oblivious to this

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.