Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule #9452

jrafanie · 2025-05-23T17:42:09Z

Upgrade cypress to 14.4
Run cypress via chrome only on pull requests

Allow full cypress runs with all browsers on:
- adhoc workflow_dispatch
- pushes
- schedule

TODO list:

Disable './miq_debug.js' in dev mode for cypress - currently blocked trying to detect if we're running under cypress via environment variable.
~~Consider dropping rate limit / rack attack entirely for cypress or change the configuration for cypress 'mode' so we don't need to do the hack in the UI rake task~~ We will handle this as a followup
~~should we extract the 14.4 upgrade out of this PR?~~ we will do it here

Fryguy · 2025-05-27T15:44:37Z

.github/workflows/cypress.yaml

+        #- firefox
+        #- edge


I know this PR is WIP, but I think we can keep these in place, and use conditionals like if: ${{ matrix.cypress-browser == 'chrome' && github.event_name == 'push' }} or something similar to enable it only in certain cases.

yeah, just trying to figure out how to get the flakiness out... it's 2009 all over again

Ok, pushed some changes... I can't access matrix early on to skip the whole workflow so I had to skip each step. To avoid having that logic copy / pasted everywhere, I added a step to contain the logic and reuse it later on.

Fryguy · 2025-05-28T13:06:06Z

Rakefile

-      exit $?.exitstatus unless system("bundle exec rails runner 'MiqServer.my_server.add_settings_for_resource(:server => {:rate_limiting => {:request => {:limit => 99999}}})'")
+      exit $?.exitstatus unless system("bundle exec rails runner 'MiqServer.my_server.add_settings_for_resource(:server => {:rate_limiting => {:request => {:limit => 99999}, :ui_login => {:limit => 99999}}})'")


Alternatively, maybe we can encode this into the rate limiter itself here with the numbers or here with the conditional in the first place? I never liked this line because it changes settings.

Good idea. I threw this in to try to get green but you bring up a good point. Is there any reason in development to have rack attack configured? Clearly, we can do an exception for cypress but I wonder what's the benefit to having it in dev mode.

Is there any reason to development has rack attack configured?

That's actually a great question. The only reason I can think of is since we disable in testing, then there's nothing else running through this code otherwise. If we get an upgrade to the gem that breaks something it won't be caught in dev or test otherwise (but probably would be caught in PEN test). Maybe the security suite should be updated to ensure it's working as intended?

.github/workflows/cypress.yaml

Fryguy · 2025-05-28T20:19:35Z

.github/workflows/cypress.yaml

    - name: Set up system
      run: bin/before_install
+      if: ${{ matrix.cypress-browser == 'chrome' || (matrix.cypress-browser != 'chrome' && github.event_name != 'pull_request') }}


I know this is WIP, so future cleanup idea.

I'm thinking let's shove this into an ENV var with a sensible name, then use that later. Either that or run an initial step that sets an output to the result and use that. So for env idea, something like

env: TEST_SUITE: spec:cypress RUN_CYPRESS: ${{ matrix.cypress-browser == 'chrome' || (matrix.cypress-browser != 'chrome' && github.event_name != 'pull_request') }} ... steps: ... - name: Set up system run: bin/before_install if: ${{ env.RUN_CYPRESS }}

LOL you read my mind.

Though I think env might be cleaner and I think it can support booleans

I like the name. I think we might want pushes, manual / adhoc, and scheduled runs to be full browser support. Either way, we can tweak it once we have better ideas.

Allow full cypress runs with all browsers on: * adhoc workflow_dispatch * pushes * schedule Currently, firefox is very slow, over 40 minutes, while chrome and edge are under 20 minutes each. For now, let's get chrome to verify PRs while we upgrade cypress and try to improve the speed further.

With this configuration, we expect each test to pass 1 out of 3 tries. If it does, it passes. We can track which tests often are flaky by watching for retries, while also keeping the test suite green.

jrafanie · 2025-05-29T14:26:24Z

app/javascript/oldjs/application.js

-if (process.env.NODE_ENV === 'development' && process.env.CYPRESS !== 'true') {
-  require('./miq_debug.js');
-  require('./miq_debug.css');
-}


TODO: this conditional doesn't work correctly as CYPRESS env variable is never defined. I tried REACT_APP_CYPRESS and other things such as trying to push it into webpack but nothing worked so for now, I have this removed in this PR as the notifications cause far too many sporadic test failures.

Fryguy · 2025-06-03T17:30:31Z

@jrafanie A conflict occurred during the backport of this pull request to spassky.

If this pull request is based on another pull request that has not been marked for backport, add the appropriate labels to the other pull request. Otherwise, please create a new pull request direct to the spassky branch in order to resolve this.

Conflict details:

diff --cc yarn.lock
index 033be2987d,fba156267c..0000000000
--- a/yarn.lock
+++ b/yarn.lock
@@@ -15837,12 -15837,12 +15837,18 @@@ __metadata
    languageName: node
    linkType: hard
  
++<<<<<<< HEAD
 +"semver@npm:^7.2.1, semver@npm:^7.3.4, semver@npm:^7.3.5, semver@npm:^7.5.3":
 +  version: 7.7.1
 +  resolution: "semver@npm:7.7.1"
++=======
+ "semver@npm:^7.2.1, semver@npm:^7.3.4, semver@npm:^7.3.5, semver@npm:^7.7.1":
+   version: 7.7.2
+   resolution: "semver@npm:7.7.2"
++>>>>>>> f2354776cc (Merge pull request #9452 from jrafanie/enable-cypress-on-push-pr-with-chrome-only)
    bin:
      semver: bin/semver.js
 -  checksum: 10/7a24cffcaa13f53c09ce55e05efe25cd41328730b2308678624f8b9f5fc3093fc4d189f47950f0b811ff8f3c3039c24a2c36717ba7961615c682045bf03e1dda
 +  checksum: 10/4cfa1eb91ef3751e20fc52e47a935a0118d56d6f15a837ab814da0c150778ba2ca4f1a4d9068b33070ea4273629e615066664c2cfcd7c272caf7a8a0f6518b2c
    languageName: node
    linkType: hard

Fryguy · 2025-06-03T17:33:14Z

Backported to spassky in commit 93cc8c7.

commit 93cc8c71de75fdf15db8f6b57bc34700be3f1b88
Author: Jason Frey <[email protected]>
Date:   Thu May 29 11:24:57 2025 -0400

    Merge pull request #9452 from jrafanie/enable-cypress-on-push-pr-with-chrome-only
    
    Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule
    
    (cherry picked from commit f2354776cc8e8b91090399336604709c1c6f3186)

…-chrome-only Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule (cherry picked from commit f235477)

jrafanie · 2025-06-03T17:56:01Z

@Fryguy did you manually fix the conflict? Is there anything more to do here? I can test it later.

Fryguy · 2025-06-03T17:58:37Z

I figured out that we had to backport #9434 first

jrafanie requested a review from a team as a code owner May 23, 2025 17:42

jrafanie added the cypress label May 23, 2025

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch 2 times, most recently from 43fce3b to 8da6be6 Compare May 23, 2025 22:11

This was referenced May 27, 2025

Fixed dashboard cypress tests #8892

Merged

Fix report cypress tests #9454

Merged

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch from 8da6be6 to f442479 Compare May 27, 2025 13:44

Fryguy reviewed May 27, 2025

View reviewed changes

miq-bot added wip dependencies labels May 27, 2025

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch from 47a81f6 to 2bf2dfd Compare May 27, 2025 21:35

Fryguy reviewed May 28, 2025

View reviewed changes

Fryguy self-assigned this May 28, 2025

Fryguy reviewed May 28, 2025

View reviewed changes

.github/workflows/cypress.yaml Outdated Show resolved Hide resolved

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch 2 times, most recently from af96e1a to dd3fd8a Compare May 28, 2025 16:20

This was referenced May 28, 2025

Bump node and ruby versions for cypress #9457

Merged

Remove incorrect and unneeded assertions #9458

Merged

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch 5 times, most recently from f131e58 to c1ebaf0 Compare May 28, 2025 20:02

Fryguy reviewed May 28, 2025

View reviewed changes

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch 3 times, most recently from d3bebb0 to 1318ebb Compare May 28, 2025 22:16

jrafanie added 2 commits May 29, 2025 09:55

Update to cypress 14.4.0

2d31782

jrafanie added 2 commits May 29, 2025 09:56

Remove UI login request limit for cypress

ba751a2

Run with retry support to deal with flaky tests

c497b67

With this configuration, we expect each test to pass 1 out of 3 tries. If it does, it passes. We can track which tests often are flaky by watching for retries, while also keeping the test suite green.

jrafanie force-pushed the enable-cypress-on-push-pr-with-chrome-only branch from 471cbf9 to 694c8ef Compare May 29, 2025 13:57

jrafanie commented May 29, 2025

View reviewed changes

Fix disabling of debug notifications for cypress tests

995315c

Fryguy force-pushed the enable-cypress-on-push-pr-with-chrome-only branch from 694c8ef to 995315c Compare May 29, 2025 15:02

jrafanie changed the title ~~[WIP] Enable cypress on push pr with chrome only~~ Enable cypress on push pr with chrome only May 29, 2025

jrafanie added spassky/yes and removed wip labels May 29, 2025

jrafanie changed the title ~~Enable cypress on push pr with chrome only~~ Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule May 29, 2025

Fryguy approved these changes May 29, 2025

View reviewed changes

Fryguy merged commit f235477 into ManageIQ:master May 29, 2025
18 checks passed

jrafanie deleted the enable-cypress-on-push-pr-with-chrome-only branch May 29, 2025 18:37

Fryguy added the spassky/conflict label Jun 3, 2025

Fryguy added spassky/backported and removed spassky/yes spassky/conflict labels Jun 3, 2025

		exit $?.exitstatus unless system("bundle exec rails runner 'MiqServer.my_server.add_settings_for_resource(:server => {:rate_limiting => {:request => {:limit => 99999}}})'")
		exit $?.exitstatus unless system("bundle exec rails runner 'MiqServer.my_server.add_settings_for_resource(:server => {:rate_limiting => {:request => {:limit => 99999}, :ui_login => {:limit => 99999}}})'")

Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule #9452

Upgrade cypress to 14.4 and enable it on for PRs with chrome only, for all browsers on manual, push, schedule #9452

Conversation

jrafanie commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrafanie May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fryguy May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fryguy commented Jun 3, 2025

Uh oh!

Fryguy commented Jun 3, 2025

Uh oh!

jrafanie commented Jun 3, 2025

Uh oh!

Fryguy commented Jun 3, 2025

Uh oh!

Uh oh!

jrafanie commented May 23, 2025 •

edited

Loading

jrafanie May 28, 2025 •

edited

Loading

Fryguy May 28, 2025 •

edited

Loading