fix: remove experimental from forked repo tests (#4088)

* feat: remove experimental from forked repo tests

* mark packages to run tests

* mark old tests as legacy

* add more comments

* add instructions to sunset old tests

* rename workflow back

* remove experimental from created checks

* remove packages to trigger tests

* experiment to see if check fulfills required tests

* add permissions to experiment

* rename lint back

* remove experiment

* only run tests on non-forks

* remove experimental

* document ci testing

* add legacy to old dev tests

* rename summary to Custard CI / test

* reorder list

* Update CONTRIBUTING.md

Co-authored-by: Katie McLaughlin <[email protected]>

* add note to pull request template

---------

Co-authored-by: Katie McLaughlin <[email protected]>

Author: davidcavazos

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,10 +8,13 @@ Note: Before submitting a pull request, please open an issue for discussion if y
 - [ ] I have followed guidelines from [CONTRIBUTING.MD](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md) and [Samples Style Guide](https://googlecloudplatform.github.io/samples-style-guide/)
 - [ ] **Tests** pass:   `npm test` (see [Testing](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md#run-the-tests-for-a-single-sample))
 - [ ] **Lint** pass:   `npm run lint` (see [Style](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md#style))
+- [ ] **Required CI tests** pass (see [CI testing](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md#ci-testing))
 - [ ] These samples need a new **API enabled** in testing projects to pass (let us know which ones)
 - [ ] These samples need a new/updated **env vars** in testing projects set to pass (let us know which ones)
 - [ ] This pull request is from a branch created directly off of `GoogleCloudPlatform/nodejs-docs-samples`. Not a fork.
 - [ ] This sample adds a new sample directory, and I updated the [CODEOWNERS file](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CODEOWNERS) with the codeowners for this sample
 - [ ] This sample adds a new sample directory, and I created [GitHub Actions workflow](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md#adding-new-samples) for this sample
 - [ ] This sample adds a new **Product API**, and I updated the [Blunderbuss issue/PR auto-assigner](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/.github/blunderbuss.yml) with the codeowners for this sample
 - [ ] Please **merge** this PR for me once it is approved
+
+> **Note**: Any check with `(dev)`, `(experimental)`, or `(legacy)` can be ignored and should **not block** your PR from merging (see [CI testing](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/CONTRIBUTING.md#ci-testing)).

.github/workflows/custard-ci-dev.yaml

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-name: Custard CI (dev)
+name: (legacy dev) Custard CI
 on:
   push:
     branches:
@@ -27,7 +27,7 @@ env:
 
 jobs:
   affected:
-    name: Finding affected tests
+    name: (legacy) Finding affected tests
     runs-on: ubuntu-latest
     timeout-minutes: 2
     outputs:
@@ -61,7 +61,8 @@ jobs:
           echo "setups=$(./cloud-samples-tools/bin/custard setup-files .github/config/nodejs-dev.jsonc paths.txt)" >> $GITHUB_OUTPUT
 
   nodejs-test:
-    name: test
+    if: github.event.pull_request && github.event.pull_request.head.repo.fork == false
+    name: (legacy) test
     needs: affected
     runs-on: ubuntu-latest
     timeout-minutes: 120 # 2 hours hard limit

.github/workflows/custard-ci.yaml

@@ -12,6 +12,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# IMPORTANT: DO NOT CHANGE THE NAME OF THIS WORKFLOW!
+# The workflow named "Custard CI" triggers both custard-run.yaml and custard-run-dev.yaml.
+# TODO: To sunset old tests remove `affected`, `lint`, and `test` jobs (only keep `region-tags`).
 name: Custard CI
 on:
   push:
@@ -28,7 +31,7 @@ env:
 # TODO: remove all jobs except region-tags (should be tested by custard-run workflows)
 jobs:
   affected:
-    name: Finding affected tests
+    name: (legacy) Finding affected tests
     runs-on: ubuntu-latest
     timeout-minutes: 2
     outputs:
@@ -119,6 +122,8 @@ jobs:
       - run: ./.github/workflows/utils/region-tags-tests.sh
 
   test:
+    if: github.event.pull_request && github.event.pull_request.head.repo.fork == false
+    name: (legacy) test
     needs: affected
     runs-on: ubuntu-latest
     timeout-minutes: 120 # 2 hours hard limit

.github/workflows/custard-run-dev.yaml

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-name: (experimental / dev) Custard run
+name: (dev) Custard run
 
 on:
   # Run tests when a pull request is created or updated.
@@ -52,7 +52,7 @@ jobs:
       head-sha: ${{ github.event.workflow_run.head_sha || inputs.ref || github.sha }}
       config-file: .github/config/nodejs-dev.jsonc
       paths: ${{ (inputs.run-all && '.') || inputs.paths || '' }}
-      check-name: (experimental / dev) Custard CI
+      check-name: (dev) Custard CI / tests
       create-check-if: ${{ !!github.event.workflow_run }}
 
   test:
@@ -77,7 +77,7 @@ jobs:
         id: queued
         with:
           sha: ${{ github.event.workflow_run.head_sha || inputs.ref || github.sha }}
-          name: (experimental / dev) Custard CI / ${{ github.job }} (${{ matrix.path }})
+          name: (dev) Custard CI / ${{ github.job }} (${{ matrix.path }})
           job-name: ${{ github.job }} (${{ matrix.path }})
           if: ${{ !!github.event.workflow_run }}
       - name: Checkout

.github/workflows/custard-run.yaml

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-name: (experimental) Custard run
+name: Custard run
 
 on:
   # Run tests when a pull request is created or updated.
@@ -57,7 +57,7 @@ jobs:
       head-sha: ${{ github.event.workflow_run.head_sha || inputs.ref || github.sha }}
       config-file: .github/config/nodejs.jsonc
       paths: ${{ (inputs.run-all && '.') || inputs.paths || '' }}
-      check-name: (experimental) Custard CI
+      check-name: Custard CI / tests
       create-check-if: ${{ !!github.event.workflow_run }}
 
   lint:
@@ -73,7 +73,7 @@ jobs:
         with:
           sha: ${{ github.event.workflow_run.head_sha || inputs.ref || github.sha }}
           status: in_progress
-          name: (experimental) Custard CI / ${{ github.job }}
+          name: Custard CI / ${{ github.job }}
           if: ${{ !!github.event.workflow_run }}
       - name: Checkout
         uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
@@ -123,7 +123,7 @@ jobs:
         id: queued
         with:
           sha: ${{ github.event.workflow_run.head_sha || inputs.ref || github.sha }}
-          name: (experimental) Custard CI / ${{ github.job }} (${{ matrix.path }})
+          name: Custard CI / ${{ github.job }} (${{ matrix.path }})
           job-name: ${{ github.job }} (${{ matrix.path }})
           if: ${{ !!github.event.workflow_run }}
       - name: Checkout

CONTRIBUTING.md

@@ -64,34 +64,35 @@ const assert = require('node:assert/strict');
 
 ### CI testing
 
-For new samples, a GitHub Actions workflow should be created to run your tests
-on the CI system:
-
-1. Check that your new samples and sample tests are on a branch created directly from this repo `GoogleCloudPlatform/nodejs-docs-samples`. Not a fork.
-
-1. Add an entry to
-   [.github/workflows/utils/workflows.json](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/.github/workflows/utils/workflows.json)
-   matching the directory with your sample code.
-
-1. From the root of the repo, generate a new workflow in the
-   [workflows](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/.github/workflows)
-   directory. You can specify a `path` to only generate the specific workflow,
-   e.g. `cloud-tasks`. If the path is omitted, all workflows will be generated.
-
-        node .github/workflows/utils/generate.js -s [path]
-
-> **Note** There are some existing samples that use an alternative CI system. It
-> is recommended to use GitHub Actions for new samples, but these instructions
-> are provided below for your reference.
->
-> Add a **build configuration file (`.cfg`)** for your samples in
-> **[`.kokoro/`](https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/main/.kokoro)**.
-> Check existing config files for the right format.
->
-> All tests need a corresponding job file outside of GitHub. If you are a
-> Googler, please provide the CL alongside your PR. See the internal codelab for
-> Kokoro for details. If you don't work at Google, the person reviewing your PR
-> will create the job config for you.
+> **tl;dr**: Any check with `(dev)`, `(experimental)`, or `(legacy)` can be ignored and should **not block** your PR from merging.
+
+This repository uses the 🍮 **Custard CI** from
+[`GoogleCloudPlatform/cloud-samples-tools`](https://github.com/GoogleCloudPlatform/cloud-samples-tools)
+to set up and run tests.
+
+First, it checks which files changed in a PR to find which packages were affected.
+In this repo, a _package_ is defined as a directory containing a `package.json` file.
+
+If a global file (not belonging to a package) is changed, all packages are marked as affected.
+Global files are can contain repo-level configurations that could affect other packages.
+Typically, contributors should only modify files in a package.
+
+Only affected packages are linted and tested.
+For tests, we use a
+[matrix strategy](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow)
+to generate isolated jobs for each affected test.
+
+We're working to improve the testing infrastructure, while keeping tests stable.
+This means you'll sometimes see some new experimental tests, you can generally ignore them.
+Or two versions of one test running while we make sure the new version works well.
+
+Here's a list of which tests are required and which ones you can ignore.
+* `Custard CI / lint`: (**required**) runs only for affected packages
+* `Custard CI / tests`: (**required**) this runs until all tests have finished
+* `Custard CI / test (package)`: (**required**) runs only for affected packages
+* `(dev) Custard CI / test (package)`: (_ignore_) this test has errors we're working on
+* `(experimental) Custard CI / ...`: (_ignore_) this is a new unstable version
+* `Custard CI / (legacy) ...`: (_ignore_) this is the old version running while we make sure the new one works
 
 ### TypeScript Support