Lock base version and all plugins and add `bump-jenkins` job

[dustymabe]

Periodically the latest plugins and the version of jenkins offered by https://github.com/openshift/jenkins can get out of sync where jenkins won't come up. Let's run tests on some time schedule (weekly?) that verify the current configuration in this repo is able to at least bring jenkins up without errors.

This will mean that if we ever need to redeploy in haste we'll be ready.

[dustymabe]

We also need to start re-building the jenkins container image periodically to pick up new plugin updates so we don't freeze for 6 months and potentially carry security issues.

This could be a periodic job that triggers a build in openshift.

[jlebon]

Let's repurpose this to go a step further and lock down all our plugins and the base Jenkins version itself and e.g. have a job that opens a PR to bump them with a link to documentation on how to test the PR in staging. (And maybe eventually it directly instruments the test in staging itself... or maybe better we request a separate namespace for this.)

This will require some investigations to figure out how we can run depsolving ourselves.

[jlebon]

This needs to account to the different clusters we support being different versions (and thus having different Jenkins base versions). We could choose to track the older of the two, or if it's fully mechanical, maintaining two separate base versions + plugins list shouldn't be too bad either.

[marmijo]

Let's repurpose this to go a step further and lock down all our plugins and the base Jenkins version itself and e.g. have a job that opens a PR to bump them with a link to documentation on how to test the PR in staging.

We've been discussing possible ways to implement this. One way is to have a job that removes the versions altogether, which would just pull all the latest versions, attempt a build like that and "see if it works" (loaded statement).

I tried to manually build the Jenkins instance without any versions and it failed to load because there seems to be an issue with the openshift/jenkins code. I opened openshift/jenkins#1709 regarding it.

In the mean time, we might have to find an alternative way to automate this.

[jlebon]

Was chatting with @marmijo about this today. One suggestion is to have the job on the prod Jenkins instrument the Jenkins on the stage cluster. Something like:

1. Clone repo, edit lockfile, push to coreosbot-releng branch
2. Start jenkins-s2i build on stage cluster, pointing at the ref
3. Monitor build and deployment of new Jenkins
4. Do a test FCOS build in the new Jenkins (see https://github.com/coreos/fedora-coreos-pipeline/blob/main/HACKING.md#triggering-builds-remotely)
5. If it succeeds, open PR with branch

We'd need to figure out how to set up authentication across the clusters.

Another approach is to instrument the Jenkins from which the job is running itself, but if the plugins break so hard that the Jenkins never comes back up, we'll never hear back from that job.

And yet another approach is to bring up a second Jenkins in the same namespace. I think for this we'd need to make sure we get the resource naming right so there's no conflicts between the two, but it could get messy. I think it's more conventional to use a separate namespace/cluster for tests like this.

[marmijo]

@aaradhak, @jlebon and I had a long discussion about this today.

2. Start jenkins-s2i build on stage cluster, pointing at the ref
3. Monitor build and deployment of new Jenkins
4. Do a test FCOS build in the new Jenkins (see https://github.com/coreos/fedora-coreos-pipeline/blob/main/HACKING.md#triggering-builds-remotely)

It would probably be beneficial to split the workload of this job into two separate jobs. We should also consider renaming the job to bump-jenkins-plugins instead of bump-jenkins since it more accurately represents the purpose of the job:

• bump-jenkins-plugins
  • Would update the plugins in plugins.txt to their latest versions and open a PR
• test-jenkins
  • Would take the contents from a PR (or even just a git ref) and test it on the FCOS staging cluster by building the jenkins instance using said contents. The job can then report the results back to GitHub.
  • Would be manually run to prevent interruptions to the FCOS stage pipeline just in case someone is running something there.
  • This job could be used for more than just this specific case. We would be able to use it to test any changes to the configuration of jenkins through just a PR to fedora-coreos-pipeline.

So the workflow would look something like this:

1. [bump-jenkins-plugins]: clone repo, edit lockfile, push to coreosbot-releng fork of f-c-p
2. [bump-jenkins-plugins]: open a PR against coreos:main
3. [CoreOS team]: manually runs/enables test-jenkins from the PR
4. [test-jenkins]: update jenkins-s2i BuildConfig on FCOS stage cluster to point at coreos-bot releng fork (contents of PR)
5. [test-jenkins]: build jenkins-s2i BuildConfig (will rebuild Jenkins)
6. [test-jenkins]: build FCOS inside a cosa pod and run kola qemu tests
7. [test-jenkins]: update commit status on PR
8. [CoreOS team]: merge PR, OR triage any failures and rerun

Ideally, we'd like to use the dependency resolution that occurs upstream in openshift/jenkins, but it was removed from the upstream code earlier this year. Our current plan is to investigate if this resolveDependencies function will affect the plugins base jenkins plugins packages in openshift/jenkins, or if it will only install the dependencies required of the plugins in our plugins.txt. More specifically, if one of our add-on plugins requires a more recent version of a base plugin, will resolveDependencies update that base plugin version? Once we have more information, we'll try to open a discussion with the team that manages openshift/jenkins to explore re-enabling this function in that code.

[marmijo]

After building a custom Jenkins image that was modified to run resolveDependencies, we found that it will install all dependencies (even optional ones) of the plugins we list in plugins.txt. However, even if the dependency is listed in base-plugins.txt, it will update it's version to the latest required.

We saw this today for example:

Examining optional dependency github-branch-source
Optional dependency github-branch-source already installed, need to determine if it is at a sufficient version
Upgrading previously downloaded plugin github-branch-source at 1725.vd391eef681a_e to 1732.v3f1889a_c475b_
Downloading plugin: github-branch-source from https://updates.jenkins.io/download/plugins/github-branch-source/1732.v3f1889a_c475b_/github-branch-source.hpi

The dependency plugins that fall into that category are all being updated to their latest versions in a pending PR: openshift/jenkins#1697