[bug] "Generate builder" and "Run sigstore/cosign-installer" steps failing with `error updating to TUF remote mirror: invalid key`

[jkreileder]

The "Generate builder" and "Run sigstore/cosign-installer" steps have started failing for my workflows. This used to work fine, not sure if it is just an intermittent error or something more fundamental:

Here's a build that worked 18 hours ago but is failing now (i.e. without any code changes): https://github.com/jkreileder/cf-ips-to-hcloud-fw/actions/runs/8339143012 (corresponding workflow)

1. Generate builder error:

Verifying artifact slsa-generator-container-linux-amd64: FAILED: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}

FAILED: SLSA verification failed: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 6.

2. Run sigstore/cosign-installer@6e04d228eb30da1757ee4e1dd75a0ec73a653e06 error:

Error: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
main.go:74: error during command execution: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 1.

[tirumerla]

Started seeing the same just now.

[brad-getpassport]

+1 Seeing this across all our builds in the last 2 hours... this is holding up our release candidate...

Generating ephemeral keys...
Retrieving signed certificate...
creating signer: getting signer: getting key from Fulcio: getting CTFE public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 1.

[jenstroeger]

‼️ @ianlewis @laurentsimon please advise! This is breaking all our build pipelines, and talking with folks their pipelines are also dead in the water ‼️

[haydentherapper]

@laurentsimon @ianlewis You just need to bump Cosign to at least v2.2.0, latest is v2.2.3.

I'd also recommend checking your renovate settings, I would have expected this bump to get picked up automatically.

[behnazh-w]

@haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

[suzuki-shunsuke]

haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

According to the announcement, v2.2.0 should support a new TUF Trust root, so I don't think we need to update Cosign in SLSA Verifier.

[suzuki-shunsuke]

I see. The latest slsa-github-generato v1.9.0 uses the old slsa-verifier which uses the old cosign v2.0.2

slsa-github-generator/.github/actions/generate-builder/action.yml

Line 95 in 07e64b6

VERIFIER_RELEASE: v2.3.0 # The version of the verifier to download.

https://github.com/slsa-framework/slsa-verifier/blob/c9abffe4d2ab2ffa0b2ea9b2582b84164f390adc/go.mod#L21

The main branch uses the latest slsa-verifier v2.4.1.

slsa-github-generator/.github/actions/generate-builder/action.yml

Line 95 in b595e06

VERIFIER_RELEASE: v2.4.1 # The version of the verifier to download.

So we need to release a new version of slsa-github-generator, then the issue would be solved.

[suzuki-shunsuke]

@laurentsimon @ianlewis
I'm sorry to bother you, but could you release a new version of slsa-github-generator to solve this issue?
#3350 (comment)

[laurentsimon]

Hi, yes let's do that. I think we need to revert a few breaking PRs we made recently in the slsa-github-generator. slsa-verifier should be ready to be released. @ramonpetgrave64 could you send PRs to temporarily revert the few breaking PRs we made recently?