Random 403 errors when pushing to GHCR

[jauderho]

Troubleshooting

Pushing to GHCR seems to randomly resulting in 403s in the last 24 hours. Rerunning the GH action seems to resolve things (partially).

Behaviour

See above.

Failure message : buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

Steps to reproduce this issue

1. Kick off GH action to rebuild Docker images
2. See https://github.com/jauderho/dockerfiles/actions
3. Check one of the recent failed runs. For example, https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Expected behaviour

Builds should complete with no issue.

Actual behaviour

See https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Configuration

• Repository URL (if public): https://github.com/jauderho/dockerfiles
• Build URL (if public): https://github.com/jauderho/dockerfiles/actions/runs/1255324536

https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/tailscale.yml

[jauderho]

To be clear, I have multiple actions that each build docker images for different apps failing, see https://github.com/jauderho/dockerfiles

I currently count 5 so it does not appear to an isolated issue.

[revolunet]

same issue here, not on all images but some and its quite systematic

[crazy-max]

@markphelps Looks similar to #251

[jauderho]

Just saw a suggestion on the orange site: https://news.ycombinator.com/item?id=28607735

I'm trying it out now and will report back.

[crazy-max]

As the ephemeral secrets.GITHUB_TOKEN would be invalidated when the job is completed, I wonder if there is a limited token pool per repo? Any idea @chrispat?

We recently faced this; if you are using the docker/login action I'd give that a check as it turned out it was logging us out by default at the end of each job; resulting in some race conditions when running multiple runners on the same machine (sharing the same docker daemon).

As I can see you're not using a self-hosted runner so it's not your case.

Simple fix was to add logout: false to the action options.

Don't think it will change smth as docker logout only erases token from the credential store.

[jauderho]

Nope, it did not work. Still fails.

[onedr0p]

Also facing this issue, it seems to happen intermittently but more times than not.

[onedr0p]

I have verified the following steps work for me to resolve the issue, however, this is not ideal.

• Delete the image from your image packages that has the 403 problem.
• Trigger the CI to build and push the image again.

Maybe this can shed a light into what's happening @chrispat ?

Build that failed:
https://github.com/k8s-at-home/container-images/runs/3656105847

Build that passed after deleting the image from GHCR:
https://github.com/k8s-at-home/container-images/runs/3674834677

[onedr0p]

I have also tried toggling Inherit access from source repository (recommended) and toggling the image visibility between Public and Private to no avail. This is definitely not an issue we cannot resolve on our end besides completely deleting the image and pushing it again.

[ondrejmirtes]

I'm experiencing the same problem. I'm logging with docker-login-action:

      - name: Login to ghcr
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

And trying to push with this action:

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          builder: ${{ steps.buildx.outputs.name }}
          context: ./docker
          file: ./docker/Dockerfile
          build-args: |
            PHPSTAN_VERSION=dev-master#${{ github.sha }}
          push: true
          tags: ghcr.io/phpstan/phpstan:nightly
          platforms: linux/amd64,linux/arm64
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache

This is how the output tail looks like:

#27 [auth] phpstan/phpstan:pull,push token for ghcr.io
#27 DONE 0.0s

#28 [auth] phpstan/phpstan:pull,push token for ghcr.io
#28 DONE 0.0s

#29 [auth] phpstan/phpstan:pull,push token for ghcr.io
#29 DONE 0.0s

#26 exporting to image
#26 pushing layers 0.5s done
#26 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

[onedr0p]

After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.

It doesn't make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.

[jauderho]

@onedr0p How are you getting to that dialog? My google-fu is failing me today and I am unable to find the location so that I can check that setting. Thanks.

[revolunet]

You can find it in https://github.com/orgs/[XXX]/packages/container/[YYY]/settings

or by going to the package page then "package settings"