Sudden increase in TOOMANYREQUESTS errors when downloading artifacts from GHCR

[knqyf263]

Select Topic Area

Question

Body

We've been using GitHub Container Registry (GHCR) to download artifacts for our open-source project. However, since earlier this week, we've been experiencing frequent TOOMANYREQUESTS errors. For example:

21:00:22 - GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:15a6fd375dca7d4d11fde458d5ba5f3fa60832e93da70143ad9e9e188d8d8c07: TOOMANYREQUESTS: retry-after: 576.466µs, allowed: 44000/minute

We're aware of the rate limiting policies, but this issue has suddenly started occurring in environments where we've never seen TOOMANYREQUESTS errors before. Moreover, many of our users have reported seeing these errors since earlier this week.

This leads us to wonder if there might have been a change on GitHub's side. We've checked the GitHub Status page for any relevant incidents but couldn't find anything that seems related.

We'd greatly appreciate any information or insights.

[billhammond-dev]

aquasecurity/trivy-action#389 - workaround using your own registry for the DB.

[sakky016]

Hi @knqyf263,

I see that java-db publishing jobs are failing from last few days: https://github.com/aquasecurity/trivy-java-db/actions.

New java-db is not available and the already available java-db's metadata.json shows past date for NextUpdate. This would cause trivy to download java-db again and again on each run even though it had downloaded java-db successfully.

"NextUpdate":"2024-09-23T01:06:28.762321626Z"

I suspect this could be an issue for the increased TOOMANYREQUESTS in ghcr.io.

[knqyf263]

Thanks for the heads-up. We'll look into it tomorrow.
aquasecurity/trivy-java-db#34

[knqyf263]

@sakky016 This is not a thread to talk about Trivy. Please open a discussion in Trivy.

[schnatterer]

I get the impression that this rate limit hits us even as authenticated users, and sometimes even when pulling the trivy image itself.

Could this be a global rate limit on the ghcr.io/aquasecurity repo?

[mmusenbr]

I also got this impression, I tried a proxy-cache for ghcr with a cache timeout bigger than the github normal rate limit reset time, and I still see the issue randomly. Also if I configure a auth-token, the same happens from time to time.
Switching to the public.ecr.aws now.

[schnatterer]

Easiest workaround at the moment, use the official mirror at AWS, e.g.

 trivy image \
  --db-repository public.ecr.aws/aquasecurity/trivy-db \
  --java-db-repository public.ecr.aws/aquasecurity/trivy-java-db \
  --scanners vuln \
  ghcr.io/cloudogu/gitops-playground:0.7.0

The upcoming release trivy 0.56 might fix this issue.

Found here: aquasecurity/trivy-action#389

BTW before I found that out, I create a GitHub Action that mirrors the trivy database automatically:
https://github.com/cloudogu/trivy-db-mirror/blob/12a100bc764e4efdf3f480866a9a2db0c89cdd57/.github/workflows/mirror-dbs.yml

It worked, but with an official mirror, there is no need for me to maintain this repo. It might be useful for anyone else who might want to mirror the DBs. They are just simple OCI artifacts.

[hcguersoy-cg]

Made my day, thank you!

[kaiorafael]

Thank you very much

[SahrulRD-Smatel]

thank u :))

[graham924]

Adding --db-repository public.ecr.aws/aquasecurity/trivy-db worked for me

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[ivan-sre]

I'm not sure it's related, but it works properly on ubuntu-22.04 github runners.
Please aware that ubuntu-latest is pointing to ubuntu-24.04 now

[TicianoH]

no reply from GitHub officially? Have you reached out to them as customers directly? Thanks for the post @knqyf263

[jitendra42]

@TicianoH this fixed issues for me
aquasecurity/trivy-action#389 (comment)

[jitendra42]

Just add different trivy db repo url and it will fix the issue

  - name: Run Trivy vulnerability scanner
    uses: aquasecurity/[email protected]
    env:
      TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
      TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
    with:
      image-ref: ${{ steps.set-image-tag.outputs.image-tag }}
      ignore-unfixed: true
      vuln-type: 'os,library'
      severity: 'CRITICAL,HIGH'
      format: 'table'

[knqyf263]

@TicianoH Unfortunately, no. We're trying to work it around on our end now.
aquasecurity/trivy#7679

[knqyf263]

This is not a Trivy thread, but I'm leaving a comment as it is likely to be visited by Trivy users. Since discussions with GitHub have not progressed, we have released v0.57.1 that primarily uses mirror.gcr.io instead of GHCR as a workaround. If you are having this problem, please try v0.57.1+ and let us know if you still face the same issue even with the new version.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[hairmare]

A post-mortem from GitHub's side would be nice, looking at related issues show that the change leading to this thread had a considerable impact on a large amount of users.

While it's understandable that MSFT feels like they do not want to foot the bill for hosting the trivy database, this change was still communicated by breaking things.

Did we all miss the memo or can the processes that were used to introduce the change get updated to ensure that people are warned ahead of time and are aware that their builds might be broken without any clear indicator of the error or path forward to a fix.