Linux package repositories are temporarily down (Debian and RPM packages)

[dagood]

This looks like this on Ubuntu 19.04, during apt-get update:

E: Failed to fetch https://packages.microsoft.com/ubuntu/19.04/prod/dists/disco/main/binary-amd64/Packages.gz  File has unexpected size (29008 != 25211). Mirror sync in progress? [IP: 13.91.48.226 443]
   Hashes of expected file:
    - Filesize:25211 [weak]
    - SHA512:9c24d870e05029726895af3f19c91cb1f300e0406e0dafc4c77868f9d0b7da93b7546546f96d73411a59661fe787e8ae70da8db499b5d0d9f9b115ace9c64d49
    - SHA256:df8e493a9716e39b680ac89ce4ad189ceb00bd78ceded6bfb50cea2f0861d0e6
    - SHA1:7d4c121da39d6322b10caa73d37caeac667799a1 [weak]
    - MD5Sum:c428296bc9b29cc7237c751dbb0ffe9d [weak]
   Release file created at: Fri, 13 Dec 2019 17:48:48 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

This is expected to some degree whenever we push a Debian or RPM package to the feeds. I'm opening this issue to track it and raise awareness. It's possible there's an underlying issue here beyond the normal delay and I'm looking into it.

I have dotnet/docs#16670 open to add some description of the problem to the docs to make it easier to tell when this is happening and how long you normally need to wait.

/cc @dleeapho @richlander @NikolaMilosavljevic

---

Here's an all-up status report of what works for me:

• Ubuntu 16.04 ✔️
• Ubuntu 18.04 ✔️
• Ubuntu 19.04 ✔️
• Debian 9 ✔️ (up at 2020-01-15 00:42:35Z) https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home
• Debian 10 ✔️ (up at 2020-01-15 00:42:35Z) https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

On the RPM side, in particular CentOS 7, ✔️ (up at 2020-01-15 00:58:37Z or earlier) #4121.

Note that 2.1.15 is having trouble on the publish side and is tracked at #4124. The above is only for 3.0.2 and 3.1.1.

[dagood]

Last I heard, the expected availability time is:

• Debian package (apt-get): 30 minutes
• RPM package (yum, zypper, dnf): 2 hours

Debian packages were published more than 3 hours ago, so I'm filing an internal ticket to get help with those.

RPM packages finished later, and there are ongoing deeper issues there. See #4121, "yum install dotnet-sdk-3.0 fails on centos7"

[dagood]

I filed https://icm.ad.msft.net/imp/v3/incidents/details/169350659/home for the Ubuntu 16.04/18.04/19.04 + Debian 9/10 issue. (Microsoft internal access required.)

[rolinge]

Docker builds are broken, since they use the debian repo

Logs

Step 4/20 : RUN wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg && mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ && wget -q https://packages.microsoft.com/config/debian/9/prod.list && mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
---> Using cache
---> 9abb9d4016fa
Step 5/20 : RUN chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg && chown root:root /etc/apt/sources.list.d/microsoft-prod.list && apt-get install -y apt-transport-https dirmngr gnupg ca-certificates && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
---> Using cache
---> a2750cab1fbf
Step 6/20 : RUN echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | tee /etc/apt/sources.list.d/mono-official-stable.list
---> Using cache
---> 66b303a74706
Step 7/20 : RUN apt-get update
---> Running in 44e44ad16f9a
Hit:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Get:5 https://download.mono-project.com/repo/debian stable-stretch InRelease [5878 B]
Get:6 https://packages.microsoft.com/debian/9/prod stretch InRelease [4009 B]
Get:8 https://download.mono-project.com/repo/debian stable-stretch/main amd64 Packages [49.4 kB]
Get:9 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [68.9 kB]
Err:9 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
Writing more data than expected (72399 > 68862)
Hashes of expected file:

• Filesize:68862 [weak]
• SHA512:c45a7156e5d4796ec88721f1385bdff78e7f0396031c008a31b75cf5faecdb726a52a428256d55c9424af16d23c939f24fbf2a0d467c7082c7cdcbfad4b323c2
• SHA256:5bccd1a4be6d9e2b60465cad6bcc793d0bd1f60a29de8fe2239453b528f8cf20
• SHA1:600a5c433ae084819b5efa743ec9ec0ea38dc130 [weak]
• MD5Sum:964348da7a5707ac25aaca7577e46728 [weak]
  Release file created at: Mon, 16 Dec 2019 21:40:05 +0000
  Fetched 59.3 kB in 1s (58.9 kB/s)
  Reading package lists...
  E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2 Writing more data than expected (72399 > 68862)
  Hashes of expected file:
  - Filesize:68862 [weak]
  - SHA512:c45a7156e5d4796ec88721f1385bdff78e7f0396031c008a31b75cf5faecdb726a52a428256d55c9424af16d23c939f24fbf2a0d467c7082c7cdcbfad4b323c2
  - SHA256:5bccd1a4be6d9e2b60465cad6bcc793d0bd1f60a29de8fe2239453b528f8cf20
  - SHA1:600a5c433ae084819b5efa743ec9ec0ea38dc130 [weak]
  - MD5Sum:964348da7a5707ac25aaca7577e46728 [weak]
  Release file created at: Mon, 16 Dec 2019 21:40:05 +0000
  E: Some index files failed to download. They have been ignored, or old ones used instead.
  The command '/bin/sh -c apt-get update' returned a non-zero code: 100

[dagood]

@rolinge I put the logs in a collapsible section to save some scrolling, hope that's ok.

The Ubuntu issue has been resolved by the linux repo team, and yeah, I was trying this out on various distros and saw the Debian 9 and Debian 10 error.

I've filed a new ticket for Debian 9/10: https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

[wenxuan-f]

It is working now! woohoo TY

[rolinge]

@rolinge I put the logs in a collapsible section to save some scrolling, hope that's ok.

The Ubuntu issue has been resolved by the linux repo team, and yeah, I was trying this out on various distros and saw the Debian 9 and Debian 10 error.

I've filed a new ticket for Debian 9/10: https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

@dagood when you have a minute, show me how to do do the 'collapsible section' trick 😄
Never mind, I found it...

A collapsible section with markdown

Like This!

• stuff
• more stuff

[rolinge]

My docker build is proceeding, so it appears to be fixed for debian 9.
Thanks!

[dagood]

Yep, they fixed up Debian 9 and 10, updated my list. Thanks for checking. 😄

[harshavardhana]

How can we ensure that this doesn't happen again? - This is this the third time in the last 1yr and whenever it happens the entire project CI/CD breaks because we are dependent on it unfortunately.

I am sure many are with me here - I hope there are more stricter guidelines on verifying what is being published. Publishing packages should be simple and shouldn't break things in this weird manner.

[dagood]

How can we ensure that this doesn't happen again?

We (.NET Core) are trying to work internally with the team that owns these package repos to improve this. packages.microsoft.com is a shared service that a bunch of Microsoft products (VS Code, PowerShell) rely on to be available on Linux through package managers.

I am sure many are with me here - I hope there are more stricter guidelines on verifying what is being published. Publishing packages should be simple and shouldn't break things in this weird manner.

Unfortunately the problem isn't what's being published--I wish it were, so we could take ownership of some fix--rather, it's with the infrastructure behind the feeds. It is super frustrating that it's "normal" to have an up to 2hr service outage every time we publish a package (or someone else publishes a package), and that we need to effectively monitor it ourselves and ping the team when it doesn't come back online.