apt-get update on manylinux_2_24 suddenly broken

[JelleBakker]

Hello,

Since last week, the CI pipeline for the open source x-IMU3 Software is suddenly broken.
https://github.com/xioTechnologies/x-IMU3-Software/actions/runs/4812594230

The CI makes use of cibuildwheel and it is now failing with the following errors:

  Ign:1 http://deb.debian.org/debian stretch InRelease
  Ign:2 http://security.debian.org/debian-security stretch/updates InRelease
  Ign:3 http://deb.debian.org/debian stretch-updates InRelease
  Ign:4 http://deb.debian.org/debian stretch Release
  Ign:5 http://security.debian.org/debian-security stretch/updates Release
  Ign:6 http://deb.debian.org/debian stretch-updates Release
  Ign:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
  Ign:8 http://deb.debian.org/debian stretch/main i386 Packages
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Ign:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:8 http://deb.debian.org/debian stretch/main i386 Packages
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Ign:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:8 http://deb.debian.org/debian stretch/main i386 Packages
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Ign:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:8 http://deb.debian.org/debian stretch/main i386 Packages
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Ign:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:8 http://deb.debian.org/debian stretch/main i386 Packages
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Ign:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Err:8 http://deb.debian.org/debian stretch/main i386 Packages
    404  Not Found
  Ign:10 http://deb.debian.org/debian stretch/main all Packages
  Ign:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
  Ign:11 http://deb.debian.org/debian stretch/main Translation-en_US
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
  Err:13 http://deb.debian.org/debian stretch-updates/main i386 Packages
    404  Not Found
  Ign:14 http://deb.debian.org/debian stretch-updates/main Translation-en_US
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Ign:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Ign:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Err:7 http://security.debian.org/debian-security stretch/updates/main i386 Packages
    404  Not Found [IP: 151.101.194.132 80]
  Ign:9 http://security.debian.org/debian-security stretch/updates/main all Packages
  Ign:15 http://security.debian.org/debian-security stretch/updates/main Translation-en_US
  Reading package lists...
  W: The repository 'http://deb.debian.org/debian stretch Release' does not have a Release file.
  W: The repository 'http://security.debian.org/debian-security stretch/updates Release' does not have a Release file.
  W: The repository 'http://deb.debian.org/debian stretch-updates Release' does not have a Release file.
  E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-i386/Packages  404  Not Found
  E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-i386/Packages  404  Not Found [IP: 151.101.194.132 80]
  E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-i386/Packages  404  Not Found
  E: Some index files failed to download. They have been ignored, or old ones used instead.

The error can be reproduced with this basic cibuildwheel setup (this code is not exactly copied from our CI but simplified for the purpose of easiliy reproducing this error).

      - name: Build wheel
        uses: pypa/cibuildwheel@v2.12.3
        env:
          CIBW_ARCHS: all
          CIBW_BUILD: cp311-manylinux_x86_64
          CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_24
          CIBW_BEFORE_BUILD: apt-get update

The error happens because of apt-get update in the CIBW_BEFORE_BUILD which we need for the CI to run properly.

Is this a bug in cibuildwheel that needs to be fixed or do you have a suggestion for me to change my code so that it works again? For instance, should I be using a different manylinux image?

Thank you,
Jelle

[henryiii]

Pretty sure it's a GitHub networking issue. Those sorts of errors up up across a variety of actions, not related to cibuildwheel.

Also, you should probably not be using manylinux_2_24. It was a failed attempt to move to a Debian base and has been replaced by manylinux_2_28. Manylinux depends on the RedHat Developer Toolkit compilers, and there wasn't anything similar in Debian. Also, the EoL's for Debian were not manageable.

See pypa/manylinux#1332 and pypa/manylinux#1369; support for manylinux_2_24 ended Jan 1, 2023.

[JelleBakker]

@henryiii
According to this, the errors are not a GitHub networking issue.

We have tried manylinux_2_28 and default (presumed to be manylinux2014) and the build never completed; we cancelled after waiting an hour. Until a few weeks ago, manylinux_2_24 completed in less than a minute.

[henryiii]

So this is a manylinux issue, but you can use echo "deb http://archive.debian.org/debian stretch main" > /etc/apt/sources.list to patch it. It would be nice to fix this before discontinuing the image production, @mayeut?

[mayeut]

It would be nice to fix this before discontinuing the image production

Not sure what you meant by that given you already mentioned that the image support was already discontinued (& image production stopped at the same time, it's not being pushed/updated anymore).

I'll mention the mirror fix in the upstream manylinux issue.