feat(datasource/deb): Support deb indices compression

[steffsas]

Changes

This MR introduces support for Debian index compression, as outlined in the Debian Wiki and discussed here.

The old behavior of the Debian datasource was:

1. Assume gzip compression and resolve Package.gz file for package meta information
2. If available, retrieve InRelease file to compare checksum and abort if they do not match

The new behavior:

1. Try to fetch InRelease or Release file as it contains information about the package file including the compression algorithm available
   1.1 If release file is found, retrieve Package file with compression algorithms (order xz, bzip2, gzip, or uncompressed) and checksum of the package file
   1.2 If no release file is found, assume Package.gz file (like old behavior)
2. Download package file and compare checksum if release file was found

Retrieving and parsing content from the package file remains the same.

Context

Recently published package files are often compressed using xz. Consequently, the Debian data source in Renovate is currently unable to resolve version information from these files. This limitation is discussed and illustrated in this reproduction repository.

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

Tested Output

Running renovate on the minimal reproduction repository steffsas/renovate-reproduction-deb-xz results in the following expected output. In this example, the tor package resides within bookworm-backports and its package file is compressed via xz.

"regex": [
           {
             "deps": [
               {
                 "depName": "tor",
                 "currentValue": "0.4.7-11",
                 "datasource": "deb",
                 "replaceString": "# renovate: suite=bookworm-backports depName=tor\nENV TOR_VERSION=\"0.4.7-11\"",
                 "updates": [
                   {
                     "bucket": "non-major",
                     "newVersion": "0.4.8.14-1~bpo12+1",
                     "newValue": "0.4.8.14-1~bpo12+1",
                     "newMajor": 0,
                     "newMinor": 4,
                     "newPatch": 8,
                     "updateType": "patch",
                     "isBreaking": false,
                     "branchName": "renovate/tor-0.x"
                   }
                 ],
                 "packageName": "tor",
                 "versioning": "deb",
                 "warnings": [],
                 "registryUrl": "https://deb.debian.org/debian?suite=bookworm-backports&components=main,contrib,non-free&binaryArch=amd64",
                 "homepage": "https://www.torproject.org/",
                 "currentVersion": "0.4.7-11",
                 "isSingleVersion": true,
                 "fixedVersion": "0.4.7-11"
               }
             ],
             "matchStrings": [
               "#\\s*renovate:\\s*?(suite=(?<suite>.*?))?\\s*depName=(?<depName>.*?)?\\sENV .*?_VERSION=\"(?<currentValue>.*)\""
             ],
             "datasourceTemplate": "deb",
             "packageFile": "Dockerfile"
           }
         ]

[viceice]

are they using any binary node modules we need to take care of for different architecture?

we build images for amd64 and arm64

[steffsas]

According to their documentation, they do not require additional binary node modules. While unbzip2-stream is fully in JavaScript, lzma-native "provides pre-built binaries for multiple Node.js versions and all major OS using node-pre-gyp".

[github-actions]

Hi there,

Please don't merge from main into your PR's branch unless you have a merge conflict or have a specific reason to do so. Doing so consumes CI systems unnecessarily, and also resets CI approvals if this is your first PR to the repo. This repo uses GitHub's Merge Queue, which will automatically test your PR against main before merging, so it's no longer necessary to always keep branches up-to-date.

Thanks,

The Renovate team

[viceice]

lockfile broken

[steffsas]

lockfile broken

Should be fixed now.

[viceice]

can you please do the refactor first in a separate PR? it would be much easier to review

[viceice]

why removal of so many tests?