Feature request: xz support for deb datasource

[Finkregh]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

github, 39.60.0

Please tell us more about your question or problem

I use the deb datasource, which currently only supports .gz compressed files. This repo though only has .xz files: https://deb.debian.org/debian-security/dists/bookworm-security/main/binary-amd64/ in comparison to https://deb.debian.org/debian/dists/bookworm/main/binary-amd64/

Edit: same with the -updates repo.

This is especially an issue for updates that are security relevant, e.g. smarty4:
https://packages.debian.org/bookworm/smarty4

4.3.0-1+deb12u2 is not available in the normal repos.

Logs (if relevant)

Logs

2024-12-13T11:20:32.682458478Z          "options": {
2024-12-13T11:20:32.682459682Z            "headers": {
2024-12-13T11:20:32.682460855Z              "user-agent": "RenovateBot/39.60.0 (https://github.com/renovatebot/renovate)",
2024-12-13T11:20:32.682462154Z              "accept-encoding": "gzip, deflate, br"
2024-12-13T11:20:32.682463371Z            },
2024-12-13T11:20:32.682464768Z            "url": "https://internal_mirror_of_deb_debian_org_security/dists/bookworm-security/main/binary-amd64/Packages.gz",
2024-12-13T11:20:32.682466039Z            "hostType": "deb",
2024-12-13T11:20:32.682467259Z            "username": "",
2024-12-13T11:20:32.682468380Z            "password": "",
2024-12-13T11:20:32.682469527Z            "method": "GET",
2024-12-13T11:20:32.682470637Z            "http2": false
2024-12-13T11:20:32.682471832Z          },
2024-12-13T11:20:32.682473138Z          "response": {
2024-12-13T11:20:32.682474295Z            "statusCode": 404,
2024-12-13T11:20:32.682475516Z            "statusMessage": "Not Found",
2024-12-13T11:20:32.682476885Z            "body": "{\n  \"errors\" : [ {\n    \"status\" : 404,\n    \"message\" : \"Not Found\"\n  } ]\n}",
2024-12-13T11:20:32.682481011Z            "headers": {
2024-12-13T11:20:32.682482169Z              "date": "Fri, 13 Dec 2024 11:20:32 GMT",
2024-12-13T11:20:32.682483389Z              "content-type": "application/json;charset=ISO-8859-1",
2024-12-13T11:20:32.682484801Z              "transfer-encoding": "chunked",
2024-12-13T11:20:32.682486071Z              "connection": "keep-alive",
2024-12-13T11:20:32.682487264Z              "x-jfrog-version": "Artifactory/xxx",
2024-12-13T11:20:32.682488489Z              "x-artifactory-id": "xxx",
2024-12-13T11:20:32.682489701Z              "x-artifactory-node-id": "internal-artifactory-0",
2024-12-13T11:20:32.682491380Z              "strict-transport-security": "max-age=31536000; includeSubDomains"
2024-12-13T11:20:32.682492522Z            },
2024-12-13T11:20:32.682493700Z            "httpVersion": "1.1",
2024-12-13T11:20:32.682494954Z            "retryCount": 0
2024-12-13T11:20:32.682496243Z          }
2024-12-13T11:20:32.682497524Z        }

[frederic-klein]

Willing to try to provide a PR. Ideas:

1. xz as fallback, if Packages.gz does not exist
2. check InRelease for Version and use xz compression, if version >= 12
3. expose compression as configuration
4. check InRelease for matching path and resolve compression
5. ...

note Packages.xz is a valid option, compare: https://wiki.debian.org/DebianRepository/Format#Compression_of_indices

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team

[code-acrobat]

Some update about the severity. If you want to use Debian 12 (bookworm).

.gz is hardcoded :/
https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/deb/index.ts#L75

main packages repos still have .gz
https://deb.debian.org/debian/dists/bookworm/main/binary-amd64/

However security updates only offer xz
https://security.debian.org/debian-security/dists/bookworm-security/main/binary-amd64/Packages.xz

Debian 11 still seems to work on gz
https://security.debian.org/debian-security/dists/bullseye-security/main/binary-amd64/Packages.gz

[github-actions[bot]]

Hi there,

This Discussion has been flagged previously as needing a minimal reproduction, but one does not yet exist. This is a reminder that the reproduction is still needed in order to progress to next steps.

If you have provided a minimal reproduction which satisfies our definitions above, but it went unnoticed, please reply to this message to help us find it.

Thanks, the Renovate team

[alessio-pascolini]

My use case that block me with the update of tzdata debian package

How are you running Renovate?

Renovate app with Github

Please tell us more about your question or problem

I would use the deb datasource to update the tzdata package in the python image. The deb package is pinned in the dockerfile, so I need renovate to warn me about the update

Dockerfile

FROM python:3.10.16-slim-bookworm

# renovate: suite=bookworm-updates depName=tzdata
ENV TZDATA_VERSION="2025a-0+deb12u1"

RUN apt-get update && apt-get install -y --no-install-recommends \
  tzdata="${TZDATA_VERSION}" \
  && rm -rf /var/lib/apt/lists/*

renovate.json

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended"
  ],
  "customManagers": [
    {
      "customType": "regex",
      "fileMatch": ["^Dockerfile$"],
      "matchStrings": [
        "#\\s*renovate:\\s*?(suite=(?<suite>.*?))?\\s*depName=(?<depName>.*?)?\\sENV .*?_VERSION=\"(?<currentValue>.*)\""
      ],
      "datasourceTemplate": "deb",
      "registryUrlTemplate": "https://deb.debian.org/debian?suite={{#if suite }}{{suite}}{{else}}stable{{/if}}&components=main,contrib,non-free&binaryArch=amd64"
    }
  ]
}

the bookworm-updates repo has Package.xz so renovate failed to get the Package file, looking for Package.gz

https://deb.debian.org/debian/dists/bookworm-updates/main/binary-amd64/Packages.xz

Logs

log snippet of renovate app

DEBUG: hostRules: no authentication for deb.debian.org
DEBUG: Skipping component due to an error
{
  "componentUrl": "https://deb.debian.org/debian/dists/bookworm-updates/main/binary-amd64"
  "error": {
    "name": "HTTPError",
    "code": "ERR_NON_2XX_3XX_RESPONSE",
    "timings": {
      "start": 1744213837700,
      "socket": 1744213837703,
      "lookup": 1744213837705,
      "connect": 1744213837712,
      "secureConnect": 1744213837719,
      "upload": 1744213837719,
      "response": 1744213837960,
      "end": 1744213837962,
      "phases": {
        "wait": 3,
        "dns": 2,
        "tcp": 7,
        "tls": 7,
        "request": 0,
        "firstByte": 241,
        "download": 2,
        "total": 262
      }
    },
    "message": "Response code 404 (Not Found)",
    "stack": "HTTPError: Response code 404 (Not Found)\n    at Request._onResponseBase (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:913:31)\n    at Request._onResponse (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:948:24)\n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:962:23)\n    at Object.onceWrapper (node:events:633:26)\n    at ClientRequest.emit (node:events:530:35)\n    at ClientRequest.emit (node:domain:489:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at HTTPParser.parserOnIncomingClient (node:_http_client:716:27)\n    at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)\n    at TLSSocket.socketOnData (node:_http_client:558:22)\n    at TLSSocket.emit (node:events:518:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at TLSSocket.Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:189:23)",
    "options": {
      "headers": {
        "user-agent": "RenovateBot/39.238.0 (https://github.com/renovatebot/renovate)",
        "accept-encoding": "gzip, deflate, br"
      },
      "url": "https://deb.debian.org/debian/dists/bookworm-updates/main/binary-amd64/Packages.gz",
      "hostType": "deb",
      "username": "",
      "password": "",
      "method": "GET",
      "http2": false
    },
    "response": {
      "statusCode": 404,
      "statusMessage": "Not Found",
      "body": "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL was not found on this server.</p>\n<hr>\n<address>Apache Server at ftp.debian.org Port 80</address>\n</body></html>\n",
      "headers": {
        "connection": "keep-alive",
        "content-length": "260",
        "server": "Apache",
        "x-content-type-options": "nosniff",
        "x-frame-options": "sameorigin",
        "referrer-policy": "no-referrer",
        "x-xss-protection": "1",
        "permissions-policy": "interest-cohort=()",
        "content-type": "text/html; charset=iso-8859-1",
        "via": "1.1 varnish, 1.1 varnish",
        "accept-ranges": "bytes",
        "age": "0",
        "date": "Wed, 09 Apr 2025 15:50:37 GMT",
        "x-served-by": "cache-ams2100106-AMS, cache-iad-kjyo7100092-IAD",
        "x-cache": "HIT, MISS",
        "x-cache-hits": "2, 0",
        "x-timer": "S1744213838.840170,VS0,VE122"
      },
      "httpVersion": "1.1",
      "retryCount": 0
    }
  }
}

[steffsas]

Is there any update here? Would like to have this feature as well. I am also willing to provide a PR.

Also according to the debian wiki page, supporting xz and bzip2 compression is mandatory, see https://wiki.debian.org/DebianRepository/Format#line-58.

[viceice]

that's not a minimal reproduction. please see corresponding description

[steffsas]

As you wish, I've created a minimal reproduction repo, see https://github.com/steffsas/renovate-reproduction-deb-xz.
Let me know whether I should create a PR for a fix.

[rarkins]

Thanks, please go ahead with a PR

[steffsas]

This is the PR: #35865

[steffsas]

I created a minimal reproduction repo, see https://github.com/steffsas/renovate-reproduction-deb-xz. I am willing to create a PR. What do you think?

[d0x2f]

I am facing this issue as well, the deb source I wish to use doesn't have either a gz or xz compressed Package file.
It supports uncompressed or bz2.
https://apt-archive.postgresql.org/pub/repos/apt/dists/bullseye-pgdg-archive/InRelease