Healthcheck with quadlets in rootless mode - connection refused

[ToBeReplaced]

Issue Description

I am unable to get any healthcheck to operate in rootless mode with quadlets.

Steps to reproduce the issue

Steps to reproduce the issue:

$ cat >~/.config/containers/systemd/sleep.container <<EOF
[Container]
Image=registry.fedoraproject.org/fedora:latest
Exec=sleep 300
Network=none
Notify=healthy
HealthCmd=true
EOF

$ systemctl --user daemon-reload
$ systemctl --user start sleep &
[1] 1357400
$ journalctl --user -xeu sleep --no-pager
May 04 19:10:34 localhost systemd[3811]: Starting sleep.service...
░░ Subject: A start job for unit UNIT has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit UNIT has begun execution.
░░ 
░░ The job identifier is 40617.
May 04 19:10:35 localhost podman[1357401]: 2025-05-04 18:10:35.01923878 +0000 UTC m=+0.035949887 container create 1b680d910228f7b0bab31854c23b888f763a0d09a7a8bd052da1aa86a947ca94 (image=registry.fedoraproject.org/fedora:latest, name=systemd-sleep, org.opencontainers.image.vendor=Fedora Project, name=fedora, org.opencontainers.image.license=MIT, org.opencontainers.image.name=fedora, io.buildah.version=1.39.2, org.opencontainers.image.url=https://fedoraproject.org/, version=41, vendor=Fedora Project, license=MIT, org.opencontainers.image.version=41, PODMAN_SYSTEMD_UNIT=sleep.service)
May 04 19:10:35 localhost sleep[1357401]: time="2025-05-04T18:10:35Z" level=error msg="unable to get systemd connection to add healthchecks: dial unix /run/user/1001/systemd/private: connect: connection refused"
May 04 19:10:35 localhost podman[1357401]: 2025-05-04 18:10:35.064615236 +0000 UTC m=+0.081326333 container init 1b680d910228f7b0bab31854c23b888f763a0d09a7a8bd052da1aa86a947ca94 (image=registry.fedoraproject.org/fedora:latest, name=systemd-sleep, org.opencontainers.image.version=41, org.opencontainers.image.url=https://fedoraproject.org/, PODMAN_SYSTEMD_UNIT=sleep.service, io.buildah.version=1.39.2, name=fedora, org.opencontainers.image.license=MIT, vendor=Fedora Project, version=41, license=MIT, org.opencontainers.image.vendor=Fedora Project, org.opencontainers.image.name=fedora)
May 04 19:10:35 localhost sleep[1357401]: time="2025-05-04T18:10:35Z" level=error msg="unable to get systemd connection to start healthchecks: dial unix /run/user/1001/systemd/private: connect: connection refused"
May 04 19:10:35 localhost podman[1357401]: 2025-05-04 18:10:35.066139202 +0000 UTC m=+0.082850289 container start 1b680d910228f7b0bab31854c23b888f763a0d09a7a8bd052da1aa86a947ca94 (image=registry.fedoraproject.org/fedora:latest, name=systemd-sleep, PODMAN_SYSTEMD_UNIT=sleep.service, org.opencontainers.image.license=MIT, license=MIT, org.opencontainers.image.vendor=Fedora Project, org.opencontainers.image.version=41, org.opencontainers.image.url=https://fedoraproject.org/, vendor=Fedora Project, io.buildah.version=1.39.2, name=fedora, org.opencontainers.image.name=fedora, version=41)
May 04 19:10:35 localhost podman[1357401]: 2025-05-04 18:10:34.999500036 +0000 UTC m=+0.016211133 image pull 9f3411e5c4ba5e876f6ca8b3db96f536973b410a6a425ec30b9bd7445d984008 registry.fedoraproject.org/fedora:latest

Describe the results you received

The line:
May 04 19:10:35 localhost sleep[1357401]: time="2025-05-04T18:10:35Z" level=error msg="unable to get systemd connection to add healthchecks: dial unix /run/user/1001/systemd/private: connect: connection refused"

Describe the results you expected

Not that.

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 99.11
    systemPercent: 0.17
    userPercent: 0.71
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 1923
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.13.6-200.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 24338124800
  memTotal: 66508546048
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250217.ga1e48a0-2.fc41.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 83h 5m 7.00s (Approximately 3.46 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /var/home/localuser/.config/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 15
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/localuser/.local/share/containers/storage
  graphRootAllocated: 2029682122752
  graphRootUsed: 884576428032
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 449
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /var/home/localuser/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.1
  BuildOrigin: Fedora Project
  Built: 1741651200
  BuiltTime: Tue Mar 11 00:00:00 2025
  GitCommit: b79bc8afe796cba51dd906270a7e1056ccdfcf9e
  GoVersion: go1.23.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

fedora:fedora/x86_64/coreos/stable
                  Version: 41.20250315.3.0 (2025-04-01T17:19:58Z)

$ env | grep XDG
XDG_SEAT=seat0
XDG_SESSION_TYPE=wayland
XDG_SESSION_CLASS=user
XDG_VTNR=1
XDG_SESSION_ID=1
XDG_RUNTIME_DIR=/run/user/1001

Additional information

I see a handful of related issues over the past few months, but nothing as tight to reproduce, and nothing with this connection error that I can spot.

I... am just a bit unclear what I need to change in my environment to resolve?

Thanks for reading.

[Honny1]

@ToBeReplaced, can you check if your user's systemd instance is active?

systemctl --user status

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[mheon]

I strongly suspect it's not. You'll likely need to loginctl enable-linger for the user running Quadlet.

[ToBeReplaced]

Unfortunately, my journalctl-foo is not strong enough to find confirmatory evidence; is there a way to retroactively check whether the instance was active?

Linger was disabled, and after a reboot, the health check works as expected / no corresponding error.

Closing, since the issue is resolved, but if you can think of a way to verify that this was in fact the problem via logs, I'd love to check.

[ToBeReplaced]

It is reocurring; In all cases when the log line first appears, the following occurred:

1. Some container runs and is removed.
2. systemd is reloaded immediately after.
3. A systemd service with a healthcheck from a quadlet is started immediately after.

All subsequent service starts with healthchecks have the log line until I run systemctl --user daemon-reexec, which makes the problem disappear, but systemctl status doesn't show anything wrong other than the failed service with the healthcheck and it's corresponding healthcheck service.

I will reopen, as the root cause is still unknown, but feel free to close it for not-enough-info. If I can cause a reliable reproduction I can reopen again. I am only reopening now in case the above description gives someone insight into a possible cause. Also because this is Podman 5.4.1 and I see some changes to quadlets and healthchecks since then so I don't want to waste anyone's time if this is plausibly already resolved.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.