cAdvisor Fails to Retrieve Metrics for Podman Container Managed by systemd

[dmennecke1]

Issue Description

When running an NGINX container manually using the following command:

podman run --name=nginx -e=CONTAINER_NAME=nginx -p=8081:80 --restart=always -d docker.io/nginx

cAdvisor is able to successfully retrieve and display container metrics:

container_cpu_load_average_10s{container_label_BACKEND_URL="",container_label_ERROR_FILE="",container_label_io_buildah_version="",container_label_maintainer="NGINX Docker Maintainers <docker maint@nginx.com>",container_label_org_opencontainers_image_ref_name="",container_label_org_opencontainers_image_version="",id="/user.slice/user-1001.slice/user@1001.service/user.slice/libpod-76e7df60cd24ecdabca6f5ef54b2ba87bbca060adeb0018ddb2c0017f18c50db.scope",image="docker.io/library/nginx:latest",name="nginx"} 0 1737716804296

However, when starting the container as a systemd service (based on the Quadlet feature), cAdvisor fails to retrieve metrics.

Systemd Service Configuration

The container is started using the following systemd service unit:

# Automatically generated by /usr/local/lib/systemd/user-generators/podman-user-generator
#
[X-Container]
ContainerName=nginx
Image=docker.io/nginx
PublishPort=8081:80

[Install]
WantedBy=default.target

[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
SourcePath=/etc/containers/systemd/users/1001/nginx.container
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/local/bin/podman run --name nginx --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --publish 8081:80 --env CONTAINER_NAME=nginx  docker.io/nginx

Observed Errors

When the container is started via systemd, the following cAdvisor errors appear in the logs:

E0124 12:16:27.057400       1 manager.go:1116] Failed to create existing container: /user.slice/user-1001.slice/user@1001.service/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

W0124 12:16:27.109060       1 manager.go:1169] Failed to process watch event {EventType:0 Name:/user.slice/user-1001.slice/user@1001.service/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d WatchSource:0}: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

Expected Behavior

cAdvisor should return metrics for the container when it is managed by systemd, similar to the manual Podman execution.

Actual Behavior

cAdvisor fails to collect container metrics and logs errors related to RW layer ID determination.

[w00di]

I've also been troubleshooting this and have yet to find a solid work around.

[w00di]

prometheus-podman-exporter has been a temporary replacement for just monitoring the containers.

[mstinsky]

I have the same issue while using Podman with Quadlet files.
I've tried multiple things but can't seem to get it to work.

Did any of you find a solution?

[savely-krasovsky]

@mstinsky I've prepared a fix, it's linked above. I tested it and now it properly shows all the containers.