[quadlet] quadlets in /etc/containers/systemd/<uid> start for every user if <uid> is a symlink

[rmsc]

Issue Description

I have a server with several quadlets in /etc/containers/systemd/<uid>/, some at the root, but most of them grouped in subdirectories.

Today I found out (in a very bad way) that if all the quadlets are in subdirectories of <uid>, then the enabled quadlets are started by ALL users. It's as if they were in /etc/containers/systemd/users/.

I found out that the enabled quadlets in /etc/containers/systemd/<uid> are being started for ALL users, as if they were in /etc/containers/systemd/users. I suspect this may be caused by the fact that <uid> is a symlink to a read-only directory in /opt.

This is happening with podman 5.5.0, running in a custom bootc distro based on fedora-bootc:42.

Steps to reproduce the issue

$ ls /etc/containers/systemd/users/ -l
lrwxrwxrwx. 1 root root 24 May 31 21:58 1000 -> /opt/somedir

Describe the results you received

The quadlets are started by ALL users. There are conflicts on network ports, and the whole server became unusable.

Describe the results you expected

The quadlets should be started only by the user with that UID.

podman info output

host:
  arch: amd64
  buildahVersion: 1.40.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 49.19
    systemPercent: 8.65
    userPercent: 42.16
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    version: "42"
  eventLogger: journald
  freeLocks: 2039
  hostname: hz01.silkmicro.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 589824
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 589824
      size: 65536
  kernel: 6.14.8-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2541277184
  memTotal: 3994800128
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.1
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250512.g8ec1341-1.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2h 21m 44.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/weaver/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/weaver/.local/share/containers/storage
  graphRootAllocated: 39255314432
  graphRootUsed: 16783691776
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /var/home/weaver/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.0
  BuildOrigin: Fedora Project
  Built: 1747180800
  BuiltTime: Wed May 14 00:00:00 2025
  GitCommit: 0dbcb51477ee7ab8d3b47d30facf71fc38bb0c98
  GoVersion: go1.24.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Custom bootc distro based on fedora-bootc:42

[rmsc]

After reverting the change, I realized that the quadlets were still starting for all users.

I suspect putting everything in subdirectories changed the timing somewhat, and the "wrong" user claimed the ports before the "right" user did. Reverting made the "right" user get all ports, probably by luck.

In my particular case, /etc/containers/systemd/<uid> was a symlink to a read-only directory in /opt/. By replacing the symlink with a copy of that directory, things got back to normal.

[ygalblum]

Thanks for reporting this. I also believe this is related to the symlink.

Scanning the code I can see that it evals the symlink here:

podman/cmd/quadlet/main.go

Line 203 in 746cbf1

resolvedPath, err := filepath.EvalSymlinks(path)

before checking if it should filter the directory here:

podman/cmd/quadlet/main.go

Line 214 in 746cbf1

if skipPath(paths, resolvedPath, isUserFlag, filterPtr) {

Currently, skipPath function handles both revisit filtering (for which the symlink eval is needed) and the UID filter check (which the symlink eval breaks).

[rmsc]

Thanks for looking into this!

I have a question regarding the current implementation. Is there a technical reason for only supporting UIDs, or could it be made to work with usernames as well?

I ask this because using UIDs forces me to "do a dance" when adding the quadlets via Containerfile. I know the username a-priori, but not the UID, so the only way to name that directory is via RUN:

RUN ln -s /opt/quadlets /etc/containers/systemd/$(uid -u myuser)
# or
RUN mv /etc/containers/systemd/rename_me /etc/containers/systemd/$(uid -u myuser)

[ygalblum]

The implementation is pretty simple.
First it finds the absolute path of /etc/containers/systemd/users (as it might be a symlink as well). Then it traverses it's internal directories while filtering out all directories that match the ^[0-9]*$ regex. Afterwards it adds the everything under /etc/containers/systemd/users/<UID>.
In your case the filtering does not work because if first translates the symlink of the directory causing it to not match the regex.

The problem with usernames is that it will be harder (or even worst) to filter out based on usernames as nothing stops you from creation a directory for a user that does not exist and while expecting it will be executed when it is and filtered out for the rest.

[rmsc]

Thanks, I understand now.

Off-topic, but perhaps supporting something like this could work?

• /etc/containers/systemd/users for all users
• /etc/containers/systemd/user/<username> for specific users

For bootc and other immutable OS's, supporting something like this in /usrwould be even better.

[ygalblum]

For backward compatibility, I would not change the support for /etc/containers/systemd/users/<UID>. But, we can look into making /etc/containers/systemd/user/<username> a special case.

As for /usr, I can see that for rootful quadlets, we also look at /usr/share/containers/systemd. I don't remember why we didn't add the support for a users directory in it. If we did, would that solve this requirement?

[rmsc]

Both would be great, thank you!

[giuseppe]

The problem with usernames is that it will be harder (or even worst) to filter out based on usernames as nothing stops you from creation a directory for a user that does not exist and while expecting it will be executed when it is and filtered out for the rest.

Would it be a breaking change to consider only files under /etc/containers/systemd/users/ for all users?

I find it confusing that we traverse the whole hierarchy, that would make it simple to deal with /etc/containers/systemd/users/<UID> and /etc/containers/systemd/users/<USERNAME>. Or am I missing the root problem here?

[rmsc]

Would it be a breaking change to consider only files under /etc/containers/systemd/users/ for all users?

I'm personally making use of subdirectories to organize different services, so such a change would definitely break things on my end. I think it's safe to assume other people are using them as well.

See #24783

[benniekiss]

As for /usr, I can see that for rootful quadlets, we also look at /usr/share/containers/systemd. I don't remember why we didn't add the support for a users directory in it. If we did, would that solve this requirement?

I was about to make a feature request for user quadlets in /usr/share/containers/systemd/user, mainly for bootc configuration. I can still go ahead and create an issue, but I would +1 this feature otherwise. It should also support the users/{uid} path logic.

On a side note, I didn't know that the /etc/containers/systemd/{uid} path was a way to restrict quadlets per user, I thought it was just limited to /etc/containers/systemd/users/{uid}. This should be added to the documentation if that's the case.

I'll also add that I rely on using subfolders under /etc/containers/systemd/users/ to organize my quadlets (each pod+containers gets its own folder), so I would like to see that functionality remain.