[zstd:chunked, rootless] `staging a partially-pulled layer: checksum mismatch`

[gucci-on-fleek]

I'm unable to pull zstd:chunked container images built on a composefs host onto a non-composefs host. Non-zstd:chunked container images work fine, and the zstd:chunked images work fine on the composefs host. I started getting this issue around the same time that #2191 was opened; I assumed that #2194 would fix it, but it didn't.

Reproduction

Build the images

$ mount | grep 'on / '
composefs on / type overlay (ro,relatime,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)

$ podman --version
podman version 5.4.2

$ cat > Containerfile <<EOF
FROM scratch
COPY /usr/bin/busybox /usr/bin/busybox
CMD ["/usr/bin/busybox", "echo", "Hello, World!"]
EOF

$ podman build --no-cache --disable-compression --file=./Containerfile --tag=maxchernoff.ca/composefs-zstd-chunked /
$ skopeo copy --dest-tls-verify=false --dest-compress-format=zstd:chunked containers-storage:maxchernoff.ca/composefs-zstd-chunked:latest docker://localhost:23719/composefs-zstd-chunked:zstd-chunked

$ podman build --no-cache --disable-compression --file=./Containerfile --tag=maxchernoff.ca/composefs-zstd-chunked /
$ skopeo copy --dest-tls-verify=false --dest-compress-format=zstd containers-storage:maxchernoff.ca/composefs-zstd-chunked:latest docker://localhost:23719/composefs-zstd-chunked:zstd

$ podman build --no-cache --disable-compression --file=./Containerfile --tag=maxchernoff.ca/composefs-zstd-chunked /
$ skopeo copy --dest-tls-verify=false --dest-compress-format=gzip containers-storage:maxchernoff.ca/composefs-zstd-chunked:latest docker://localhost:23719/composefs-zstd-chunked:gzip

My container registry is public, so feel free to pull from it.

Run the images on composefs

$ mount | grep 'on / '
composefs on / type overlay (ro,relatime,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)

$ podman --version
podman version 5.4.2

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:zstd-chunked
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked...
Getting image source signatures
Copying blob f10b285e2625 skipped: already exists
Copying config 66342d349b done   |
Writing manifest to image destination
Hello, World!

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:zstd
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd...
Getting image source signatures
Copying blob 3b2700f8dc1a skipped: already exists
Copying config 26e0464471 done   |
Writing manifest to image destination
Hello, World!

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:gzip
Trying to pull maxchernoff.ca/composefs-zstd-chunked:gzip...
Getting image source signatures
Copying blob ed4c195b07bd skipped: already exists
Copying config c814e33ebf done   |
Writing manifest to image destination
Hello, World!

podman info

host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 96.63
    systemPercent: 1.54
    userPercent: 1.82
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: iot
    version: "42"
  eventLogger: journald
  freeLocks: 2047
  hostname: maxchernoff.ca
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.14.5-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2463440896
  memTotal: 8309497856
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250507.geea8a76-1.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16898842624
  swapTotal: 16898842624
  uptime: 0h 29m 11.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:23719:
    Blocked: false
    Insecure: true
    Location: localhost:23719
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:23719
    PullFromMirror: ""
  maxchernoff.ca:
    Blocked: false
    Insecure: true
    Location: localhost:23719
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: maxchernoff.ca
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/max/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/max/.local/share/containers/storage
  graphRootAllocated: 261466619904
  graphRootUsed: 139354480640
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 24
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/max/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Fedora Project
  Built: 1743552000
  BuiltTime: Tue Apr  1 18:00:00 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.24.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Run the images on not composefs

$ mount | grep 'on / '
/dev/nvme1n1p3 on / type btrfs (rw,noatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)

$ podman --version
podman version 5.4.2

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:zstd-chunked
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked...
Getting image source signatures
Error: partial pull of blob sha256:f10b285e2625d1374b4383ebd7caaa9ab7557e1f5b5337bab4cb490c009a02df: staging a partially-pulled layer: checksum mismatch for "/usr/bin/busybox" (got "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" instead of "sha256:ac4814e3ed58c1d53367f0705f7026bd0bb672a536cb07cb5c9d96f9d3fcfaed")

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:zstd
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd...
Getting image source signatures
Copying blob 3b2700f8dc1a done   |
Copying config 26e0464471 done   |
Writing manifest to image destination
Hello, World!

$ podman run --pull=newer -it --rm maxchernoff.ca/composefs-zstd-chunked:gzip
Trying to pull maxchernoff.ca/composefs-zstd-chunked:gzip...
Getting image source signatures
Copying blob ed4c195b07bd done   |
Copying config c814e33ebf done   |
Writing manifest to image destination
Hello, World!

podman info

host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 97.11
    systemPercent: 0.89
    userPercent: 2
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2048
  hostname: max-new-laptop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.14.5-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1220481024
  memTotal: 32949731328
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250507.geea8a76-1.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 42949029888
  swapTotal: 42949660672
  uptime: 15h 3m 26.00s (Approximately 0.62 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/max/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/max/.local/share/containers/storage
  graphRootAllocated: 1985756528640
  graphRootUsed: 1291860529152
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/max/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Fedora Project
  Built: 1743552000
  BuiltTime: Tue Apr  1 18:00:00 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.24.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

[mtrmac]

FWIW:

Error: partial pull of blob sha256:f10b285e2625d1374b4383ebd7caaa9ab7557e1f5b5337bab4cb490c009a02df: staging a partially-pulled layer: checksum mismatch for "/usr/bin/busybox" (got "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" instead of "sha256:ac4814e3ed58c1d53367f0705f7026bd0bb672a536cb07cb5c9d96f9d3fcfaed")

ac4814e3ed58c1d53367f0705f7026bd0bb672a536cb07cb5c9d96f9d3fcfaed is a correct digest of the file as it exists in the traditional tar view (and the contest are a plausible ELF executable).

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is a sha256 digest of an empty input, so this looks like a logic error of some kind.

Just reading the code, I can’t see anything — but I didn’t actually try running the pull of the zstd:chunked view.

[gucci-on-fleek]

@mtrmac

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is a sha256 digest of an empty input, so this looks like a logic error of some kind.

Ah, yes indeed; good catch.

Just reading the code, I can’t see anything — but I didn’t actually try running the pull of the zstd:chunked view.

Your comment prompted me to look into this again, and for some odd reason, the minimal maxchernoff.ca/composefs-zstd-chunked:zstd-chunked example seems to work now??? But pulling any of my other (non-trivial) container images still fails with the exact same error. The composefs-zstd-chunked test images are unmodified from when I first posted this issue (podman pull shows that they have the same digests), but the other images are all updated daily. And they all show the exact same sha256(null) digest, so that's definitely a common factor here:

$ podman --version
podman version 5.5.0

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/max/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y

$ podman pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked...
Getting image source signatures
Copying blob f10b285e2625 done  849.4KiB / 849.4KiB (skipped: 201.0b = 0.02%)
Copying config 66342d349b done   |
Writing manifest to image destination
66342d349b933459ecb51902f31592d5231e8b2f6729319988aa301b11b0e670

$ podman pull maxchernoff.ca/composefs-zstd-chunked:zstd
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd...
Getting image source signatures
Copying blob 3b2700f8dc1a done   |
Copying config 26e0464471 done   |
Writing manifest to image destination
26e04644713c8583b07e37dd18f1f3f1696813c0dfdc754612c3131133e0e743

$ podman pull maxchernoff.ca/composefs-zstd-chunked:gzip
Trying to pull maxchernoff.ca/composefs-zstd-chunked:gzip...
Getting image source signatures
Copying blob ed4c195b07bd done   |
Copying config c814e33ebf done   |
Writing manifest to image destination
c814e33ebf2a95a692d72da721aa5899a605c23834aa239329600f68e679e407

$ podman pull maxchernoff.ca/flask:latest
Trying to pull maxchernoff.ca/flask:latest...
Getting image source signatures
Error: internal error: unable to copy from source docker://maxchernoff.ca/flask:latest: partial pull of blob sha256:ca3f97c90aeb845e2ea2edd55cb276d775d2ed21f43fa1651c70ca4b0557102d: staging a partially-pulled layer: checksum mismatch for "/etc/aliases" (got "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" instead of "sha256:a4c569569f893bc22fbe696c459f8fba0fe4565022637300b705b54a95c47bce")

$ podman pull maxchernoff.ca/caddy:latest
Trying to pull maxchernoff.ca/caddy:latest...
Getting image source signatures
Copying blob 27c12a73b40d done  302.0b / 302.0b (skipped: 105.0b = 34.77%)
Error: internal error: unable to copy from source docker://maxchernoff.ca/caddy:latest: partial pull of blob sha256:4894bd06bfd139a00c7aa0d711b272d26a15e3483597010a4da7d947773ee83c: staging a partially-pulled layer: checksum mismatch for "/bin/busybox" (got "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" instead of "sha256:7da62d35e17190bb032a55c5e0390e0d832cb2f3f7fc5c4e83ae48fff471f2d6")

$ podman pull maxchernoff.ca/tex:latest
Trying to pull maxchernoff.ca/tex:latest...
Getting image source signatures
Error: internal error: unable to copy from source docker://maxchernoff.ca/tex:latest: partial pull of blob sha256:4e2ac1dd0d588ee3e085b8a118fb258f91bbf23ba1f06d311fa1796e776b919f: staging a partially-pulled layer: checksum mismatch for "/.profile" (got "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" instead of "sha256:ff7bed69c2ac8dc64e6bf8ae6da026a0f906e00c4f954064bc857b4a1eafdad5")

$ printf | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -

[mtrmac]

FYI @giuseppe — this is not directly related to the convert_images work, but is probably in the same area around appendCompressedStreamToFile etc.

[giuseppe]

@gucci-on-fleek I am not able to reproduce locally using the images you've provided above (and starting from a fresh storage), have you done any change to the storage.conf file?

[gucci-on-fleek]

@giuseppe

I am not able to reproduce locally using the images you've provided above (and starting from a fresh storage),

Huh, I tried the tests from #2332 (comment) again just now and everything seems to work properly:

$ podman --version
podman version 5.5.0

$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/home/max/.local/share/containers/storage"
        - the runRoot directory: "/run/user/1000/containers"
Are you sure you want to continue? [y/N] y

$ podman pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd-chunked...
Getting image source signatures
Copying blob f10b285e2625 done  849.4KiB / 849.4KiB (skipped: 201.0b = 0.02%)
Copying config 66342d349b done   | 
Writing manifest to image destination
66342d349b933459ecb51902f31592d5231e8b2f6729319988aa301b11b0e670

$ podman pull maxchernoff.ca/composefs-zstd-chunked:zstd
Trying to pull maxchernoff.ca/composefs-zstd-chunked:zstd...
Getting image source signatures
Copying blob 3b2700f8dc1a done   | 
Copying config 26e0464471 done   | 
Writing manifest to image destination
26e04644713c8583b07e37dd18f1f3f1696813c0dfdc754612c3131133e0e743

$ podman pull maxchernoff.ca/composefs-zstd-chunked:gzip
Trying to pull maxchernoff.ca/composefs-zstd-chunked:gzip...
Getting image source signatures
Copying blob ed4c195b07bd done   | 
Copying config c814e33ebf done   | 
Writing manifest to image destination
c814e33ebf2a95a692d72da721aa5899a605c23834aa239329600f68e679e407

$ podman pull maxchernoff.ca/flask:latest
Trying to pull maxchernoff.ca/flask:latest...
Getting image source signatures
Copying blob 6dc30020b08e done  34.8MiB / 34.8MiB (skipped: 13.1KiB = 0.04%)
Copying config cda0ac94df done   | 
Writing manifest to image destination
cda0ac94dfa0d553035f1f770ec054e1917cc1801717608792e74503df10cc94

$ podman pull maxchernoff.ca/caddy:latest
Trying to pull maxchernoff.ca/caddy:latest...
Getting image source signatures
Copying blob ac56b9f3fd95 done  15.9MiB / 15.9MiB (skipped: 313.0b = 0.00%)
Copying blob 4894bd06bfd1 done  3.7MiB / 3.7MiB (skipped: 6.1KiB = 0.16%)
Copying blob d8ff28361f4f done  8.9KiB / 8.9KiB (skipped: 288.0b = 3.15%)
Copying blob f61a989c86eb done  14.9MiB / 14.9MiB (skipped: 318.0b = 0.00%)
Copying blob 27c12a73b40d done  302.0b / 302.0b (skipped: 105.0b = 34.77%)
Copying blob 44b55ce83bbd done  448.3KiB / 448.3KiB (skipped: 179.8KiB = 40.12%)
Copying config e78331e83b done   | 
Writing manifest to image destination
e78331e83bb7eeb09a8985e2a0734481054fb653fa4f1aaa0ee0e160974dc7f0

$ podman pull maxchernoff.ca/tex:latest
Trying to pull maxchernoff.ca/tex:latest...
Getting image source signatures
Copying blob 93584183b0dd done  190.8MiB / 190.8MiB (skipped: 17.1MiB = 8.98%)
Copying blob 985080596b1b done  50.8MiB / 50.8MiB (skipped: 13.5MiB = 26.63%)
Copying blob 5fafecf00b39 done  2.1KiB / 2.1KiB (skipped: 262.0b = 12.10%)
Copying blob c5e9f7a4cfe1 done  2.2KiB / 2.2KiB (skipped: 263.0b = 11.91%)
Copying blob c5f18b27fa87 done  124.3MiB / 124.3MiB (skipped: 670.0b = 0.00%)
Copying config 793aad3022 done   | 
Writing manifest to image destination
793aad30221539a9a1d6c09ba18f0855c48951388b654e7c0a62a902135daa99

So problem solved I guess; thanks!

have you done any change to the storage.conf file?

Nope. Not on the composefs system and not on my non-composefs system:

$ ls -lAt /etc/containers/ ~/.config/containers/
/home/max/.config/containers/:
total 4
drwxr-xr-x. 1 max max  14 May  9 14:54 podman/
drwxr-xr-x. 1 max max 174 May  9 14:53 systemd/
-rw-r--r--. 1 max max 119 Dec 16 02:04 storage.conf.old

/etc/containers/:
total 16
-rw-r--r--. 1 root root  699 Jun  2 18:00 toolbox.conf
-rw-r--r--. 1 root root   30 May 16 17:30 containers.conf
drwxr-xr-x. 1 root root    0 Apr 16 18:00 certs.d/
drwxr-xr-x. 1 root root    0 Apr 16 18:00 networks/
drwxr-xr-x. 1 root root   14 Apr 16 18:00 oci/
-rw-r--r--. 1 root root 3935 Apr 16 18:00 registries.conf
drwxr-xr-x. 1 root root   38 Apr 16 18:00 registries.conf.d/
drwxr-xr-x. 1 root root  132 Apr 16 18:00 registries.d/
drwxr-xr-x. 1 root root    0 Apr 16 18:00 systemd/
-rw-r--r--. 1 root root  256 May 23  2024 policy.json