Description
I have both zstd:chunked and composefs enabled in my rootless storage (so really right now, just zstd:chunked), i.e. in my /etc/containers/storage.conf:
[storage.options.pull_options]
enable_partial_images = "true"
convert_images = "true"
[storage.options.overlay]
use_composefs = "true"
Recently this seems to have basically started reliably failing to some images. Now here's the thing...because of bootc requirements 90% of my podman usage is actually podman-machine where none of this is enabled by default, so I don't have a rough sense of how recently this actually broke. (The non-podman-machine rootless use cases are mostly toolbox containers for me)
$ Client: Podman Engine
Version: 5.3.1
API Version: 5.3.1
Go Version: go1.23.3
Built: Wed Nov 20 19:00:00 2024
OS/Arch: linux/amd64
$ podman --log-level debug pull -q quay.io/rhel-devel-tools/rhel-developer-toolbox
...
DEBU[0002] Downloading /v2/rhel-devel-tools/rhel-developer-toolbox/blobs/sha256:65370aed77e699049343d487c0cb354188d87709caddece2324eedca262c8d1f
DEBU[0002] GET https://quay.io/v2/rhel-devel-tools/rhel-developer-toolbox/blobs/sha256:65370aed77e699049343d487c0cb354188d87709caddece2324eedca262c8d1f
INFO[0002] Error loading cache file for layer "345778062ceff8d733b1b965f15736af7afb36d5ae484b313ce88bf1021a3404": EOF
DEBU[0002] Written lookaside cache for layer "345778062ceff8d733b1b965f15736af7afb36d5ae484b313ce88bf1021a3404" with length 68
INFO[0002] Error loading cache file for layer "8991c80d55c1b1cb5c97b1f9ebab59a40595b30b529f2ed15ddf5e99809c67ab": EOF
DEBU[0002] Written lookaside cache for layer "8991c80d55c1b1cb5c97b1f9ebab59a40595b30b529f2ed15ddf5e99809c67ab" with length 68
DEBU[0002] Applying differ in /var/home/walters/.local/share/containers/storage/overlay/staging/1902067082/dir
DEBU[0002] Downloading /v2/rhel-devel-tools/rhel-developer-toolbox/blobs/sha256:a17582e99e9e72dad1792fc6b6a54ee3ab42e5dd76961a2fc41477765b507073
DEBU[0002] GET https://quay.io/v2/rhel-devel-tools/rhel-developer-toolbox/blobs/sha256:a17582e99e9e72dad1792fc6b6a54ee3ab42e5dd76961a2fc41477765b507073
DEBU[0026] Missing 456022165 bytes out of 1090565019 (41.82 %)
DEBU[0026] Retrieved partial blob sha256:41a322248f00a9d001b4ee2f566854b116b68ee6e92382cdd7ae02867a1f10f7
DEBU[0054] Error pulling candidate quay.io/rhel-devel-tools/rhel-developer-toolbox:latest: copying system image from manifest list: partial pull of blob sha256:f03b41672556b172dd0cd9cc881f4bcda546abeaf5b22fe794c1af5409d89ca7: staging a partially-pulled layer: mkdirat /: no such file or directory
Error: copying system image from manifest list: partial pull of blob sha256:f03b41672556b172dd0cd9cc881f4bcda546abeaf5b22fe794c1af5409d89ca7: staging a partially-pulled layer: mkdirat /: no such file or directory
Other images seem to pull fine, this is the first I have seen this error. In fact there are zero hits for "staging a partially-pulled layer" (with quotes, i.e. literally) on Google, so I appear to be lucky. Probably almost no one is enabling convert_images = "true" which is probably what is triggering this if I had to guess (all the input images are plain tar+gzip of course).
Ahh wait, it also now fails with fedora-bootc:42 - full log.txt.
Commenting out the convert_images = true indeed makes things work.