pod checkpoint/restore - incomplete restore?

[edsantiago]

new e2e test flake:

           [It] podman checkpoint and restore container out of and into pod (net,uts,pid)
...
           # podman [options] pod create --name test_pod --share net,uts,pid
           2d3a6acebc57859090f43a10ada336d7112ab1370e9b58d0147e8b31e5bafc1b
           # podman [options] container restore --pod 2d3a6acebc57859090f43a10ada336d7112ab1370e9b58d0147e8b31e5bafc1b -i /tmp/CI_q0NH/podman-e2e-162141281/subtest-687432410/checkpoint-e0242610380e2cacb60e6b5b344a4762ddc945a6eb0f968bb9cfbb2b31dec9ec.tar.gz
           e0242610380e2cacb60e6b5b344a4762ddc945a6eb0f968bb9cfbb2b31dec9ec
           # podman [options] ps -q
           5a3fadf99b88
...
   FAILS - expected two lines of output

Seen twice in the last few days. It could easily be a bug in the test itself: please note that pod checkpoint/restore is new as of crun-1.18. This is a test that has been skipped for years, except for runc.

• fedora-41 : int podman fedora-41 root host sqlite
  • PR [v5.3] Fix for podman machine init not creating necessary JSON file when an ignition-path is passed #24560
    • 11-14 06:57 in Podman checkpoint podman checkpoint and restore container out of and into pod (net,uts,pid)
  • PR fix(deps): update module github.com/moby/sys/capability to v0.4.0 #24534
    • 11-11 17:09 in Podman checkpoint podman checkpoint and restore container out of and into pod (net,uts,pid)

x	x	x	x	x	x
int(2)	podman(2)	fedora-41(2)	root(2)	host(2)	sqlite(2)

[edsantiago]

This actually reproduces reliably on my f40 laptop with crun-1.18.2-1 from updates-testing. I think this is only working in CI by some miracle of slow-CI-VM timing. 100% reliable reproducer:

#!/bin/bash

set -e

# Clean slate
bin/podman rm -f -a -t0
bin/podman pod rm -a

bin/podman pod create --name foo --share net,uts,pid
bin/podman run -d --pod foo --name c_foo quay.io/libpod/testimage:20241011 top
bin/podman container checkpoint -e /tmp/foo.tar.gz c_foo

# Create a new pod
bin/podman pod rm foo
bin/podman pod create --name foo --share net,uts,pid
bin/podman container restore --pod foo -i /tmp/foo.tar.gz

# There should be two containers running: infra and top.
echo
bin/podman ps -a

Invocation:

$ sudo ./foo.sh
775e8b2b618eec71b4be130605b5a854ab6c6493c1e7b7c082a75b88bb7ae991
799179c00f1283c30e0620dddcf5baeca4a3ee7fbeaa624fbe57c5293f16fc0f
e905309a80bd8bb6fe4a8f8d528e4e664a12a57bc787e02d895e26b8ed33f04e
c_foo
799179c00f1283c30e0620dddcf5baeca4a3ee7fbeaa624fbe57c5293f16fc0f
40ec4511d1d451704b86ce4d8daa85c358913c6f7e718a9bb5edf1a34888d848
e905309a80bd8bb6fe4a8f8d528e4e664a12a57bc787e02d895e26b8ed33f04e

CONTAINER ID  IMAGE                                        COMMAND     CREATED                 STATUS                             PORTS       NAMES
f831fcc8620c  localhost/podman-pause:5.4.0-dev-1731614350              Less than a second ago  Up Less than a second                          40ec4511d1d4-infra
e905309a80bd  quay.io/libpod/testimage:20241011            top         Less than a second ago  Exited (1) Less than a second ago              c_foo

...and then what I think might be a hint:

$ sudo bin/podman logs e905
top: can't open '/proc': Permission denied

@rst0git PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rst0git]

I was able to replicate this error locally and it looks like crun doesn't use the correct SELinux mount context when creating mounts such as /proc for the restored container.

I will try to open a pull request with a fix for this problem soon.

cc: @giuseppe

[rst0git]

@edsantiago The same error occurs with both runc and crun.

It looks like Podman (incorrectly) restores the SELinux labels from the checkpoint, instead of specifying the SELinux labels of the new Pod for the restored container.

[Luap99]

This is still happening quite often in CI.
https://api.cirrus-ci.com/v1/artifact/task/5498409928359936/html/int-podman-rawhide-root-host-sqlite.log.html#t--Podman-checkpoint-podman-checkpoint-and-restore-container-out-of-and-into-pod--net-uts-pid---1

@rst0git Do you ever worked on patch for this? sounds like you already found the root cause.

[Luap99]

Test is skipped with 0849cbc as it is flaking to much, if someone fixes the bug please make sure to remove the skip again.