Spawning hundreds of thousands files in emptyDir makes kubelet unable to restart

[Diegunio]

Issue:
The main issue is that after a very large number of files are created in the emptyDir, the kubelet on that node is unable to restart. The service fails due to an "error" in restorecon, which is executed as a PreStart dependency in the kubelet.service unit

Initially, I used git clone inside a container, which writes files to an emptyDir. However, I discovered that the problem wasn't related to git clone itself but rather the large number of files appearing in the emptyDir. After all files are created in the container, I enter with ssh into the node where the emptyDir was mounted and attempt to restart the kubelet. Every time, the restart fails, and the service logs only mention SELinux denials for files created in the container.

I’ve determined that the kubelet’s ability to restart is dependent on how fast the node’s hardware is. Slower nodes fail when trying to process around 400,000 files. Faster nodes handle that, but even they fail when the file count reaches 900,000.

Version:
UPI 4.18.0-okd-scos.8
registry.ci.openshift.org/origin/release-scos@sha256:de900611bc63fa24420d5e94f3647e4d650de89fe54302444e1ab26a4b1c81c6

Issue Behavior:
The issue always occurs and can be reproduced every time.

How to reporduce:

1. Create any container that spawns hundreds of thousands files to an emptyDir (make sure to note the node on which the pod is created).

Example of container that spawns many files

apiVersion: apps/v1
kind: Deployment
metadata:
  name: repo-cloner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: repo-cloner
  template:
    metadata:
      labels:
        app: repo-cloner
    spec:
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/hostname: worker-4.dev.example.com
      securityContext:
        fsGroupChangePolicy: Always
      volumes:
        - name: repo-storage
          emptyDir: {}
      containers:
        - name: containerasfa
          securityContext:
            capabilities:
              drop:
                - ALL
            privileged: false
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
          image: docker.io/alpine/git:latest
          command:
            - sh
            - -c
            - |
                echo "Generating  files..." && \
                mkdir -p /data/files && \
                seq 1 900000 | xargs -I {} sh -c 'echo "content" > /data/files/file_{}.txt' && \
                echo "Done." && \
                sleep 22222
          volumeMounts:
            - name: repo-storage
              mountPath: /data

2. Log into the node and execute the following command:

systemctl restart kubelet.service

The result should be that the kubelet fails to start due to issues with the directory containing the data.

Example kubelet service log (in practice, this is just one repeating log for various files):

May 07 08:23:58 worker-4.dev.example.com restorecon[53570]: /var/lib/kubelet/pods/f61afe9e-7fc3-413c-8f61-xd41affe9f73/volumes/kubernetes.io~empty-dir/repo-storage/files/file_264137.txt not reset as customized by admin to system_u:object_r:container_file_t:s0:c5,c35

Troubleshooting performed:
I verified multiple times that the SELinux context on the files is correct and consistent. I compared emptyDirs from containers that succeed and fail, using:

I checked the security contexts for the emptyDir that causes the issue and one that does not perform the git clone and does not cause any issue. Both directories and files within these directories had exactly the same security contexts, verified using:
ls -lZa

#Not working
-rw-r--r--. 1 1001200000 1001200000 system_u:object_r:container_file_t:s0:c5,c35 1140 Apr 25 05:42 index.js

#Working
-rw-r--r--. 1 1001200000 1001200000 system_u:object_r:container_file_t:s0:c5,c35   2 Apr 25 05:46 nginx.pid

lsattr:

#Not working
---------------------- indexeddb.js
#Working
---------------------- nginx.pid

getfattr -d -m -:

#Not working
# file: indexeddb.js
security.selinux="system_u:object_r:container_file_t:s0:c5,c35"
#Working
# file: nginx.pid
security.selinux="system_u:object_r:container_file_t:s0:c5,c35"

All files in this directory have the same security context (the same one set in the pod under spec.securitycontext.selinuxoptions). I verified this as follows:

#Not working
ls -Z emptydir-build | cut -d':' -f5 | sort | uniq
c5,c35 u/typescript-eslint
c5,c35 ignore
c5,c35 minimatch
c5,c35 semver

#Working
ls -Z emptydir-tmp | cut -d':' -f5 | sort | uniq
c5,c35 client_temp
c5,c35 fastcgi_temp
c5,c35 nginx.pid
c5,c35 proxy_temp
c5,c35 scgi_temp
c5,c35 uwsgi_temp

  securityContext:
    seLinuxOptions:
      level: 's0:c35,c5'
    fsGroup: 1001200000
    fsGroupChangePolicy: Always
    seccompProfile:
      type: RuntimeDefault

I tried using spec.volumes.emptydir.medium: Memory in the deployment definition, but the issue still occurred.

I set the most restrictive possible security context in the pod definition, with the SCC set to restricted-v2.

Pod-level securityContext:

      securityContext:
        fsGroupChangePolicy: Always

initContainer securityContext:

          securityContext:
            capabilities:
              drop:
                - ALL
            privileged: false
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault

container securityContext:

          securityContext:
            capabilities:
              drop:
                - ALL
            privileged: false
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault

Expected behavior:

The kubelet should be able to restart regardless of how many files appear in an emptyDir. Files with valid SELinux policies should not interfere with the restart process, even when their count is extremely high.

[GingerGeek]

What an interesting bug

I know we have had a couple of issues related to SELinux within CentOS based nodes

Could you try and reproduce on the 4.19 engineering candidates?