Description
Describe the bug
We are running a Docker server for hosting an application. This application requires a fair number of cron jobs to run on a frequent schedule. These cron jobs are run within a Docker container using docker compose run --rm -T <service> <command>. We have noticed that this causes a memory leak in the Loki driver.
As you can see in this memory usage graph from our Grafana instance, the memory usage rises at a steady rate. Each time it drops, we have restarted the Docker service. As you can see, after the last drop, the line remains quite horizontal. This is when we switched from using docker compose run to docker compose exec, effectively running the cron jobs inside a running container. This seems to mitigate the issue. The only drawback, however, is that the console output is not sent to Loki.
We have identified the Loki driver as the culprit by analyzing the output of a ps faux command on the Docker host. In this output we found the following output:
<redacted> 802396 0.0 0.0 1238360 14616 ? Sl May12 0:44 \_ /usr/bin/containerd-shim-runc-v2 -namespace plugins.moby -id a9412b1b92456373da8a797a9ddcf97f30f3efd4699d0fe43ad672bb0d46bc22 -address /run/user/1001/docker/containerd/containerd.sock
<redacted> 802422 6.7 0.3 1391068 113336 ? Ssl May12 315:53 | \_ /bin/docker-driver
The hash shown in the output matches the hash shown besides the loki driver when running docker plugin ls. In the output, you'll see the process currently uses 6.7% CPU and 0.3% memory. This is the current usage, so it's within the expected range. It used to go well beyond 25% (of a total of 32GB available) before we switched from run to exec. It seems that when running lots of short lived containers, traces of these containers (logs?) remain in memory, even though the containers finish running and are removed (the --rm flag is passed).
To Reproduce
This may be a tough one. Start a Docker instance, configure the Loki driver, create a compose project and have the crontab run a command within a service container on a high frequency (something like docker compose run --rm -T myservice ls -al). Switch to docker compose exec to see that this impacts memory usage a lot less.
Expected behavior
I would expect the memory usage not to continually rise, even when using docker compose run to run short lived containers.
Environment:
- Debian 12.10
- Docker 28.0.4
- Loki driver 3.4.2 (AMD64)
Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.