[Support]: Out of memory: Killed process (frigate.capture) #16710
Replies: 10 comments 2 replies
-
|
Sounds similar to my long-standing issue with wireless Hikvision cameras - #10814 - though I was able to reproduce it simply by disconnecting the ethernet cable on a wired camera too. 9 times out of 10 connection loss for a camera is harmless, but 1 in 10 times it causes something to crap out. Wifi cameras I suspect are just more prone to triggering. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having oom-killer events as well. Running a simple installation, no gen ia with 3 wired cameras on a 6 GB Proxmox LXC. I'm having these issues most of the time when using the web interface , especially the video and object life cycle in the explore section are prone to do this. |
Beta Was this translation helpful? Give feedback.
-
|
It's becoming more and more of a problem to the point where I may have to drop Frigate entirely. |
Beta Was this translation helpful? Give feedback.
-
|
The above issue seems to be because docker-compose v3+ can't handle setting limits on swap usage, only memory usaged - https://docs.docker.com/engine/containers/resource_constraints/. To get around this issue I've got a script checking the memory usage of the Frigate container every 60s. If it's above 4GB memory usage the container is rebooted. This obviously isn't the best solution as if there's a recording in progress this will get interrupted. It is however the more preferred option over oom-killer events. |
Beta Was this translation helpful? Give feedback.
-
|
HI. I have just started experiencing this issue, I have been running frigate on my Lenovo m700 with 16 gb of memory for about 2 years, however I have only noticed this in the last few days I did change my docker compose file to increase the shm_size: to 128mb as I was noticing errors in the frigate log and asking to increase it. I am not sure if this is the cause I am no linux or docker expert, but do wonder if increasing this setting might be the issue This script you have, could you provide it, or any other finding you might have Thank You |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem with my Amcrest WIFI doorbell camera. Hangs with the graceful exit message. I am running both my cameras through Go2RTC and downsampling the higher-bitrate stream to 1280x960 for my detect stream (the substream is only 720x576 and doesn't give great detect results.) I have no idea what is happening and don't even really know where to look - but I've just solved my problem by watching for either of my camera feeds to die in either Frigate or Go2RTC with the following script. If it dies I'm using a go2rtc API call to restart Go2RTC (only). This has caused a Go2RTC restart to occur about 5 times a day over the last 4 days. I lose about 8 seconds of recording during that restart but that is much shorter than restarting the whole FRIGATE process. Thus far it has worked like a champ. No more "dead camera" in my otherwise working Frigate instance. Frigate is running inside an privileged Proxmox LXC container (because I'm lazy and couldn't get the Coral card passthru to work inside the LXC correctly.) The host system is an old laptop and has Plex, Immich and Frigate running on it and all are using Intel GPU (shared) for video processing. It is probably not enough resources but it works for me. Anyway - I run this shell script inside the container with root permissions and have it set up as a systemd cron job that runs ever 10 seconds and looks for the camera to be missing. If it is then it issues the API call and sends a notification to Home Assistant. I'm not sure if the sleep/wait thing is really required but I didn't want systemd to launch another watch instance while Go2RTC was restarting - so I just have it wait. This seems to kill the cron process so then I restart the cron process with the last two lines. Super kludge and I don't really know how to code. ChatGPT wrote the "jq" steps for me. But they work... #!/bin/sh
GO2RTC_URL="http://192.168.xxx.yyy:1984/api/streams"
FRIGATE_URL="http://192.168.xxx.yyy:5000/api/stats"
FRIGATE_CAMERAS_DOWN=$(curl -s $FRIGATE_URL | jq -r '.cameras | to_entries[] | select(.value.camera_fps == 0.0) | .key')
GORTC_CAMERAS_DOWN=$(curl -s $GO2RTC_URL | jq -r 'to_entries[] | select(.value.producers == null or .value.producers == []) | select(.key | test("rest_command|other_non_camera_items") | not) | .key')
switch=0
# Check if any FRIGATE cameras are down
if [ -n "$FRIGATE_CAMERAS_DOWN" ]; then
echo "Camera(s) down: $FRIGATE_CAMERAS_DOWN"
switch=$((switch+1))
fi
# Check if any GO2RTC cameras are down
if [ -n "$GO2RTC_CAMERAS_DOWN" ]; then
echo "Camera(s) down: $GO2RTC_CAMERAS_DOWN"
switch=$((switch+1))
fi
if [ "$switch" -gt 0 ];
then
switch=$((0))
echo "Restarting Go2RTC"
curl -X POST http://192.168.xxx.yyy:1984/api/restart
curl -X POST http://192.168.xxx.zzz:8123/api/services/notify/mobile_app_<myphoneid> -H "Authorization: Bearer <TokenfromHA>" -H "Content-Type: application/json" -d '{"message": "A camera stream may be down - Go2RTC has restarted!", "title": "Frigate Alert"}'
sleep 300 &
wait
systemctl enable --now frigate-check.timer
systemctl restart frigate-check.timer
fithen inside the Frigate Config file under my Go2RTC config I added: No original work here - flagrantly plagiarized from other people's work - just sharing what I have working. Didn't include the steps to get the systemd cron process running - ask the goog or ai. |
Beta Was this translation helpful? Give feedback.
-
|
I have only been running Frigate for 5 days and this has happen to me 2 times already. I have a single TP-Link C320WS connected by Ethernet and every time the network goes down, for whatever reason, I get an And there is also the memory issue, last time it consumed over 10GB of RAM before the OOM came to kill Here is the system log for the last time it happen: Note: Yes, I'm aware that running a docker container in a privileged LXC is far from ideal, but it was the "fastest" way to get it working (for now) and I don't think it is related wit the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Same: |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 for me and probably other user it's critical issue. It's ok when it affects only itself like restarting container but not whole system. I migrated from QNAP to UnRAID which much better specs -switched from i965 to iHD (J4125 -> i5-12500) but still it's unstable. |
Beta Was this translation helpful? Give feedback.
-
|
Been experiencing random freezes on an UnRAID server that could only be recovered from a force shutdown the past couple of months as a result of this. Recently discovered it was caused by Frigate and limited its container memory to 2GB though it still manages to cause a full server freeze every few weeks. Frigate is unfortunately unusable with this issue present. Log Dump |
Beta Was this translation helpful? Give feedback.
-
|
I have also been suffering from the same problem for several months now. Since I run several Docker containers, including Frigate, I started measuring the memory usage and the problem clearly comes from Frigate:
Since I have been unable to fix the problem and my entire server kept crashing, bringing down Home Assistant, Z2M... I finally set up a Home Assistant automation which restarts the Frigate service every time it hits
So again, not a solution to the problem, but my system does not crash anymore... |
Beta Was this translation helpful? Give feedback.
-
|
Was having the same problem with lockups and memory usage, so added this to the "Extra Parameters" for the Docker containers settings in Unraid. It still consumes memory but stops my Unraid locking up when it is exhausted. Not a fix but a workaround until I can track down the cause. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
My UnRAID server has been crashing lately. The logs during the crash show the issue to be with Frigate. See the below logs.
This has happened across multiple versions of Frigate. I thought it may be a 0.15beta issue so reverted back to 0.14 and the issue persisted.
I have tried capping the memory of the container to 4G (seen commented in compose) but that didn't seem to make a difference either.
I'm yet to find a pattern to the issue, it is seemingly random.
Unfortunately getting any Frigate logs is proving difficult as the whole system is generally unresponsive until it is rebooted.
I understand ReoLink cameras can sometimes be problematic but as this is a restream of a Scrypted broadcast (used for the HomeAssistant integration) I'm not sure that would be a problem here.
Any help is appreciated.
Regards,
Jack
Version
0.15.0-cea210d
What browser(s) are you using?
No response
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
FFprobe output from your camera
Frigate stats
{"cameras":{"doorbell":{"camera_fps":10.1,"process_fps":10.1,"skipped_fps":0.0,"detection_fps":0.0,"detection_enabled":true,"pid":447,"capture_pid":449,"ffmpeg_pid":925,"audio_rms":0.0,"audio_dBFS":0.0}},"detectors":{"tensorrt":{"inference_speed":5.71,"detection_start":0.0,"pid":394}},"detection_fps":0.0,"gpu_usages":{"NVIDIA GeForce GTX 1060 6GB":{"gpu":"0.0%","mem":"5.04%","enc":"0.0%","dec":"0.0%"}},"cpu_usages":{"frigate.full_system":{"cpu":"6.5","mem":"42.7"},"1":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service"},"15":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6-linux-init-shutdownd"},"16":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B"},"24":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6rc-oneshot-runner"},"25":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise s6rc-fdholder"},"26":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise certsync"},"27":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise certsync-log"},"28":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise frigate"},"29":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise frigate-log"},"30":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc"},"31":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc-healthcheck"},"32":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise go2rtc-log"},"33":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise nginx"},"34":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-supervise nginx-log"},"44":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --"},"46":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules"},"94":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate"},"95":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/certsync"},"96":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx"},"97":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc"},"105":{"cpu":"2.1","cpu_average":"1","mem":"0.1","cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml"},"113":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash ./run.user go2rtc-healthcheck"},"114":{"cpu":"1.5","cpu_average":"1","mem":"1.5","cmdline":"python3 -u -m frigate"},"131":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: master process nginx"},"177":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"178":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"179":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"185":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: worker process"},"195":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"nginx: cache manager process"},"314":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash ./run.user certsync"},"378":{"cpu":"1.1","cpu_average":"1","mem":"0.8","cmdline":"frigate.recording_manager"},"387":{"cpu":"0.3","cpu_average":"0","mem":"0.7","cmdline":"frigate.review_segment_manager"},"394":{"cpu":"0.1","cpu_average":"6","mem":"1.6","cmdline":"frigate.detector.tensorrt"},"397":{"cpu":"0.1","cpu_average":"0","mem":"2.4","cmdline":"frigate.embeddings_manager"},"418":{"cpu":"0.5","cpu_average":"0","mem":"0.8","cmdline":"frigate.output "},"423":{"cpu":"0.0","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -threads 1 -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"},"431":{"cpu":"0.0","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:"},"447":{"cpu":"2.5","cpu_average":"36","mem":"0.8","cmdline":"frigate.process:doorbell"},"449":{"cpu":"1.9","cpu_average":"1","mem":"0.7","cmdline":"frigate.capture:doorbell"},"915":{"cpu":"0.6","cpu_average":"0","mem":"0.1","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.15.0-cea210d -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/doorbell?video&audio -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/doorbell@%Y%m%d%H%M%S%z.mp4"},"925":{"cpu":"2.3","cpu_average":"2","mem":"0.6","cmdline":"/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.15.0-cea210d -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8554/doorbell_sub -r 10 -vf fps=10,scale_cuda=w=640:h=480,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:"},"13667":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sh"},"13764":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"bash"},"13933":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sleep 60"},"13951":{"cpu":"0.0","cpu_average":"0","mem":"0.0","cmdline":"sleep 30s"}},"service":{"uptime":34716,"version":"0.15.0-cea210d","latest_version":"0.15.0","storage":{"/media/frigate/recordings":{"total":11682023.0,"used":9725586.2,"free":1954786.9,"mount_type":"fuse.shfs"},"/media/frigate/clips":{"total":11682023.0,"used":9725586.2,"free":1954786.9,"mount_type":"fuse.shfs"},"/tmp/cache":{"total":953.7,"used":5.1,"free":948.6,"mount_type":"tmpfs"},"/dev/shm":{"total":128.0,"used":23.2,"free":104.8,"mount_type":"tmpfs"}},"temperatures":{},"last_updated":1740078235},"processes":{"recording":{"pid":378},"review_segment":{"pid":387},"go2rtc":{"pid":105},"embeddings":{"pid":397}}}Install method
Docker Compose
docker-compose file or Docker CLI command
Object Detector
TensorRT
Network connection
Wired
Camera make and model
ReoLink DB_566128M5MP_W
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions