[Camera Support]: ffmpeg doesnt recover after "camera exceeded fps limit. Exiting ffmpeg..." #18502
Unanswered
foxtrotcz
asked this question in
Camera Support
Replies: 6 comments 7 replies
-
|
We need to see a full copy of logs |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
Ok, here. Frigate: Go2rtc: |
Beta Was this translation helpful? Give feedback.
-
|
There are no logs that are printed after the waiting for it to exit gracefully, so it seems after this occurred the camera reconnected correctly |
Beta Was this translation helpful? Give feedback.
-
|
Well, depends what part you mean. As I wrote in main post, it worked in live view and ffmpeg process thats serves for live view was running. Info from Live view process that was running: Recording process that wasnt running: To me it seems like some bug that causes ffmpeg process not being started again. |
Beta Was this translation helpful? Give feedback.
-
|
I tried my own little debugging. Line 131 in f141b58 But I couldnt reproduce the issue this way. The log output was little different when done this way. In my reproduction attempt logs look like this: I doubt the Non-monotonic DTS is relevant but you can see that after waiting to exit ffmpeg, frigate.video actually reports other errors. And when checking container, the recording ffmpeg process is there. So its strange in my original issue it was different. To me it seems that sometimes Frigate still waits for ffmpeg to "gracefully exit" and even thou it did and isnt in listed processes Frigate doesnt know that and its still waiting. Because if it did actually notice it exited it should report "Unable to read frames from ffmpeg process" and the other errors. Fortunately this only happens cca once every week and on my wifi Tapo which isnt much important for recording. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
So, to me it looks like, if ffmpeg exited gracefully on line Line 267 in f141b58 Then in the next loop capture_thread shouldnt be alive and I should get Line 236 in f141b58 This should next attempt to start ffmpeg process again. But I didnt get this. Frigate likely ran its normal loop thinking everything is ok even though ffmpeg process was down. Line 274 in f141b58 It seems this is some kind of special case that happens with the combination of timeout camera and the ffmpeg exit. |
Beta Was this translation helpful? Give feedback.
2 replies
-
that's not possible, because the only way the frame time is updated is if the capture process actually gets a valid frame of data from ffmpeg. But if ffmpeg was stopped that would not be the case.
maybe, but in any case I am not able to reproduce with any combination of unplugging my camera |
Beta Was this translation helpful? Give feedback.
-
|
Got it. I will try to check it more if it happens again. |
Beta Was this translation helpful? Give feedback.
-
|
Please keep this discussion updated with any progress you make @foxtrotcz! This seems to be the same core issue I'm encountering (also with Tapo cameras), see #18350 and #18405. I also have the symptom that the camera live feed and go2rtc still works, but recording does not. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Ok, I got the same issue again. I outputted processes in Frigate container: The way I understand it |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I looked into this a little bit - could you try adding a If there's an exception here, we'll catch it and it shouldn't become a zombie. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I did add this try block into the stable 0.15.0-6cb5cfb version. Btw, I also tried restarting the camera itself, but as expected, this didnt help. Only way is to restart Frigate. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, so it occurred again and no exception was raised. 0.16.0 beta3 processes list: I will think some more about how to reproduce it or get some more data. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Checklist
Describe the problem you are having
Note: Logs and configs cropped to only include relevant camera and info.
I have wifi Tapo camera that sometimes times out as seen from attached go2rtc log.
This for some reason causes
tapo exceeded fps limit. Exiting ffmpeg...It then exits ffmpeg
Waiting for ffmpeg to exit gracefullyBut after this point ffmpeg stays down and doesnt recover which causes recording not working.
To be clear, go2rtc and live view works after the timeout. Only ffmpeg which handles recording doesnt recover.
Also, in different cases when ffmpeg crashes for different reason it recovers successfully with this same camera.
Example when ffmpeg recovers successfully:
Go2rtc timeout:
2025-05-29 12:36:39.138648802 12:36:39.138 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.16.1.2:54300->192.168.1.204:554: i/o timeout" url=rtsp://username:[email protected]/stream1Frigate error:
So it looks like if ffmpeg really crashes it does recover. But in my specific case there is no crash logged, only attempt to exit and it doesnt recover.
In System metrics - Cameras the Tapo chart shows 92 FPS, but when I use probe it shows FPS: 57/4.
When I access the camera stream either directly or via go2rtc in VLC is shows 14 FPS.
This means that FPS is good but Frigate and ffmpeg still refuses to recover.
I also checked list of processes in container and the ffmpeg process responsible for recording the tapo camera was missing so it looks like it exited successfully.
On the other hand the ffmpeg process responsible for live view was running.
So it looks like: If ffmpeg process crashes it recovers. If its gracefully shutdown or because of exceeded fps limit it does not recover.
So I wonder if Frigate can add some ffmpeg handling to make sure the recording process actually recovers even in this case.
Relevant source code:
frigate/frigate/video.py
Line 263 in f141b58
From what I see in source code, if fps exceeds limit ffmpeg gets shutdown. But I dont understand why I kept seeing 92 FPS in stats when ffmpeg probe and VLC confirmed fps of camera stream is actually 14 fps. Maybe because of the killed ffmpeg process Frigate cant update the stats correctly? I wasnt able to find down how exactly Frigate gets the camera_fps value.
But I think after the ffmpeg exits because of this there should be mechanism to auto-recover.
I saw threads here with similar issue but I want to focus this more on the recovery of ffmpeg process.
Thanks.
Steps to reproduce
tapo exceeded fps limit. Exiting ffmpeg...andWaiting for ffmpeg to exit gracefullyin Frigate.Version
0.15.0-6cb5cfb
In which browser(s) are you experiencing the issue with?
Google Chrome 137.0.7151.56
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Relevant go2rtc log output
Operating system
Debian
Install method
Docker Compose
Network connection
Wireless
Camera make and model
TP-Link Tapo C100
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions