Using VideoDecoder for security camera app leads to codec reclaimation

[dominic-p]

As suggested in #121, I'm opening a new issue to discuss my use case.

I have just started using WebCodecs for a security camera application, and my VideoDecoders are getting reclaimed due to inactivity if the tab stays in the background for too long.

Use Case

The app is fairly simple, I spawn a single Worker which makes a WebSocket connection to the server. The server makes an RTSP connection to each of the configured cameras and then sends frames to the client through the WebSocket. The worker then uses VideoDecoders to draw the frames on canvases (essentially following this sample). It works really well until the I put the tab in the background. After a short while I get QuotaExceeded errors due to the "inactive" codecs being reclaimed.

I understand from these docs that the only way to keep an active VideoDecoder in the background from being reclaimed is to have an active VideoEncoder or an active AudioEncoder on the same global object. The problem for my use case is that these are security camera feeds that don't have audio (even if they did, I would only be decoding it).

You might ask what's the point of having a security camera app in a background tab? That's a good question. In common usage, you would expect to keep the app active all the time to view the video feeds. That said, in testing I have found that I often need to switch to another browser tab, and I imagine that will be the case at least some of the time in production.

Workarounds

Of course, there are a few workarounds I can implement. The simplest thing will be to watch for these errors and then attempt to restart the video streams when I run into them. The next idea is to run a dummy AudioEncoder of some kind in the Worker just to keep the VideoDecoders alive. But, it would be nice if there was a less hacky solution.

Solutions

I'm not sure what the best solution would be here. Maybe we can ask for user permission to run background video like we do for wake locks, location, etc.?

Also, it was suggested that I file a bug with Chromium since that's where I'm seeing this behavior. I'm happy to do that, but since the issue seems to be with the spec itself, I thought it would make sense to discuss it here first.

[dalecurtis]

Is it not possible to pause and resume the VideoDecoder using the PageVisibility API? You'd need to buffer since the most recent key-frame or have the ability to request a key-frame on demand though.

[dominic-p]

That's an approach I haven't considered. It would add a fair amount of complexity, but it may be doable.

My first question is what does "pausing" a VideoDecoder look like? I'm still pretty new to WebCodecs, so I'm trying to understand what that would mean. I'm assuming that no mater what I do though, my codecs are going to be reclaimed if the tab stays in the background (paused or not).

[dalecurtis]

I just meant close the VideoDecoder once it goes into the background (or just let the QuotaExceeded happen).

[dominic-p]

Ahh, that makes sense. Yes, I think the best workaround for the time being is to just handle the QuotaExceeded error and then restart when the tab is active again.

That's not the smoothest UX though because when the user transitions back to the tab, they will have to wait for all of the video feeds to start up again. Depending on how many cameras there are, the camera response time, keyframe settings, etc. initialization can be fairly slow.

[dalecurtis]

I acknowledge it presents some challenges, but keep in mind the UX of having a bunch of unused videos decoded in the background isn't great either though. It wastes a fair amount of resources and may even preclude foreground tabs from utilizing hardware codecs.

Codec reclamation shouldn't be kicking in immediately in this case, only after enough time has elapsed that the user probably isn't coming back soon. If the user needs wants the feeds to remain alive they could choose to move that tab to another window which remains in the foreground.

[dominic-p]

I can definitely understand that. From my perspective, it should be up to the user if they want to dedicate those resources or not. That's why I kind of like the idea of a permission request for background video.

If they have enough screen real estate to move the tab to another window they will probably do so, but if not, it makes sense to just open a new tab and do the work there. In that case, the expectation is that the video feeds will be there waiting when they come back to that tab (assuming they've opted in to keep them alive).