[HW Accel Support]: Possible how to use both openvino and tensorrt simultaneously in single container? Or create separate containers?

[chrislawso]

Describe the problem you are having

I repurposed an old pc with gtx950 and have frigate 0.16.0 running with tensorrt for object detection and video decode. I noticed that the intel cpu is i7 6700 which has Intel® HD Graphics 530 which I have separately tested and is supported in frigate with openvino.

Is there any advantage or benefits to be had to add openvino support to a frigate container that is already running tensorrt? If the nvidia gpu begins to hit memory or core limits is there any method to offload any cameras, object detections, or anything else, for the openvino to process any loads in one container?

Or would the only solution be to run one frigate tensorrt container with some number of cameras and then separately run a different frigate container that uses openvino for the remaining cameras?

Version

0.16.0

Frigate config file

tbd

docker-compose file or Docker CLI command

tbd

Relevant Frigate log output

tbd

Relevant go2rtc log output

tbd

FFprobe output from your camera

tbd

Install method

Docker Compose

Object Detector

TensorRT

Network connection

Wired

Camera make and model

tbd

Screenshots of the Frigate UI's System metrics pages

tbc

Any other information that may be helpful

No response

[hawkeye217]

You can't mix and match the two detector types for object detection, but you could use one detector for object detection and the other for hardware accelerated enrichments (LPR, face recognition, semantic search). You could decide what your priority is and base the Docker image you use off of that.

https://deploy-preview-16390--frigate-docs.netlify.app/configuration/hardware_acceleration_enrichments

[hawkeye217]

Probably best to do some tests and see which GPU gives you the best performance for object detection, and then just use that Docker image. Enrichments running on CPU are pretty lightweight, anyways.

[chrislawso]

you could use one detector for object detection and the other for hardware accelerated enrichments (LPR, face recognition, semantic search).

For object detection, you specify the detector in your config as you noted. For enrichments, the GPU will be automatically selected based on the Docker image used. There's no config option other than setting device: GPU.

I am confused by the different responses. For tensorrt the use of the tensorrt docker image is required which according to replies the enrichments would automatically also use tensorrt gpu.

So, in one container cannot use tensorrt for object detection and openvino for enrichments, or vice versa.

[hawkeye217]

I am confused by the different responses. For tensorrt the use of the tensorrt docker image is required which according to replies the enrichments would automatically also use tensorrt gpu.

So, in one container cannot use tensorrt for object detection and openvino for enrichments, or vice versa.

Sorry for the confusion. Yes, this is correct and is because the utility Frigate uses to run enrichment models, onnxruntime, will prefer TensorRT when you're running the -tensorrt Docker image. The default Docker image (which would work with OpenVINO), does not support TensorRT.

We'll update the docs to make this more clear.

[NickM-27]

So, in one container cannot use tensorrt for object detection and openvino for enrichments, or vice versa.

correct, but you can use openvino for object detection and tensorrt for enrichments

[chrislawso]

Thanks everyone.