Frigate NVR with Coral TPU & iGPU passthrough using ESXi on Intel NUC

For the past couple of weeks, I have been investigating some issues reported by the community when using ESXi with the popular Google Coral Edge TPU for accelerating machine learning (ML) inferencing. Fortunately, with the help from one of our engineers, Songtao, we were able to find a solution! You can find the complete write-up HERE and it also works with the latest ESXi 8.0 Update 1 release.

I was actually surprised at how popular the combination of the Google TPU and ESXi was from the community, which I guess should not come as a surprise, especially with the capabilities of ESXi coupled with all the interests in AI/ML these days.

Another popular use case of the Google TPU, which I had recently learned about is for real-time AI object detection using the Frigate NVR (Network Video Recorder) software, which is a commonly deployed solution that enable various home security and automation capabilities.

In fact, during a conversation with my buddy Alan Renouf, who is a Product Manager focused on running modern Edge workloads and is also a Frigate user, I discovered that the Frigate stack, which encompasses inferencing, video decoding, and the integration of cameras and sensors, closely resembles the components that you would find in many Edge deployments with simliar set of use cases.

Funny enough, I ended up leveraging a lot of my existing work with running ESXi on Intel NUCs and iGPU passthrough, while learning about and setting up Frigate! This was definitely an interesting project to explore and as shared, I now have a complete working setup with the full setup and write-up below.

JFYI - I have already submitted a PR 6576 to update the Frigate ESXi documentation as it is severely out of date and help folks quickly find the latest setup instructions.

Earlier this week I had no idea what Frigate NVR was ...

Today, full setup w/ESXi on Intel NUC (this thing is amazing, SO many use cases) 🥳

✅ Passthrough Google Coral USB TPU (inferencing)
✅ Passthrough Intel iGPU (vid
decoding)
✅ RTSP enabled camera #AlwaysBeLearning pic.twitter.com/Qghj7qwOFp

— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) May 18, 2023

Hardware

• Intel NUC 13 Pro
  • Earlier Intel NUC models or simliar systems should also work but iGPU passthrough with ESXi is only be possible with Intel 12th Gen CPUs or later, please see this post HERE for more details
• Intel 13th Gen Xe Graphics Integrated GPU (iGPU)
  • iGPU passthrough with ESXi is only be possible with Intel 12th Gen CPUs or later, please see this post HERE for more details
• Google Coral USB TPU Accelerator
• EZVIZ C8PF Camera

Software

• VMware ESXi 8.0 Update 1
• Ubuntu 20.04 (Focal) Virtual Machine
  • Meets the minimum python package dependencies for the Coral TPU software but you can certainly use newer versions of Ubuntu and leverage pyenv to meet the specific python requirements. Alternatively, you can deploy an Ubuntu 20.04 VM just for the Coral TPU initialization and then once its functional, you can detach/re-attach to another VM on the same ESXi host
• Frigate NVR (deployed as a Docker container)
• Docker Engine

Setup

The instructions below will result in a setup where the Frigate application is running as a Docker container inside of an Ubuntu 20.04 VM using ESXi 8.0 Update 1 utilizing two passthrough devices: Coral TPU (inferencing) and Intel iGPU (video decoding) all hosted on an Intel NUC 13 Pro.

Instructions

While the guide below is for the setup that has been described above, it should also be applicable to other systems with similiar capabilities.

Step 1 - Follow this blog post HERE to properly configure and enable ESXi passthrough of the Google USB TPU. After this step, you should be able to run "lsusb" command inside of the Ubuntu VM and see that Coral TPU is listed with 18d1:9302 (Google Inc.). If you do not, please stop and carefully re-read the instructions as you likely may have missed a step.

Step 2 (Optional) - If you wish to enable ESXi passthrough of an Intel iGPU (see Hardware requirement above), then you will need to install additional packages to get the iGPU recognized from within the Ubuntu VM. Please see this blog post HERE for detailed instructions for configuring iGPU passthrough for ESXi running on various Intel NUCs.

• For Ubuntu 20.04 (Focal), please follow Intel's iGPU package installation HERE
• For Ubuntu 22.04 (Jammy), please follow the Intel iGPU package installation HERE

After this step, you should be able to run "sudo lshw -c display" command within the Ubuntu VM and see that the i915 driver is loaded for the Intel iGPU. If you do not, please stop and carefully re-read the instructions as you likely may have missed a step.

Additionally, we also need to find device render ID for our iGPU, which will be needed in Step 5. You can see the list of render devices by running the following

ls /dev/dri/render*

Since multiple rendering devices will be listed, we need to identify the iGPU device which will be used for video decoding in Frigate. The easiest way is to use vainfo utility (install it if you do not have it installed by default) and attempt to connect to all rendering devices by providing the path to render device ID like the following:

vainfo --display drm --device /dev/dri/renderD129

until you have identified the correct device which is our iGPU and the output will look something like the following:

After this step, you would have correctly identified the rendering device ID for the iGPU to be used by Frigate. If you do not, please stop and carefully re-read the instructions as you likely may have missed a step.

Step 3 - Create the two required Frigate directories: frigate which will be used for the Frigate configuration YAML file and frigate/storage which is where Frigate will save all its generated files.

mkdir -p frigate/storage

Step 4 - Create the Frigate configuration file (config.yml) and store that in frigate directory that you had created in the previous step.

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  enabled: False

cameras:
  william_office:
    ffmpeg:
      hwaccel_args: preset-vaapi # Use Intel NUC iGPU (optional)
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.30.106:554/H.264
          roles:
            - clips
            - detect
    rtmp:
      enabled: false
    detect:
      width: 1280
      height: 960
      fps: 5
    objects:
      track:
        - person
        - dog
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      retain:
        default: 5
    record:
      enabled: True
      retain:
        days: 1
      events:
        retain:
          default: 5

Note: If you do not have an Intel iGPU or do not intend to passthrough an Intel iGPU to Frigate, you should remove Line 12 (hwaccel_args)

Step 5 - Create the Frigate Docker compose YAML file (docker-compose.yml)

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true
    network_mode: "host"
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "64mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb # Passthrough of Coral USB TPU
      - /dev/dri/renderD129 # Passthrough Intel NUC iGPU (optional)
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/vmware/frigate/config.yml:/config/config.yml:ro
      - /home/vmware/frigate/storage:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1g
    ports:
      - "5000:5000"
      - "8554:8554"
      - "8555:8555/tcp"
      - "8555:8555/udp"
    environment:
      - FRIGATE_RTSP_USER=REPLACE_WITH_YOUR_RTSP_USERNAME
      - FRIGATE_RTSP_PASSWORD=REPLACE_WITH_YOUR_RTSP_PASSWORD

Note: If you do not have an Intel iGPU or do not intend to passthrough an Intel iGPU to Frigate, you should delete Line 12 (-/dev/dri/renderD129)

Step 6 - Start the Frigate docker container by running the following command:

sudo docker compose up

If everything was configured correctly, we should see the following lines during the the Frigate startup as shown in the screenshot above where Frigate is able to successfully detect and load the Coral TPU. If you do not see this message, then you most likely made a mistake somewhere in the instructions.

Finally, we can also connect to the Frigate management portal by opening a browser to the Hostname/IP Address of your Ubuntu VM that is running the Frigate container (https://192.168.30.51:5000) and we should see that everything is up and running based on your configurations.

If you click on the System tab, we can also that both the Coral TPU and Intel iGPU has been successfully configured and is already in use as shown in the screenshot below.

In addition to this Frigate setup, I know many in the Home Automation community typically integrate this with the popular Home Assistant Solution, which should work but I will leave that as an exercise for readers, if you have not set that up before 🙂