IRiS - What is it?
IRiS (Infra-red Ring Imaging System) is my soon to be open-source eye tracking system designed to bring accurate, low-latency gaze and blink detection to consumer VR headsets, without relying on expensive commercial solutions.
The Problem
Eye tracking in VR is still largely only on expensive headsets. The Meta Quest Pro, PlayStation VR2, and Apple Vision Pro all ship with it, but the hardware and software is proprietary. For developers, researchers, and modders who want eye tracking on a headset that doesn't natively support it, or who want to understand how it works, there's almost nothing that can match the performance of these headsets (no shame).
IRiS is my attempt to fix that.
Two Approaches
IRiS is built on two tracking technologies that complement each other.
Cameraless IR Phototransistor Array
The first approach uses a ring of 8 IR phototransistors and 4 IR emitters arranged around the eye. Instead of capturing an image, each phototransistor measures reflected infrared light intensity. As the eye moves, the reflection pattern across the array shifts, and from that shift, gaze direction can be found. To accurately find the gaze direction a basic ML model will be used, which would be very small and able to run at the full 500hz on practically any hardware.
The part that makes reading from this sensor array possible is a custom PCB, the IRiS ADC Hat, a Raspberry Pi Pico hat using two XADC78H90 ADCs on SPI0 for 16-channel simultaneous readout.
Camera-Based ML Pipeline
Note: Most of this section has not been fully developed, things can change!
The second approach uses a small camera module paired with a neural network for eyelid tracking and pupil dilation tracking. The model, though unfinished, would be used primary for social vr users.
The camera node itself is a XIAO ESP32-S3 Sense streaming frames at 30+ FPS, grayscale, 320ร240, to be fed directly into the model, if the user opts in.
Why Both?
The phototransistor array is fast and power-efficient but has limited accuracy, is sensitive to IR interference, and cannot accurately detect partially closed eyes, and pupil size. The camera pipeline is more robust and can detect pupil dilation and eyelid state, but has higher latency at a slower speed.
Running both in parallel and fusing the outputs gives the best of both, speed from the sensor array, accuracy and feature richness from the camera model.
Hardware Stack
- RP2040 - The brains of the phototransistor array
- XIAO ESP32-S3 Sense - OV2640 camera, stream at 30+ FPS
- IRiS Sensing Ring - Custom PCB, 8 IR phototransistors and 4 IR emitters
- IRiS ADC Hat - Custom PCB, dual XADC78H90, 16-channel SPI ADC
Current Status
- โ The camera based model is trained and running
- โ All PCB's are currently in production
- ๐ The ESP32-S3 camera node firmware is in active development
- ๐ Next steps: building and testing PCBs, training the gaze model
All hardware design files, firmware, and model training code will be released openly as the project matures.