Hands-free computer control using eye gaze, blinks, winks, and voice, with optional hand-gesture enhancements. Built at Hack The Valley X at UTSC. Winner: Future Impact Category, 4th Place Overall.
Iris is a Python accessibility tool that lets you control your computer without a mouse. Using a standard webcam it tracks your eye gaze to move the cursor, detects blinks and winks as clicks, and activates speech-to-text by mouth detection, all in real time with no OpenCV window required.
The gaze engine was reworked to be smooth and accurate. It uses a distance-invariant signal that fuses head pose and iris position (both divided by the inter-eye width so it does not matter how close you sit), a One Euro Filter for velocity-adaptive smoothing (steady when you hold still, responsive when you move fast), a magnitude clamp so a single bad frame cannot fling the cursor, and a blink freeze so blinking to click does not jerk the aim.
Everything runs from a single main.py. The eye-tracking path is fully hands-free; the hand-gesture controls are optional extras that only activate when your hands are visible.
Core capabilities:
- Gaze-driven cursor movement with head-pose fusion and One Euro smoothing
- Double blink for left click, triple blink for right click
- Left wink for Alt+Tab, right wink at screen edges to scroll
- Hold mouth open 1.5s to toggle speech recognition (VOSK, preloaded)
- Global hotkeys to enable/disable and quit from anywhere
- Optional hand gestures: pinch to click and drag, thumb-index gap to set cursor speed, prayer hands to pause everything
- All inputs are locked automatically during transcription and typing
- A working webcam.
- Python 3.12. MediaPipe does not ship wheels for Python 3.13/3.14 yet, so a 3.12 environment is required. Your system default can stay on a newer version; the steps below pin this project to 3.12 in a virtual environment.
From a terminal in the project folder:
py -3.12 -m venv .venv
.venv\Scripts\activate
pip install "mediapipe==0.10.18" pyautogui scipy sounddevice vosk keyboardNotes on the pinned install:
mediapipe==0.10.18is used because newer builds have dropped the legacysolutionsface-mesh API this project relies on. It automatically pulls in a compatiblenumpy<2andopencv-contrib-python.- Do NOT also install
opencv-python. It conflicts with theopencv-contrib-pythonthat MediaPipe provides; having both breakscv2. keyboardis optional and only enables the global on/off hotkeys. Without it the app still runs and prints a note.
Speech dictation needs a VOSK model folder in the project root. Without it, everything else runs and speech simply reports as unavailable.
Small model (about 40 MB, good for testing). From the project folder:
curl -L -o vosk.zip https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
tar -xf vosk.zip
ren vosk-model-small-en-us-0.15 vosk-model-en-us-0.22
del vosk.zipFor full accuracy, download vosk-model-en-us-0.22 from https://alphacephei.com/vosk/models and extract it into the project root (no rename needed).
python main.pyIt prints system info, waits 3 seconds, then starts. A [perf] ~NN FPS line prints every couple of seconds so you can confirm the loop rate. Press Ctrl+C in the terminal to stop (or Ctrl+Alt+Q if keyboard is installed).
| Input | Action |
|---|---|
| Eye gaze | Move cursor |
| Double blink | Left click |
| Triple blink | Right click |
| Left wink | Alt+Tab |
| Right wink (cursor at top/bottom 30%) | Scroll |
| Hold mouth open 1.5s | Toggle speech recognition |
| Hold mouth open again | Stop recording and type |
| Ctrl+Alt+E | Enable/disable all control (needs keyboard) |
| Ctrl+Alt+Q | Quit the app (needs keyboard) |
Optional hand gestures (require HAND_TRACKING_ENABLED = True):
| Input | Action |
|---|---|
| Prayer hands (palms together) | Pause/resume everything |
| Left hand pinch | Click and drag (eye tracking pauses while dragging) |
| Right hand thumb-index gap | Adjust cursor speed (small gap = precise, large = fast) |
All inputs are locked while speech is being transcribed and while text is being typed. Console output gives real-time feedback on every detected action.
All knobs are constants near the top of main.py:
ONE_EURO_MIN_CUTOFF(default 0.4): lower is steadier when holding still (less jitter); try 0.25 if it is still twitchy.ONE_EURO_BETA(default 0.06): raise if the cursor feels laggy catching up to fast looks.GAZE_SENSITIVITY_X/GAZE_SENSITIVITY_Y(default 2.0): cursor travel per unit of gaze; raise to reach screen edges with less movement.GAZE_INVERT_X/GAZE_INVERT_Y: flip if the cursor moves the wrong way.HAND_TRACKING_ENABLED(default True): set to False for the smoothest possible gaze (disables all hand gestures and gives the face mesh the full frame rate).
The scripts/ folder has launchers for an always-on setup:
scripts/run_iris.batruns the app with a visible console (uses the.venv, falls back topy -3.12).scripts/start_iris_hidden.vbsruns it hidden, for the Startup folder. Add a shortcut to it inshell:startup(press Win+R, typeshell:startup) so Iris launches at login and the Ctrl+Alt+E hotkey is always available.
- Distance-invariant head + iris gaze with One Euro smoothing and blink freeze
- Adaptive EAR baseline calibration so blink/wink thresholds self-adjust at runtime
- Input locking during transcription and typing
- Global enable/disable and quit hotkeys
- Per-user calibration flow to tune gaze range and sensitivity for different faces and lighting
- macOS / Linux support by replacing the Windows-only notification and cursor calls
Contributions are welcome:
- Fork the repo
- Create a feature branch (
git checkout -b feature/your-feature) - Commit your changes (
git commit -m 'Add your feature') - Push to the branch (
git push origin feature/your-feature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.
- Hack The Valley X for hosting and the Future Impact award
- MediaPipe Face Mesh for landmark detection
- VOSK for offline speech recognition
- Img Shields for badges