Air Mouse

Description

Air Mouse is an innovative human-computer interaction project that combines motion sensing and gesture recognition to create a natural and intuitive way to control your computer. The project consists of two main components:

1. A hardware controller using ESP32 and MPU6050 for motion-based cursor control
2. A computer vision system using OpenCV and MediaPipe for hand gesture recognition

This combination allows users to control their computer through natural hand movements and gestures, creating an immersive and intuitive interaction experience.

Features

• Motion-Based Cursor Control

  • Precise cursor movement using gyroscope and accelerometer data
  • Complementary filter for smooth motion tracking
  • Moving average filter to reduce noise
  • Bluetooth connectivity for wireless operation

• Gesture Recognition

  • Index finger pinch: Left mouse click and drag
  • Middle finger pinch: Right mouse click
  • Ring finger pinch: Scroll down
  • Pinky finger pinch: Scroll up
  • Real-time hand tracking and visualization

Hardware Requirements

• ESP32 microcontroller
• MPU6050 6-axis IMU (accelerometer + gyroscope)
• USB cable for programming
• Power source (battery or USB)

Software Requirements

• Arduino IDE
• Python 3.x
• Required Python packages:

  opencv-python
  mediapipe
  pyautogui
  

• Required Arduino libraries:

  Wire.h
  MPU6050_tockn.h
  BleKeyboard.h
  

Installation

Hardware Setup

1. Connect the MPU6050 to ESP32:
   • VCC → 3.3V
   • GND → GND
   • SDA → GPIO21
   • SCL → GPIO22

Software Setup

ESP32 Configuration

1. Install the Arduino IDE
2. Add ESP32 board support to Arduino IDE
3. Install required libraries through Arduino Library Manager
4. Upload the esp32code.ino to your ESP32

Python Environment Setup

1. Install required Python packages:

   pip install opencv-python mediapipe pyautogui

2. Run the gesture recognition script:

   python main.py

Usage

1. Power on the ESP32 device
2. Connect to the ESP32 through Bluetooth (it will appear as "ESP32 Virtual Mouse")
3. Launch the Python script for gesture recognition
4. Hold your hand in front of the camera
5. Use the following gestures:
   • Move the ESP32 device to control cursor position
   • Pinch index finger and thumb for left click/drag
   • Pinch middle finger and thumb for right click
   • Pinch ring finger and thumb for scrolling down
   • Pinch pinky finger and thumb for scrolling up

Technical Details

ESP32 Component

• Uses complementary filter to combine accelerometer and gyroscope data
• Implements moving average filter for noise reduction
• Bluetooth communication using BleKeyboard library
• Adjustable sensitivity and filter parameters

Computer Vision Component

• Hand landmark detection using MediaPipe
• Real-time finger tracking and gesture recognition
• Multi-threaded scroll functionality
• Configurable pinch threshold

Customization

• Adjust sensitivity variable in ESP32 code to modify cursor movement speed
• Modify PINCH_THRESHOLD in Python script to change gesture sensitivity
• Change FILTER_WINDOW size in ESP32 code to adjust motion smoothing
• Customize gesture actions by modifying the gesture detection logic

Troubleshooting

• No Bluetooth Connection: Ensure ESP32 is powered and in range
• Erratic Cursor Movement: Adjust sensitivity and filter parameters
• Gesture Not Detected: Check camera and lighting conditions
• Delayed Response: Verify system resources and USB camera connection

Contributing

Feel free to fork this project and submit pull requests with improvements. Some areas for potential enhancement:

• Additional gesture support
• Improved motion filtering algorithms
• Battery optimization for ESP32
• Custom gesture mapping interface

License

This project is released under the MIT License. See LICENSE file for details.

Acknowledgments

• MediaPipe team for their hand tracking solution
• MPU6050_tockn library developers
• ESP32 BleKeyboard library contributors