AeroSense AI — Vision-Based Smart HVAC Optimization Platform

Real-time occupancy-aware air conditioning control for auditoriums, powered by YOLOv8 computer vision, Arduino Uno edge processing, and explainable AI decision-making.

Project Overview
System Architecture
Hardware Components
Software Modules
Dashboard Parameters — Complete Reference
Equations & Mathematical Models
State Machine & Decision Engine
Data Flow & Interlinking
Dual-Zone AC Optimization
Setup & Installation
Running the System
API Endpoints
Project Structure
Troubleshooting

Project Overview

AeroSense AI solves a fundamental problem in building HVAC management: conventional air conditioning systems cool rooms at a fixed rate regardless of how many people are present. This wastes enormous amounts of energy — especially in auditoriums and conference halls that cycle between empty and fully occupied states.

What AeroSense Does

Detects people using a YOLOv8n neural network running on a webcam feed
Tracks individuals across frames using centroid-based multi-object tracking
Classifies room state (EMPTY → IDLE → OCCUPIED → BUSY → HIGH_LOAD → COOLING_BOOST) using a hysteresis-protected state machine
Optimizes dual-zone AC units — independently controlling two air conditioners based on which zone has more people
Commands physical hardware via Arduino Uno (IR blaster for Midea AC, relay control, LED indicators)
Displays a real-time dashboard with live video, metrics, thermal models, charts, and an explainable AI engine

Key Innovation

Unlike simple on/off timer systems, AeroSense uses temporal intelligence — it considers:

How many people are present (occupancy count)
How fast they are moving (activity level from centroid tracking)
Whether there was a sudden influx (rapid entry detection)
Hysteresis debouncing to prevent HVAC mode oscillation

System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        VISION PIPELINE                         │
│  Webcam (Logitech C270)                                       │
│      │                                                         │
│      ▼                                                         │
│  YOLOv8n ONNX Detector  ──▶  Centroid Tracker  ──▶  Speeds   │
│  (person bounding boxes)     (persistent IDs)       (px/frame) │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                     DECISION ENGINE                            │
│                                                                 │
│  Zone Assignment ──▶ Dual-AC Optimizer ──▶ AC Commands         │
│  (left/right split)   (per-zone BTU calc)  (temp, fan, on/off) │
│                                                                 │
│  State Machine ──▶ HVAC Mode (0-4) ──▶ Comfort Score           │
│  (temporal debounce)  (ECO→BOOST)       (50-100%)              │
│                                                                 │
│  Thermal Model ──▶ Cooling Load (TR) ──▶ Plant Loading (%)     │
│  (BTU calculation)   (tons required)      (used vs installed)  │
└────────────────────────────┬────────────────────────────────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
┌──────────────────┐ ┌────────────┐ ┌────────────────┐
│  Arduino Uno Q   │ │  Secondary │ │  Web Dashboard  │
│  (Main MCU)      │ │  AC via    │ │  (FastAPI +     │
│  - Relay         │ │  Wi-Fi UDP │ │   WebSocket)    │
│  - LEDs          │ │  IR Blaster│ │  - Live Video   │
│  - Serial 115200 │ │            │ │  - Metrics      │
└──────────────────┘ └────────────┘ └────────────────┘

Hardware Components

Component	Role	Connection
Logitech C270 Webcam	Person detection camera	USB to host PC
Arduino Uno Q (Main)	Edge controller: relay, LEDs	USB Serial (115200 baud)
Arduino (Secondary)	IR blaster for Midea AC unit 2	Wi-Fi UDP on port 8888
3-Color LED Array	Visual HVAC mode indicator (Green/Yellow/Blue)	GPIO pins 2, 3, 4
5V Relay Module	AC compressor simulation / control	GPIO pin 5
IR Transmitter LED	Sends Midea AC protocol codes	Digital Pin 3 (Secondary)
IR Receiver (TSOP)	Reads remote control codes	Digital Pin 9 (Secondary)

Main Controller Circuit Diagram

Secondary Controller Circuit Diagram

Software Modules

Python Backend (runs on host PC)

File	Purpose
`linux/run.py`	Main entry point — orchestrates vision pipeline, decision engine, serial bridge, and web server
`linux/runnew.py`	Alternative entry point (same architecture, extended features)
`linux/src/detector.py`	YOLOv8n ONNX inference — detects people in camera frames
`linux/src/tracker.py`	Centroid-based multi-object tracker with speed estimation
`linux/src/state_machine.py`	Room Intelligence Engine — state transitions, comfort score, energy metrics
`linux/src/serial_bridge.py`	Serial communication bridge to Arduino (with mock fallback)
`linux/src/server.py`	FastAPI web server — dashboard HTML, WebSocket telemetry, REST APIs

Arduino Firmware

File	Purpose
`arduino/src/Main/Main.ino`	Main MCU loop — serial commands, watchdog
`arduino/src/Main/HVACControl.cpp/h`	HVAC mode management (LED + relay control)
`arduino/src/Main/SerialComm.cpp/h`	Serial protocol handler (command parsing, telemetry output)
`arduino/src/Secondary/Secondary.ino`	IR blaster controller for Midea split AC
`arduino/src/Secondary/MideaIR.cpp/h`	Midea IR protocol encoder (38kHz carrier modulation)

Dashboard Parameters — Complete Reference

Top Header Bar

Parameter	Source	Description
MCU Link	`serial_bridge.get_telemetry()["mcu_connected"]`	Shows whether the Arduino Uno is physically connected via USB Serial. Green = connected, Red = mock mode
Uptime	`time.time() - GlobalState.start_time`	How long the server has been running since launch (HH:MM:SS)
AUTO-OPTIMIZE ACTIVE	Static badge	Indicates the system is in autonomous control mode (not manual override)

Live Metrics Grid (Right Column)

Metric	Key	Source	How It's Calculated
Occupancy	`occupancy`	`len(boxes_dict)` from tracker	Count of currently tracked person bounding boxes
Room State	`state`	`RoomIntelligenceEngine.update()`	State machine output: EMPTY, IDLE, OCCUPIED, BUSY, HIGH_LOAD, or COOLING_BOOST
Comfort Score	`comfort_score`	`RoomIntelligenceEngine.update()`	50-100% — measures how well HVAC mode matches ideal mode for current occupancy
Target Temp	`target_temp`	`{0:26, 1:24, 2:22, 3:20, 4:18}[hvac_mode]`	Setpoint temperature mapped from HVAC mode
Estimated Power	`power_kw`	`RoomIntelligenceEngine.get_current_power_kw()`	Power draw for current mode: ECO=0.15kW, LOW=1.1kW, COMFORT=2.2kW, HIGH=3.2kW, BOOST=4.0kW
Energy Savings (₹)	`savings_inr`	`total_energy_saved_kwh × 8.0`	Cumulative rupee savings vs. always-on baseline at ₹8/kWh tariff
CO₂ Avoided	`co2_saved_kg`	`total_energy_saved_kwh × 0.823`	Cumulative CO₂ avoided using India's grid emission factor (0.823 kg CO₂/kWh)

Explainable AI (XAI) Engine Panel

Parameter	Source	Description
HVAC Policy	`mode_names[hvac_mode]`	Current operating mode name: ECO, LOW_LOAD, COMFORT, HIGH_LOAD, BOOST
Confidence Score	Fixed 98% (occupied) / 100% (empty)	Confidence in the occupancy count accuracy
Explanation Text	`engine.get_explanation(count)`	Human-readable rationale for the current HVAC decision
Manual Override Buttons	`app.state.manual_override_mode`	Force ECO (mode 0), COMFORT (mode 2), or BOOST (mode 4)

Auditorium Cooling Capacity (Thermal Model Panel)

Parameter	Key	How It's Calculated
TR Required	`cooling_tr`	`total_load_btu ÷ 12,000`
Space Load	`space_load_btu`	`volume_ft³ × 6 = 15,000 × 6 = 90,000 BTU/hr`
People Load	`people_load_btu`	`occupancy × 500 BTU/hr per person`
Plant Capacity	`installed_capacity_tr`	Fixed at 12.0 TR (installed cooling capacity)
Plant Loading	`plant_load_pct`	`min(100, (cooling_tr ÷ installed_capacity_tr) × 100)`
Grid CO₂ Impact	`current_co2_kgph`	`power_kw × 0.823 kg/kWh` — real-time emission rate

People Distribution & Airflow Plan

Parameter	Source	Description
Left Zone	People with `center_x < frame_width/3`	Count of people in the left third of camera view
Centre Zone	People with `frame_width/3 ≤ center_x < 2×frame_width/3`	Count of people in the centre third
Right Zone	People with `center_x ≥ 2×frame_width/3`	Count of people in the right third
Recommended Action	`build_airflow_recommendation()`	Plain-English instruction for where to direct airflow

Real-time Analytics Chart

Dataset	Y-Axis	Description
Occupancy (people)	Left axis (0-5+)	Blue line — tracked person count over time
HVAC Power (kW)	Right axis (0-5)	Yellow line — estimated power consumption

Calibration Tab

Parameter	Description
HVAC 1-4 Position	Pixel coordinates (x, y) of where each HVAC unit is located relative to camera frame
Calibration Status	Whether each HVAC unit has been mapped to camera coordinates
2D Plan Map	Live canvas showing HVAC positions, person dots, and cooling beam targeting lines

Equations & Mathematical Models

1. Cooling Load Calculation (BTU/hr)

The total cooling load determines how much cooling capacity the room needs:

Total Load = Space Load + People Load

Space Load (BTU/hr) = Volume (ft³) × Load Factor
                    = 15,000 × 6
                    = 90,000 BTU/hr

People Load (BTU/hr) = Occupancy × Heat per Person
                     = N × 500 BTU/hr

Total Load (BTU/hr) = 90,000 + (N × 500)

Where:

15,000 ft³ = auditorium volume (~425 m³)
6 BTU/hr/ft³ = envelope + equipment load factor
500 BTU/hr/person = sensible + latent heat from a seated adult (ASHRAE standard)

2. Tonnage of Refrigeration (TR)

TR Required = Total Load (BTU/hr) ÷ 12,000

Example: 5 people
  = (90,000 + 2,500) ÷ 12,000
  = 92,500 ÷ 12,000
  = 7.71 TR

1 Ton of Refrigeration (TR) = 12,000 BTU/hr = the heat needed to melt 1 short ton of ice in 24 hours.

3. Plant Loading Percentage

Plant Loading (%) = min(100, (TR Required ÷ Installed TR) × 100)

Example: 
  = min(100, (7.71 ÷ 12.0) × 100)
  = 64.2%

This tells operators what fraction of the installed cooling infrastructure is being utilized.

4. Zone-Level Cooling Load (Per-AC Optimization)

Each zone (front-left, front-right) is calculated independently:

Zone Space Load (BTU/hr) = Zone Area (ft²) × 50
                         = 150 × 50
                         = 7,500 BTU/hr

Zone People Load (BTU/hr) = Zone People × 500

Zone Total (BTU/hr) = Zone Space + Zone People

Zone TR Required = Zone Total ÷ 12,000

5. Dual-AC Control Logic

Based on zone TR, each AC is set individually:

Zone TR	AC Status	Temperature	Fan Speed	Power Draw
0 (empty)	OFF	26°C	OFF	0.00 kW
< 0.5 TR	ON	25°C	LOW	0.605 kW (55% of 1.1 kW)
0.5–0.9 TR	ON	23°C	MEDIUM	0.880 kW (80% of 1.1 kW)
≥ 0.9 TR	ON	21°C	HIGH	1.100 kW (100%)

6. Energy Savings Calculation

Baseline Power = 3.2 kW (conventional always-on operation)
Current Power  = mode_powers[hvac_mode]   (0.15 to 4.0 kW)

Energy Saved per tick = (Baseline - Current) × (Δt / 3600) kWh
Cumulative Savings (₹) = Total kWh Saved × ₹8.0/kWh

7. Carbon Emission Calculation

CO₂ Avoided (kg) = Energy Saved (kWh) × 0.823 kg CO₂/kWh

Real-time CO₂ Rate = Current Power (kW) × 0.823 kg/kWh

The emission factor 0.823 kg CO₂/kWh is the CEA (Central Electricity Authority, India) weighted-average grid emission factor.

8. Centroid Tracking Speed (EMA)

The tracker calculates per-person movement speed using Exponential Moving Average:

Raw Speed = ‖centroid_current − centroid_previous‖₂   (Euclidean distance in pixels)

Smoothed Speed = α × Raw Speed + (1 − α) × Previous Smoothed Speed

Where α = 0.3 (smoothing factor)

9. Comfort Score

If HVAC mode < ideal mode for occupancy:
    Comfort = max(50, 100 − (mode_gap × 20))     [under-cooling penalty]

If HVAC mode > ideal mode:
    Comfort = 95                                    [over-cooling, energy waste]

If HVAC mode = ideal mode:
    Comfort = 98 (occupied) or 100 (empty)

10. Activity Level Classification

Activity Level:
  "HIGH"   → sum of all tracked speeds > 50 px/frame
  "MEDIUM" → occupancy > 0 and speeds ≤ 50
  "LOW"    → occupancy = 0

State Machine & Decision Engine

Room States & Transitions

                    ┌────────────────────────────────────┐
                    │         COOLING_BOOST (4)          │
                    │  Triggered by rapid entry          │
                    │  (3+ people in 10s from EMPTY/IDLE)│
                    │  Times out after 90s               │
                    └────────────────┬───────────────────┘
                                     │ timeout
                    ┌────────────────▼───────────────────┐
                    │          HIGH_LOAD (3)             │
                    │  6+ people                         │
                    └────────────────┬───────────────────┘
                                     │ count drops
                    ┌────────────────▼───────────────────┐
                    │           BUSY (3)                 │
                    │  3-5 people, high activity (>30)   │
                    └────────────────┬───────────────────┘
                                     │ activity drops
                    ┌────────────────▼───────────────────┐
                    │         OCCUPIED (2)               │
                    │  3-5 people low activity, or       │
                    │  1-2 people high activity (>25)    │
                    └────────────────┬───────────────────┘
                                     │ count drops
                    ┌────────────────▼───────────────────┐
                    │           IDLE (1)                 │
                    │  1-2 people, low activity          │
                    │                                    │
                    └────────────────┬───────────────────┘
                                     │ count = 0, no motion
                    ┌────────────────▼───────────────────┐
                    │          EMPTY (0)                 │
                    │  No people, no motion              │
                    └────────────────────────────────────┘

Hysteresis Rules

Rule	Behavior	Purpose
Rapid Ramp-Up	If target state priority > current → transition immediately	Don't leave people sweating
Delayed Ramp-Down	If target state priority < current → wait 30 seconds	Prevents oscillation from brief exits
Boost Timeout	COOLING_BOOST → normal state after 90 seconds	Prevents indefinite overdrive

HVAC Mode Mapping

Room State	HVAC Mode	Mode ID	Target Temp	Power (kW)
EMPTY	ECO	0	26°C	0.15
IDLE	LOW_LOAD	1	24°C	1.10
OCCUPIED	COMFORT	2	22°C	2.20
BUSY	HIGH_LOAD	3	20°C	3.20
HIGH_LOAD	HIGH_LOAD	3	20°C	3.20
COOLING_BOOST	BOOST	4	18°C	4.00

Data Flow & Interlinking

The system is a closed-loop pipeline where every component feeds into the next:

CAMERA FRAME
     │
     ▼
[YOLOv8 Detector] ──▶ List of bounding boxes [x1,y1,x2,y2]
     │
     ▼
[Centroid Tracker] ──▶ Persistent IDs + Bounding Boxes + Speeds
     │
     ├──▶ [Zone Assignment] ──▶ front_zone_1 count, front_zone_2 count
     │         │
     │         ▼
     │    [Zone Cooling Load] ──▶ BTU/hr per zone ──▶ TR per zone
     │         │
     │         ▼
     │    [Dual-AC Optimizer] ──▶ AC1 commands (Serial to Arduino)
     │         │                  AC2 commands (UDP to Wi-Fi secondary)
     │         │
     │         ▼
     │    [Serial Bridge] ──▶ Arduino Uno (relay, LEDs)
     │                        ◀── MCU state telemetry
     │
     ├──▶ [State Machine] ──▶ Room State + HVAC Mode + Comfort Score
     │         │
     │         ▼
     │    [Energy Calculator] ──▶ Power (kW), Savings (₹), CO₂ (kg)
     │
     ├──▶ [3-Zone Distribution] ──▶ Left/Centre/Right people counts
     │
     ├──▶ [Thermal Model] ──▶ Cooling TR, Plant Loading %, CO₂/hr
     │
     └──▶ [Dashboard Server] ──▶ WebSocket broadcast to browser
              │
              ▼
         [Browser UI] ──▶ Charts, Metrics, Video Feed, Calibration Map

How Components Interlink

Detector → Tracker: Raw bounding boxes become persistent person IDs with speed
Tracker → Zone Assignment: Person center-x determines which zone they're in
Zone Assignment → AC Optimizer: Zone population drives per-zone cooling load calculation
AC Optimizer → Serial Bridge: Translated to relay commands for Arduino Uno (AC1) and UDP packets for Wi-Fi controller (AC2)
Tracker → State Machine: Person count + speed feed into temporal intelligence engine
State Machine → Energy Calculator: HVAC mode determines power draw, savings vs baseline
Everything → Dashboard: All metrics are compiled into a single WebSocket payload broadcast every 1 second

Dual-Zone AC Optimization

The room is split into two front zones (left half and right half of the camera frame):

┌─────────────────────────────────────────┐
│              CAMERA VIEW                │
│                                         │
│   ┌──────────────┬──────────────┐       │
│   │  FRONT       │  FRONT       │       │
│   │  ZONE 1      │  ZONE 2      │       │
│   │  (AC1)       │  (AC2)       │       │
│   │              │              │       │
│   │  👤 👤       │  👤          │       │
│   │              │              │       │
│   └──────────────┴──────────────┘       │
│          boundary = frame_width/2       │
└─────────────────────────────────────────┘

AC1 (Front Zone 1) is controlled via Serial → Arduino Uno relay
AC2 (Front Zone 2) is controlled via Wi-Fi UDP → Secondary Arduino IR blaster

Each AC is independently set to the optimal temperature and fan speed based only on the people in its zone. If one zone is empty, that AC turns off — saving energy while the other zone still provides full comfort.

Setup & Installation

Prerequisites

Python 3.10+
pip (Python package manager)
Arduino IDE (for firmware upload)
A webcam (Logitech C270 recommended)

1. Clone the Repository

git clone <repository-url>
cd ard18

2. Install Python Dependencies

pip install -r requirements.txt

The key dependencies are:

fastapi + uvicorn — Web server and dashboard
opencv-python — Video capture and image processing
numpy — Numerical operations
onnxruntime — YOLOv8 model inference
pyserial — Arduino serial communication
websockets — Real-time dashboard updates

3. Download YOLOv8 Model

The YOLOv8n ONNX model should be placed at:

linux/config/yolov8n.onnx

If missing, run the download script:

python linux/src/download_model.py

4. Flash Arduino Firmware

Open arduino/src/Main/Main.ino in Arduino IDE
Select Arduino Uno board and correct COM port
Upload

For the secondary AC controller:

Open arduino/src/Secondary/Secondary.ino
Upload to the secondary Arduino board

Running the System

Full Mode (with Arduino)

cd linux
python run.py --port COM3 --camera 0

Simulation Mode (no hardware)

cd linux
python run.py --no-serial --camera 0

If no webcam is available, the system automatically enters Virtual Room Simulator mode with animated person movement.

Command-Line Arguments

Argument	Default	Description
`--port`	`COM3`	Serial port for Arduino Uno
`--camera`	`0`	Camera index (0 = try external USB first, then built-in)
`--no-serial`	`false`	Run without physical Arduino
`--web-port`	`8000`	FastAPI dashboard port

Access the Dashboard

Open your browser to:

http://localhost:8000

API Endpoints

Method	Endpoint	Description
`GET`	`/`	Dashboard HTML page
`GET`	`/api/video_feed`	MJPEG video stream
`GET`	`/api/telemetry`	JSON snapshot of all current metrics
`WS`	`/ws/telemetry`	WebSocket — real-time telemetry broadcast (1Hz)
`POST`	`/api/override`	Manual HVAC mode override (`{"mode": 0-4}`)
`POST`	`/api/camera`	Switch camera source (`{"camera_index": 0 or 2}`)
`POST`	`/api/calibrate`	Capture HVAC position calibration (`{"hvac_id": 1-4}`)

Project Structure

ard18/
├── README.md                          # This file
├── explanation.md                     # Brief explanation with demo points
├── requirements.txt                   # Python dependencies
│
├── arduino/
│   └── src/
│       ├── Main/                      # Primary Arduino firmware
│       │   ├── Main.ino               # Main loop (serial + watchdog)
│       │   ├── HVACControl.cpp/h      # LED + relay mode control
│       │   └── SerialComm.cpp/h       # Serial protocol handler
│       └── Secondary/                 # Secondary AC controller
│           ├── Secondary.ino          # IR blaster + receiver loop
│           ├── MideaIR.cpp/h          # Midea AC IR protocol encoder
│           └── wifi_bridge.py         # Wi-Fi UDP bridge script
│
├── linux/
│   ├── run.py                         # Main entry point
│   ├── runnew.py                      # Alternative entry point
│   ├── config/
│   │   └── yolov8n.onnx              # YOLOv8n neural network model
│   └── src/
│       ├── detector.py                # YOLOv8 ONNX person detection
│       ├── tracker.py                 # Centroid multi-object tracker
│       ├── state_machine.py           # Room Intelligence Engine
│       ├── serial_bridge.py           # Arduino serial bridge
│       └── server.py                  # FastAPI dashboard + WebSocket
│
├── calibration/                       # HVAC position calibration data
│   ├── calibration_coords.json        # Saved (x,y) per HVAC unit
│   └── hvac_*_calib.jpg              # Annotated calibration snapshots
│
└── scratch/                           # Development utility scripts

Troubleshooting

Problem	Solution
"Camera not detected"	System auto-switches to simulator mode. Check USB webcam connection
Dashboard shows "MCU: Mock Link"	Arduino not connected or wrong COM port. Use `--no-serial` for demo
Port 8000 busy	System auto-increments port. Check console for actual URL
All metrics frozen at zero	WebSocket `/ws/telemetry` should be connected. Check browser console
HVAC mode oscillating rapidly	Debounce is set to 30s. If still oscillating, increase `debounce_down_seconds`

License

This project was built for the Arduino HVAC Hackathon competition. See repository license for terms.

Name		Name	Last commit message	Last commit date
arduino/src		arduino/src
calibration		calibration
images		images
linux		linux
scratch		scratch
README.md		README.md
brief_explanation.md		brief_explanation.md
camera_success_idx_1_MSMF.jpg		camera_success_idx_1_MSMF.jpg
camera_test_DEFAULT_0.jpg		camera_test_DEFAULT_0.jpg
explanation.md		explanation.md
requirements.txt		requirements.txt
test_cameras.py		test_cameras.py
test_index_1.py		test_index_1.py
test_secondary.py		test_secondary.py

Latest commit

History

Folders and files

Repository files navigation