EmotionSense

EmotionSense is a free-tier SaaS MVP for facial emotion recognition, face matching, and multi-face detection. It ships as a clean full-stack project with a Next.js frontend, FastAPI backend, Supabase PostgreSQL persistence, and a separate ML package for training and inference.

Preview

What It Does

• User signup and login with JWT authentication
• Multi-face detection from uploaded images or browser webcam capture
• Face matching against user-owned registered embeddings
• Multi-sample face enrollment and per-user registry management
• Seven-class emotion prediction with confidence scores
• Visual overlays with bounding boxes and labels in the browser
• Detection history persisted per user in Supabase-backed PostgreSQL
• CPU-friendly inference path for deployment on free tiers
• Calibrated emotion inference with test-time augmentation, context-aware face crops, and quality-aware confidence gating
• Emotion prediction debiased with class-prior correction, reliability-aware calibration, and stronger training-time regularization support
• Recognition-aware identity matching with embedding test-time augmentation, grouped identity profiles, adaptive thresholds, and duplicate-resistant enrollment
• Face-level diagnostics with top emotion candidates, quality scoring, and identity candidate inspection

Architecture

• frontend/: Next.js 14 app with Tailwind CSS, auth flow, webcam capture, and dashboard UI
• backend/: FastAPI API with JWT auth, file upload handling, SQLAlchemy data access, and ML orchestration
• ml_model/: training scripts for FER2013 and additional datasets plus calibrated inference utilities shared by the backend

For a deeper system walkthrough, see docs/ARCHITECTURE.md.

Project Structure

.
|-- backend
|   |-- app
|   |   |-- api
|   |   |   |-- routes
|   |   |-- core
|   |   |-- db
|   |   |-- models
|   |   |-- schemas
|   |   |-- services
|   |   `-- main.py
|   |-- supabase
|   |   `-- schema.sql
|   |-- tests
|   `-- requirements.txt
|-- docs
|   |-- ARCHITECTURE.md
|   |-- DEPLOYMENT.md
|   |-- images
|   |   `-- emotionsense-dashboard-preview.png
|   `-- TRAINING.md
|-- frontend
|   |-- app
|   |-- components
|   |-- lib
|   `-- package.json
|-- ml_model
|   |-- artifacts
|   |-- inference
|   `-- training
|-- .env.example
|-- render.yaml
`-- requirements.txt

Backend API

Base URL: http://localhost:8000/api

• POST /auth/register
• POST /auth/login
• POST /detect-image
• POST /register-face
• POST /register-face-batch
• POST /recognize-face
• GET /face-registry
• DELETE /face-registry/{name}
• GET /history

Example detection response:

{
  "faces": [
    {
      "name": "John",
      "emotion": "happy",
      "confidence": 0.92,
      "box": [48, 76, 132, 132],
      "detection_confidence": 0.99,
      "match_score": 0.83
    }
  ],
  "image_width": 1280,
  "image_height": 720,
  "detected_at": "2026-03-26T10:15:00Z"
}

Data Model

Supabase PostgreSQL tables are defined in backend/supabase/schema.sql.

• users
• face_embeddings
• emotion_logs

Local Setup

Recommended Python version: 3.11 or 3.12.

This project should not be set up with Python 3.13 right now because the facenet-pytorch dependency requires numpy < 2.0, while the compatible NumPy line for that constraint is 1.26.x, which is intended for Python up to 3.12. It also constrains torch, torchvision, and Pillow, so the repo pins are intentionally aligned to older compatible releases.

1. Create the database

1. Create a free Supabase project.
2. Open the SQL editor.
3. Run the contents of backend/supabase/schema.sql.
4. Copy the Supabase Postgres connection string and convert it to async SQLAlchemy format:

postgresql+asyncpg://postgres:[PASSWORD]@db.[PROJECT-REF].supabase.co:5432/postgres?ssl=require

2. Configure environment variables

1. Copy .env.example to .env.
2. Copy frontend/.env.local.example to frontend/.env.local.
3. Fill in:
   • DATABASE_URL
   • JWT_SECRET
   • CORS_ORIGINS
   • NEXT_PUBLIC_API_BASE_URL

3. Install dependencies

Backend:

python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Frontend:

cd frontend
npm install

4. Prepare the emotion model artifact

Train the model using the instructions in docs/TRAINING.md.

Expected artifact files:

• ml_model/artifacts/emotion_mobilenet_v3_small.pth
• ml_model/artifacts/emotion_labels.json

If the custom weight file is missing, the backend attempts to download a free ONNX fallback model automatically on first run. The fallback is helpful for MVP setup, but the system is materially better when you supply your own trained checkpoint and metrics file from the upgraded trainer.

5. Run locally

Backend:

python -m uvicorn backend.app.main:app --reload

Frontend:

cd frontend
npm run dev

Open http://localhost:3000.

Deployment

Free-tier deployment instructions live in docs/DEPLOYMENT.md.

• Frontend: Vercel free tier
• Backend: Render free tier
• Database: Supabase free tier
• Model training: Google Colab or Kaggle free tier

The codebase is production-structured, but a fully free hosting stack still has platform-level limits such as idle spin-down and lower uptime guarantees. That tradeoff is the main constraint of staying at $0.

Also note that Vercel's Hobby plan is a free tier aimed at personal and small-scale use, so treat the zero-cost deployment path as MVP, demo, and pilot infrastructure unless your usage and terms fit that plan.

Recommended Workflow

1. Train the emotion classifier in Colab or Kaggle, ideally with FER2013 plus additional aligned emotion datasets.
2. Upload the trained emotion_mobilenet_v3_small.pth artifact into the repo or your deployment build context.
3. Deploy the backend to Render using the root render.yaml.
4. Deploy the frontend to Vercel with frontend/ as the root directory.
5. Build each identity with several clean samples in Face Registry.
6. Use the diagnostics panel to inspect identity candidates, emotion confidence, and face-quality signals while testing uploads or webcam captures.

Notes

• The backend uses MTCNN for face detection and InceptionResnetV1 embeddings for recognition.
• Emotion inference now uses expanded face-context crops, multi-view test-time augmentation, temperature-scaled probabilities, class-prior correction, reliability-aware calibration, and signal-guided calibration before returning a label.
• Identity matching now groups multiple embeddings per person into a single profile, blends centroid plus medoid plus top-sample similarity, and uses quality-aware adaptive thresholds before returning a name.
• Enrollment rejects near-duplicate reference images so each identity profile stays cleaner and more diverse over time.
• The upgraded trainer supports MobileNetV3-Small and MobileNetV3-Large, class-balanced sampling, focal loss, mixup, macro-F1 model selection, per-class validation metrics, early stopping, and temperature calibration.
• Face registration stores embeddings per authenticated user, so each workspace can maintain its own recognition catalog.