Proxim

Demo Video (3-5 min): ADD_YOUR_VIDEO_URL_HERE

Proxim is a browser-based training lab where pharmaceutical sales representatives rehearse live conversations with AI-simulated healthcare professionals. Each persona speaks with streamed dialogue, ElevenLabs speech, and timestamp/alignment-derived viseme keyframes that drive a 3D avatar in Three.js, keeping voice and lip motion tightly coupled for a face-to-face feel without native binary installs.

Hackathon Requirements Checklist

Requirement	Status	Evidence
Real-time lip sync (<300ms target)	Partial	Streaming audio+visemes in server/src/websocket/wsHandler.ts; diagnostics in client/src/components/LipSyncDiagnostics.tsx; benchmark procedure in docs/latency-benchmarks.md
Voice input + transcription	Pass	client/src/hooks/useVoiceInput.ts (Web Speech + MediaRecorder fallback)
LLM integration (Claude/GPT-4)	Partial	Claude integration in server/src/services/claudeService.ts; GPT-4 dialogue adapter pending
Expressive facial animation	Partial	Emotion tags + morph driving in server/src/services/claudeService.ts and client/src/components/AvatarModel.tsx
Browser-based delivery	Pass	React/Vite client + Node/ws server
Persona configuration	Pass	client/src/config/personas.json and server/src/config/personas.json
Multi-turn session context (10-15+)	Pass	Session history per sessionId in server/src/websocket/wsHandler.ts
Mobile browser support	Partial	Mobile fallback flow in client/src/hooks/useVoiceInput.ts
Containerizable (Docker preferred)	Pass	Dockerfile.server, docker-compose.yml

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         Browser (React + Vite)                       │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────┐  ┌─────────────┐ │
│  │ VoiceInput   │  │ WebSocket    │  │ Web Audio  │  │ R3F Avatar  │ │
│  │ Web Speech + │─▶│ useWebSocket │─▶│ playback + │─▶│ morph + idle│ │
│  │ silence 1.5s │  │              │  │ viseme sync│  │ blink + sway│ │
│  └──────────────┘  └──────┬───────┘  └────────────┘  └─────────────┘ │
└───────────────────────────┼──────────────────────────────────────────┘
                            │ ws:// + http://
                            ▼
┌─────────────────────────────────────────────────────────────────────┐
│                 Node.js + Express + ws (TypeScript)                  │
│  user_input ─▶ Claude Sonnet stream ─▶ sentence buffer              │
│              ─▶ ElevenLabs with-timestamps (audio + alignment)       │
│                  (text-viseme fallback if alignment unavailable)      │
│              ─▶ { audio_chunk + visemes } frames over WebSocket      │
└─────────────────────────────────────────────────────────────────────┘

Repository layout

• client/ — React 18, TypeScript, Vite, Tailwind, Zustand, React Three Fiber avatar renderer.
• server/ — Express HTTP + WebSocket server, Groq/Anthropic + ElevenLabs + alignment-based lip-sync.
• Dockerfile.server + docker-compose.yml — containerized backend (no native binaries required).
• docs/ — submission docs (technical approach, latency, licensing).

Prerequisites

• Node.js 20+
• Anthropic API key (ANTHROPIC_API_KEY)
• ElevenLabs API key (ELEVENLABS_API_KEY) — voice IDs are pre-configured in personas.json
• Optional: OPENAI_API_KEY for Whisper transcription when Web Speech API is unavailable

Setup

git clone <your-repo-url> proxim
cd proxim

# Server
cd server
cp .env.example .env
# Minimum: fill in ANTHROPIC_API_KEY and ELEVENLABS_API_KEY
npm install
npm run dev

# Client (new terminal)
cd ../client
cp .env.example .env
npm install
npm run dev

Defaults:

• Client Vite dev server: http://localhost:5173
• Backend HTTP + WS: http://localhost:3001 / ws://localhost:3001

Docker (backend only)

cp server/.env.example server/.env
# Populate ANTHROPIC_API_KEY and ELEVENLABS_API_KEY
docker compose up --build

The UI still runs via npm run dev inside client/ unless you also build static assets and serve them separately.

Personas

Three HCP personas are pre-configured in personas.json.

Submission Docs

• Technical approach: docs/technical-approach.md
• Latency benchmarking: docs/latency-benchmarks.md
• Third-party licensing: docs/licensing.md

Known limitations

• Binary WebSocket frames are not yet used; audio is base64 JSON.
• Sentence buffering trades some latency for sync stability.
• GPT-4 as primary dialogue path is not yet wired.
• Mobile Safari speech support can vary by version/device.

Third-party APIs

• Anthropic — Claude API
• ElevenLabs — TTS + timestamps
• OpenAI — optional Whisper transcription
• Ready Player Me — avatar GLB models

License

MIT