OmniChat

A private, CPU-first multimodal RAG assistant engineered end to end under strict compute, cost, privacy, and deployment constraints.

OmniChat Phase 1 is a standalone local AI system built for a CPU-only Windows laptop. It combines authenticated multi-user chat, local document retrieval with citations, images, speech, conversation management, exports, administration, and secure remote demonstration through an outbound tunnel. The operational source and data remain private; this repository is a sanitized engineering case study.

Why this project matters

Many AI demos assume cloud APIs, expensive GPUs, public data, and unlimited memory. OmniChat explored the opposite constraint set:

• local inference on modest CPU hardware
• no paid model API required for core features
• documents and conversations retained on the active host
• several signed-in users with serialized inference to prevent resource exhaustion
• public demonstration without opening router ports or exposing model/database services
• backup and portability planning between laptops

The result is not positioned as a high-concurrency production service. It is a complete Phase 1 demonstration of product engineering under real constraints.

Capabilities

• Secure accounts, administrator role, TOTP, server-side sessions, CSRF and rate-limit protections
• Private user conversations and shared administrator-managed knowledge base
• Local text generation and image understanding with model-aware routing
• Document ingestion and RAG citations
• Browser audio capture, local transcription, and offline speech where supported
• Streaming responses, cancellation, queue visibility, and resilient UI states
• Conversation search, rename, pin, archive, export, and responsive workspace design
• Local-only internal services and public access through an outbound tunnel
• Health, audit, backup, restore, and operator documentation

Architecture

flowchart LR
  R[Remote reviewer] -->|HTTPS| T[Outbound tunnel]
  L[Local user] -->|localhost| W[Next.js workspace]
  T --> W
  W -->|private container network| A[FastAPI]
  A --> Q[Serialized CPU queue]
  Q --> O[Ollama local models]
  A --> D[(SQLite)]
  A --> V[(Local vector index)]
  A --> F[Uploads and exports]

Loading

Only the web workspace is published. Ollama, backend APIs, databases, indexes, debug endpoints, and files are not directly exposed.

Security boundary

• Public registration disabled; administrators provision accounts
• Password hashing and administrator TOTP
• Secure cookie sessions, CSRF protection, authorization and ownership checks
• File type/size/decompression limits and untrusted-document handling
• No public model, database, vector, filesystem, Docker, metrics, or debug port
• Audit events avoid passwords, prompts, documents, tokens, and secrets
• Sanitized public assets contain synthetic content only

This design improves exposure control but does not claim formal certification or complete protection against every threat.

Performance and hardware reporting

The archived Phase 1 table reports the actual two-run text observations, focused domain smoke checks, and a cold vision observation with explicit limitations. The host used the 16 GB CPU-only Windows profile, but the archived table did not capture the CPU model or power state, so the figures are not presented as a complete hardware comparison. See BENCHMARK_RESULTS.md and BENCHMARK_PROTOCOL.md.

Limitations

• CPU inference is slower than modern GPU or hosted inference.
• Heavy model work is serialized; multiple users may wait in a visible queue.
• The public link works only while the active host, application, model service, and tunnel are running.
• Speech, vision, and long documents have higher latency and memory cost.
• It is a Phase 1 local deployment and demonstration, not an enterprise high-availability product.
• It is not represented as officially deployed or endorsed by NCP or any institution.

Verification performed in the private project

The operational project includes frontend and backend tests, ownership/access checks, model/media failure handling, migration and backup procedures, deployment diagnostics, and responsive browser QA. Public artifacts are separately reviewed for secrets, privacy, paths, hostnames, branding, and synthetic content.

Engineering lessons

1. Constraint-driven design is a feature: queues, fallbacks, explicit service states, and bounded uploads make local AI dependable.
2. RAG quality is a pipeline problem: extraction, metadata, retrieval, citations, deletion, and evaluation matter as much as the model.
3. Local inference and public access are separate design axes; an outbound tunnel can publish the UI without publishing internal services.
4. Model size is not product quality. A smaller responsive model can provide a better demonstration on limited hardware.
5. Security and UX meet at failure states: users need clear messages for unavailable models, uploads, queues, and permissions.

Repository boundary

This public repository does not contain operational source, .env files, credentials, TOTP material, databases, vector indexes, uploads, logs, backups, model weights, private documents, user data, internal paths, or permanent demo credentials.

Portfolio summary

OmniChat demonstrates full-stack AI engineering, local model integration, RAG, multimodal UX, authentication and access control, CPU-aware concurrency, Docker/Windows operations, secure remote access, testing, documentation, and product tradeoff judgment.