Problem

NEXUS recommends local models based on detected GPU hardware, but CI only runs on standard GitHub runners (no GPU). There is no automated validation that:

• detectGPU() parses real nvidia-smi / system_profiler output correctly
• Recommended models actually fit in detected VRAM
• MCP tool response times match documented benchmarks

Design Principles

• Benchmarks never block releases. CI is the release gate. Benchmarks are a post-release quality signal.
• Results are editable after release. Run benchmarks when hardware is available, update release notes as results come in. No new tag needed.
• Contributors can submit results. Anyone with hardware runs the script, submits JSON via PR or issue.
• TUI integration. nexus benchmark command runs the suite locally. Opt-in anonymous upload for community dataset.
• Uses unified data schema. See Benchmark data schema — hardware specs, model throughput, and test result format #74 for the shared schema across benchmarks, sessions, and analytics.

Hardware Tiers

Release Notes Format

## Hardware Benchmarks
- ✅ Low tier (RTX 3050, 4GB) — tested
- ⏳ Mid tier — pending
- ⏳ High tier — pending

Updated as results arrive. GitHub release notes are editable post-publish.

Phased Rollout

Phase 1 — v0.2.0: Benchmark script + repo storage

• Ship scripts/benchmark.sh and TUI nexus benchmark command
• Detects hardware, pulls recommended models, runs all 6 MCP tools with standardized inputs
• Outputs schema-compliant JSON (Benchmark data schema — hardware specs, model throughput, and test result format #74)
• Store results in docs/benchmarks/<version>/<tier>.json
• Release checklist includes benchmark run (non-blocking)

Phase 2 — v0.3.0: Cloud GPU + remote storage

• Cloud GPU scripting (Vast.ai or Lambda Cloud): spin up, benchmark, tear down (~pennies per release)
• Migrate storage from repo to S3/R2
• TUI opt-in upload for community-contributed results
• Quality score (1-5) added to benchmark results

Phase 3 — v0.4.0+: Website + automation

• Website reads benchmark data from S3/R2, renders charts and hardware comparisons
• Optional self-hosted GitHub Actions runners for fully automated per-release benchmarks
• Same data store serves website, TUI dashboard, and future Pro analytics

Acceptance Criteria (Phase 1)

• scripts/benchmark.sh exists and produces schema-compliant JSON (Benchmark data schema — hardware specs, model throughput, and test result format #74)
• TUI nexus benchmark command wraps the same logic
• Covers: GPU detection, model pull, all 6 MCP tools with sample inputs, response times
• Baseline thresholds defined per tier
• docs/benchmarks/ directory with at least one versioned result
• Release checklist documents benchmark as non-blocking step

Related

• Benchmark data schema — hardware specs, model throughput, and test result format #74 Unified data schema
• Session logging — track model usage, latency, and costs #9 Session logging
• Cost tracker — monitor cloud API spend vs local compute #11 Cost tracker
• Hardware detection and model recommendations in TUI #29 Hardware detection (shipped in feat: strategic research items — positioning, MCP improvements, TUI e… #73)
• Community benchmark submissions #30 Benchmarks
• docs/model-configuration.md (existing manual test results)