Epic: add-regression-detection

Overview

Add a built-in quality monitoring system to opencode. Every AI response is scored 0-100 using code correctness, user feedback, and static analysis signals. Scores are stored in a per-project SQLite database (via bun:sqlite). The system detects regressions by comparing against rolling averages, generates diagnostic reasoning for low scores, and exposes a /quality-stats command with dashboard summary and drill-down. A toggleable log file provides full diagnostic tracing.

Architecture Decisions

• bun:sqlite for storage — Bun has built-in SQLite support, zero additional dependencies needed. Per-project DB stored at .opencode/quality.db (configurable).
• Event bus integration — Subscribe to existing Session.Event.Updated, Session.Event.Diff, LSPClient.Event.Diagnostics, and Command.Event.Executed events. No modifications to existing event producers needed.
• Async scoring pipeline — Scoring runs in a microtask after response delivery via queueMicrotask() or setTimeout(0) to ensure zero impact on response latency.
• New src/quality/ module — Self-contained module following existing patterns (Instance.state() for per-project state, Zod schemas for type safety, Bus.subscribe() for event collection).
• Config extension — Add quality key to Config.Info schema following existing patterns (e.g., similar to compaction or experimental sections).
• /quality-stats as a slash command — Register via the existing command system (src/command/) as a built-in command, rendering formatted output to the TUI.

Technical Approach

New Module: src/quality/

packages/opencode/src/quality/
  index.ts        — Public API, Instance.state() init, event subscriptions
  schema.ts       — Zod schemas (ResponseScore, SessionScore, Regression, Config)
  store.ts        — SQLite wrapper (init, migrations, CRUD, queries, retention cleanup)
  scorer.ts       — Scoring engine (sub-score computation, composite weighting)
  collector.ts    — Event listeners, signal aggregation, user feedback inference
  detector.ts     — Regression detection (rolling averages, severity, diagnostic reasoning)
  logger.ts       — Diagnostic log file (rotation, levels, formatting)

Data Flow

Response delivered → Session.Event.Updated fires
    → collector.ts captures: modelID, providerID, tokens, cost, tool results, errors
    → scorer.ts computes sub-scores asynchronously:
        - correctness: LSP diagnostics delta, tool success/failure, error presence
        - userFeedback: deferred — updated when user accepts/reverts/redoes
        - staticAnalysis: LSP diagnostic count delta, severity-weighted
    → store.ts persists to SQLite
    → detector.ts compares against rolling average, flags regressions
    → logger.ts writes to log file
    → If regression severity >= alertLevel, emit Quality.Event.Regression for TUI warning

SQLite Schema

CREATE TABLE responses (
  id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  model_id TEXT NOT NULL,
  provider_id TEXT NOT NULL,
  timestamp INTEGER NOT NULL,
  prompt_hash TEXT,
  response_hash TEXT,
  composite_score REAL,
  correctness_score REAL,
  feedback_score REAL,
  analysis_score REAL,
  diagnostic_notes TEXT,  -- JSON
  is_collaborative INTEGER DEFAULT 0,
  tokens_input INTEGER,
  tokens_output INTEGER,
  cost REAL
);

CREATE TABLE regressions (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  response_id TEXT NOT NULL REFERENCES responses(id),
  detected_at INTEGER NOT NULL,
  score_delta REAL NOT NULL,
  severity TEXT NOT NULL,  -- minor|moderate|severe
  component TEXT,          -- correctness|feedback|analysis|composite
  description TEXT
);

CREATE INDEX idx_responses_session ON responses(session_id);
CREATE INDEX idx_responses_model ON responses(model_id);
CREATE INDEX idx_responses_timestamp ON responses(timestamp);
CREATE INDEX idx_regressions_severity ON regressions(severity);

Sessions and model aggregates computed via SQL queries (no denormalized tables needed — SQLite is fast enough for local data).

User Feedback Signal Collection

Infer implicit feedback from observable user actions:

• Accept: User continues working after response (no revert, no redo) — positive signal
• Revert: Session.Info.revert is populated — strong negative signal
• Redo: User sends a correction prompt referencing the same topic — moderate negative signal (detected via prompt similarity heuristic)
• Edit after apply: User modifies files that were just changed by the AI — mild negative signal

Feedback scores are updated retroactively in the database as signals arrive.

Regression Detection Algorithm

1. Fetch rolling window (last N responses, default 20) from SQLite
2. Compute rolling average for composite and each sub-score
3. Calculate delta: current_score - rolling_average
4. Classify severity:
   • minor: delta between -5 and -15
   • moderate: delta between -15 and -30
   • severe: delta below -30
5. Generate diagnostic reasoning by identifying which sub-score(s) dropped and correlating with observable signals (e.g., "3 new type errors detected", "user reverted previous change")

/quality-stats Command

Registered as built-in command. Output modes:

Default summary — formatted terminal table:

Quality Dashboard (last 30 days)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall: 78/100 (▲ +3 from last week)

Model Comparison:
  claude-opus-4-6    82 avg  (142 responses)  ▲ +2
  claude-sonnet-4-5  74 avg  (89 responses)   ▼ -1

Recent Regressions:
  ⚠️  moderate  2h ago  correctness dropped (3 type errors)
  ●   minor    1d ago  user reverted change

This Session: 81/100 (4 responses)

--session <id> — per-response score timeline with diagnostic notes
--model <id> — score distribution and trend for specific model
--export json|csv — machine-readable export

Configuration

Extend Config.Info with:

quality: z.object({
  enabled: z.boolean().default(true),
  logging: z.boolean().default(true),
  logLevel: z.enum(["minimal", "normal", "verbose"]).default("normal"),
  logPath: z.string().default(".opencode/quality.log"),
  dbPath: z.string().default(".opencode/quality.db"),
  retentionDays: z.number().default(90),
  weights: z.object({
    correctness: z.number().default(40),
    userFeedback: z.number().default(35),
    staticAnalysis: z.number().default(25),
  }).default({}),
  regression: z.object({
    threshold: z.number().default(15),
    windowSize: z.number().default(20),
    alertLevel: z.enum(["minor", "moderate", "severe"]).default("moderate"),
  }).default({}),
}).optional()

Implementation Strategy

Phase 1 (Foundation): Storage + config + scoring engine — get data flowing into SQLite
Phase 2 (Intelligence): Regression detection + diagnostic reasoning + logging
Phase 3 (Presentation): /quality-stats command + TUI regression warnings

Each phase is independently testable and delivers incremental value.

Task Breakdown Preview

• Task 1: SQLite storage layer — Schema, migrations, CRUD operations, retention cleanup, bun:sqlite integration
• Task 2: Quality config — Extend Config.Info with quality section, defaults, validation
• Task 3: Scoring engine — Sub-score computation (correctness, feedback, static analysis), composite weighting, async execution
• Task 4: Event collector — Subscribe to session/LSP/command events, aggregate signals, infer user feedback
• Task 5: Regression detection — Rolling average comparison, severity classification, diagnostic reasoning generation
• Task 6: Diagnostic logging — Log file writer, rotation, level filtering, formatting
• Task 7: /quality-stats command — Summary dashboard, session drill-down, model drill-down, JSON/CSV export
• Task 8: Tests — Unit tests for scorer, detector, store; integration test for full pipeline

Dependencies

Internal

• src/bus/ — Event subscriptions (no changes needed)
• src/session/ — Message metadata access (no changes needed)
• src/lsp/ — Diagnostic data (no changes needed)
• src/provider/ — Model/provider identification (no changes needed)
• src/config/config.ts — Add quality config key (schema extension)
• src/command/index.ts — Register /quality-stats command

External

• bun:sqlite — Built-in, no package installation needed

Success Criteria (Technical)

• Scoring adds < 5ms to response processing (async, non-blocking)
• SQLite queries for /quality-stats summary complete in < 200ms with 10K responses
• Database stays under 50MB for 1 year of daily use
• Zero test failures in existing test suite after integration
• Scoring failures are caught and logged without affecting main workflow
• All 6 PRD user stories have corresponding implementation coverage

Estimated Effort

• 8 tasks total, each independently implementable
• Tasks 1-4 (foundation) can be parallelized partially (1+2 in parallel, then 3+4)
• Tasks 5-6 (intelligence) depend on 1-4
• Task 7 (presentation) depends on 1-5
• Task 8 (tests) runs alongside all phases
• Critical path: Storage → Scoring → Detection → Stats command

Tasks Created

• SQLite storage layer #13202 - SQLite storage layer (parallel: true)
• Quality configuration #13207 - Quality configuration (parallel: true)
• Scoring engine #13208 - Scoring engine (parallel: false, depends: SQLite storage layer #13202, Quality configuration #13207)
• Event collector and user feedback tracking #13209 - Event collector and user feedback tracking (parallel: false, depends: SQLite storage layer #13202, Quality configuration #13207, Scoring engine #13208)
• Regression detection and diagnostic reasoning #13203 - Regression detection and diagnostic reasoning (parallel: false, depends: SQLite storage layer #13202, Scoring engine #13208)
• Diagnostic logging #13204 - Diagnostic logging (parallel: true, depends: Quality configuration #13207, Scoring engine #13208)
• /quality-stats command #13205 - /quality-stats command (parallel: false, depends: SQLite storage layer #13202-Scoring engine #13208, Regression detection and diagnostic reasoning #13203)
• Tests for quality monitoring system #13206 - Tests for quality monitoring system (parallel: false, depends: all)

Total tasks: 8
Parallel tasks: 3 (001, 002, 006)
Sequential tasks: 5 (003, 004, 005, 007, 008)
Estimated total effort: 51-63 hours