Skip to content

Regression detection and diagnostic reasoning #13203

Closed

Description

@florianleibert

Task: Regression detection and diagnostic reasoning

Description

Implement the regression detection algorithm that compares each response's score against rolling averages and flags quality drops. Includes severity classification (minor/moderate/severe) and diagnostic reasoning that explains why a score was low, correlating sub-score drops with observable signals.

Acceptance Criteria

  • packages/opencode/src/quality/detector.ts created
  • Fetches rolling window (last N responses, configurable via regression.windowSize) from SQLite
  • Computes rolling average for composite score and each sub-score
  • Calculates delta: current_score - rolling_average
  • Severity classification: minor (-5 to -15), moderate (-15 to -30), severe (below -30)
  • Regressions persisted to regressions table with: response_id, score_delta, severity, component, description
  • Diagnostic reasoning generated for each regression: identifies which sub-scores dropped, correlates with signals (e.g., "3 new type errors", "user reverted change", "tool invocation failed")
  • Emits Quality.Event.Regression event on bus when severity >= configured alertLevel
  • Handles edge cases: fewer than windowSize responses (use available data), first response (no comparison, no regression)
  • Regression detection runs as part of the scoring pipeline (after score computed, before response to user)

Technical Details

  • Define Quality.Event.Regression using BusEvent.define() pattern
  • Query rolling averages from SQLite: SELECT AVG(composite_score), AVG(correctness_score), AVG(feedback_score), AVG(analysis_score) FROM responses ORDER BY timestamp DESC LIMIT ?
  • Diagnostic reasoning uses a template system: check each sub-score delta, match to known signal patterns, compose human-readable explanation
  • Regression event payload: { responseId, severity, delta, component, description } — TUI can subscribe to show inline warnings

Dependencies

  • Task 001: SQLite storage layer (for queries)
  • Task 003: Scoring engine (provides scores to compare)

Effort Estimate

  • Size: M
  • Hours: 6-8
  • Parallel: false (depends on 001, 003)

Activity

  1. github-actions commented on Apr 13, 2026

    @github-actions
    Contributor

    To stay organized issues are automatically closed after 90 days of no activity. If the issue is still relevant please open a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions