Skip to content

Scoring engine #13208

Closed
Closed

Description

@florianleibert

Task: Scoring engine

Description

Implement the core scoring engine that computes a composite 0-100 quality score for each AI response. The engine calculates three sub-scores (code correctness, user feedback, static analysis) and combines them using configurable weights. Scoring runs asynchronously to avoid blocking response delivery.

Acceptance Criteria

  • packages/opencode/src/quality/scorer.ts created
  • Correctness sub-score (0-100): based on tool invocation success/failure, error presence in response, LSP diagnostic delta after changes
  • User feedback sub-score (0-100): starts at neutral (70), adjusted retroactively when signals arrive (accept: +, revert: -30, redo: -15, edit-after-apply: -5)
  • Static analysis sub-score (0-100): based on LSP diagnostic count delta (new errors: heavy penalty, new warnings: moderate, resolved issues: bonus)
  • Composite score = weighted sum using config weights, clamped to 0-100
  • Scoring is async — uses queueMicrotask() or setTimeout(0) after response delivery
  • Partial scoring supported — if LSP data unavailable, score with available signals only (weight redistributed)
  • Score results persisted to SQLite via store.ts
  • Scoring errors caught and logged, never propagated to main workflow

Technical Details

  • Import config weights from quality config
  • Import store functions from store.ts for persistence
  • Correctness signals extracted from message metadata: tool results (success/error), response errors, token usage patterns
  • Static analysis signals from LSP diagnostics snapshot before/after response application
  • User feedback is deferred — initial score computed without it, updated retroactively
  • Expose Quality.scoreResponse(messageMetadata, lspBefore, lspAfter) as main API

Dependencies

  • Task 001: SQLite storage layer (for persistence)
  • Task 002: Quality configuration (for weights and settings)

Effort Estimate

  • Size: L
  • Hours: 8-10
  • Parallel: false (depends on 001, 002)

Activity

  1. github-actions commented on Apr 13, 2026

    @github-actions
    Contributor

    To stay organized issues are automatically closed after 90 days of no activity. If the issue is still relevant please open a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions