Skip to content

feat: add repeated-trial stability analysis - #3483

Draft
nik13 wants to merge 2 commits into
feat/TH-8579-case-release-gatefrom
feat/TH-8586-trial-stability
Draft

nik13 wants to merge 2 commits into
feat/TH-8579-case-release-gatefrom
feat/TH-8586-trial-stability

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two agent runs can have the same mean while repeated trials are unstable. The comparison endpoint currently exposes case deltas and pooled aggregates, but no repeatability signal or per-example denominator.

Change

  • Add include_trial_stability to the existing authenticated run-comparison POST (default false).
  • When requested, return a bounded, read-only trial_stability projection for every pinned example, including missing examples on both sides.
  • Report per-side recorded/missing/assessed trial counts, finite assessed score mean/min/max/population standard deviation, and paired candidate-minus-baseline delta mean/stddev.
  • Keep duplicates, failed/unfinished/unknown/unsupported/not-assessed evidence explicit; no score or delta is invented. Null standard deviations mean fewer than two assessed values, not zero variance or statistical significance.
  • Synchronize OpenAPI, generated frontend/Zod contracts, MCP manifest, and contract fixtures.

No tables, routes, warehouse, scheduler, evaluator resolution, CI policy, or UI are added. The existing release gate remains unchanged.

Validation

  • 239 backend/PostgreSQL/MCP tests passed (including 17 new pure stability tests, existing analysis/comparison/gate suites, API route, MCP generation/registry/executor) in 16.59s.
  • 57 comparison API tests passed against owned disposable PostgreSQL (postgres:16-alpine, 127.0.0.1:15432, FI_SKIP_CH25_MIGRATION=1 FI_SKIP_CH25_SCHEMA_APPLY=1).
  • 81 persisted comparison/release-gate tests passed against the same database.
  • 114 frontend contract tests passed (read/comparison, OpenAPI, experiment); contract generation/check passed with 1,063 paths / 1,437 operations and zero contract gaps.
  • MCP manifest generation/check passed (96 tools).
  • Ruff check/format, staged-safety, and diff checks passed.

Evidence boundaries

This is a descriptive read projection over persisted evidence. It does not claim inferential statistics, confidence intervals, model/provider execution, live Observe ingestion, worker behavior, browser/UI qualification, hosted deployment, or release readiness. PostgreSQL ran in an owned disposable Docker container; ClickHouse was unavailable and its migration/schema steps were explicitly skipped.

Linear: TH-8586.

Stacked on core #3476 (c2bded160f). Internal docs and public docs/tutorials: docs #907.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant