Skip to content

docs: document opt-in trial stability summaries - #907

Draft
nik13 wants to merge 1 commit into
docs/TH-8579-case-release-gatefrom
docs/TH-8586-trial-stability
Draft

nik13 wants to merge 1 commit into
docs/TH-8579-case-release-gatefrom
docs/TH-8586-trial-stability

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Repeated trials can have the same mean while behaving very differently across attempts. The draft comparison contract adds an opt-in include_trial_stability field, but the public reference and CI workflow did not explain its denominator, eligibility rules, or limits.

Change

  • Document the REST request flag and the conditional trial_stability response.
  • Describe expected, recorded, missing, assessed, mean/min/max, and population standard deviation fields.
  • Document paired same-trial candidate-minus-baseline deltas.
  • Add an equal-mean example: baseline scores 0, 1 versus candidate 0.5, 0.5.
  • Clarify that failed, invalid, duplicate, and unassessed evidence is excluded from score statistics; incomplete denominators remain explicit.
  • Update the existing release-gate CI tutorial and external workflow guide. Trial stability remains descriptive and does not alter the gate or claim significance/reliability.

Validation

  • node scripts/audit-links.mjs: 0 broken navigation/content links; 29 pre-existing orphan pages.
  • npm run build: 1,190 pages built; Pagefind indexed 725 pages.
  • git diff --check passed.
  • Helper-backed check verified the equal-mean, paired-delta, incomplete-count, and REST flag-preservation examples.

Base: docs #904 (docs/TH-8579-case-release-gate).

Limitations: the option requires a compatible test deployment. This docs change does not establish hosted API, generated SDK, model, Observe, browser, or deployment qualification. No PR checks were reported at creation time.

Linear: TH-8586.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant