Skip to content

feat(sdk): expose opt-in trial stability comparison - #33

Draft
nik13 wants to merge 2 commits into
feat/TH-8583-sdk-release-gatefrom
feat/TH-8587-sdk-trial-stability
Draft

nik13 wants to merge 2 commits into
feat/TH-8583-sdk-release-gatefrom
feat/TH-8587-sdk-trial-stability

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The existing typed experimentRuns.compare facade can compare persisted runs, but SDK consumers cannot request the descriptive repeated-trial projection added by core #3483. That leaves Python and TypeScript consumers without typed per-example spread and paired-delta analysis.

Change

  • Regenerate only the existing comparison POST contract from core #3483 (32ea09fa63d0ffa2e5178d1ede94f0f8b5796b7a).
  • Add optional include_trial_stability request typing and optional trial_stability response typing with nested TrialAggregate/TrialDeltaAggregate models.
  • Keep the existing facade, route, transport, six lifecycle response-style hooks, release-gate behavior, and single-attempt semantics unchanged.
  • Preserve Python generated-model positional constructor compatibility by appending new properties after existing schema properties in the scoped SDK overlay.
  • Keep omitted Python model fields omitted on the wire; explicit False is serialized, while the API schema still documents the server default as false.
  • Add Python/TypeScript wire and parser coverage for omitted, false, and true opt-in, nullable versus zero spread, missing trials, empty stability output, and release-gate coexistence.
  • Document the generated source scope, usage, and evidence boundary.

Validation

  • Current head dde3ef590728c6b03332465c1218116cfe37f610: all five feature-base CI jobs passed in run 38416429101; GitHub reports CLEAN/MERGEABLE. Draft review and release gates remain open.
  • Python SDK: PYTHONPATH=python .venv/bin/python -m pytest -q python/tests — 114 passed, 5 skipped.
  • TypeScript SDK: pnpm --dir typescript/futureagi test --runInBand — 106 passed.
  • TypeScript: pnpm --dir typescript/futureagi typecheck — passed before the final Python-only follow-up.
  • TypeScript: pnpm --dir typescript/futureagi build — passed before the final Python-only follow-up.
  • Ruff focused generated/client/test checks and git diff --check — passed. Full-tree Ruff reports nine pre-existing unrelated findings.
  • Contract scope assertion: only /model-hub/experiment-runs/compare/ and five trial/comparison schemas changed; all other generated paths remained byte-equivalent and all six lifecycle response-style hooks remain present.
  • Controlled authenticated local HTTP/PostgreSQL receipt (owned disposable service): Python and TypeScript completed snapshot, run/result ingestion, omitted/false parity, opt-in null/zero/stddev/paired-delta parsing, and failed full-plan release gate. Python and TypeScript returned identical comparison JSON; scores were synthetic protocol fixtures.

Evidence boundaries

The nested response fixture and local HTTP scores are synthetic contract evidence, not a hosted or model-backed evaluation. This PR does not claim model/provider execution, live Observe ingestion, UI/browser qualification, deployment, package publication, or backend calculation qualification. Core behavior is provided by future-agi PR #3483.

Linear: TH-8587.

Stack: SDK #25 compatibility -> SDK #30 lifecycle -> SDK #31 release gate -> this PR. Base: feat/TH-8583-sdk-release-gate.

@nik13
nik13 marked this pull request as draft October 13, 2026 14:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant