Skip to content

docs: add SDK trial stability comparison examples - #908

Draft
nik13 wants to merge 2 commits into
docs/TH-8586-trial-stabilityfrom
docs/TH-8587-sdk-trial-stability
Draft

nik13 wants to merge 2 commits into
docs/TH-8586-trial-stabilityfrom
docs/TH-8587-sdk-trial-stability

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The comparison reference and CI tutorial still said generated SDK support for trial stability was unavailable after SDK #33 implemented the typed flag and response.

Change

  • Add Python, TypeScript, and cURL comparison snippets using FI_API_KEY, FI_SECRET_KEY, and FI_BASE_URL.
  • Name the generated Python RunComparisonRequest and existing experiment_runs.compare / experimentRuns.compare facades.
  • Pin the draft backend/SDK prerequisites and preserve the distinction between test builds and published package/hosted availability.
  • Add short saved-report examples that keep null statistics distinct from observed zero spread.
  • Update the existing external-workflow and CI guide links; no new page, route, feature, or UI is added.

Metadata selector follow-up

The existing comparison reference and CI guide now document exact metadata_equals (20-key bound) and split_name selectors, selection counts, whole-identity semantics, and null/zero JSON preservation.

Validation

  • npm run build: passed; Pagefind indexed 725 pages.
  • npm run audit-links: zero broken navigation/content links; 29 existing orphan pages.
  • Exact Python and TypeScript reference snippets were extracted from MDX and run using SDK prompt template sdk #33 against a temporary local fixture server. The server asserted both opt-in bodies and auth header injection, then returned the retained authenticated backend response fixture. Both printed unavailable statistics and zero spread correctly.
  • Exact Python and TypeScript saved-report snippets were extracted and executed against the retained JSON response.
  • git diff --check passed; the temporary dependency symlink was removed before staging.

The fixture-server check validates these exact snippets and generated SDK decoding. Earlier authenticated HTTP/PostgreSQL probes validate the SDK backend data path using deterministic synthetic scores. Neither establishes model/provider execution, live Observe ingestion, browser/UI qualification, hosted rollout, or package publication.

Stack: docs #904 -> docs #907 -> this PR. SDK dependency: futureagi-sdk #33, commit dde3ef590728c6b03332465c1218116cfe37f610. Backend contract: core #3483, commit 32ea09fa63d0ffa2e5178d1ede94f0f8b5796b7a.

Linear: TH-8587.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant