Skip to content

docs: add dataset snapshot and external run workflow - #903

Draft
nik13 wants to merge 2 commits into
docs/TH-8574-compare-runsfrom
docs/TH-8565-external-run-lifecycle
Draft

nik13 wants to merge 2 commits into
docs/TH-8574-compare-runsfrom
docs/TH-8565-external-run-lifecycle

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Adds draft API references and one runnable lifecycle tutorial on top of docs #902.

Scope:

  • document immutable dataset-version creation, including explicit case identity, provenance, bounds, permissions, and deliberate non-idempotent retry behavior;
  • document external run creation and bounded result ingestion, including pinned evaluator versions, exact plans, receipt replay, atomic batches, and caller-owned execution boundaries;
  • add a Python standard-library workflow that creates a snapshot, runs a deterministic retrieval/tool function twice, uploads results, exposes an incomplete comparison, and reports one improvement plus one regression despite equal means;
  • update API and dataset guide navigation.

Validation:

  • node scripts/audit-links.mjs: 726 pages, zero broken navigation/content links; 29 existing orphan pages;
  • npm run build: 1,187 pages built in 20.33 seconds;
  • python3 -m py_compile public/cookbook/external-run-comparison.py, Ruff, and git diff --check passed;
  • exact tutorial ran through local HTTP with real API-key authentication and disposable PostgreSQL against core lifecycle head 4b30a10796d85bad9390831a87236ffa1ad1eb84: snapshot 201, run/replay 200, incomplete candidate missing one, final means 0.5/0.5 with improved one/regressed one, terminal replay unchanged;
  • separate authenticated conflict probe returned 409 for a changed terminal receipt;
  • no hosted, model-backed, Observe, browser, SDK, CI release, or deployment qualification is claimed.

The API/SDK endpoints remain draft until their dependent core PRs are reviewed and released. The tutorial uses synthetic callbacks and null trace IDs; it does not imply remote task execution or trace ingestion.

@nik13
nik13 marked this pull request as draft October 13, 2026 08:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant