Skip to content

feat(experiments): expose scoped run comparison API [agent] - #3472

Draft
nik13 wants to merge 1 commit into
feat/TH-8572-run-comparison-evidencefrom
feat/TH-8574-comparison-api
Draft

nik13 wants to merge 1 commit into
feat/TH-8572-run-comparison-evidencefrom
feat/TH-8574-comparison-api

Conversation

@nik13

@nik13 nik13 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Expose stored run comparisons at POST /model-hub/experiment-runs/compare/. The endpoint returns per-case outcomes, assessment/completeness counts, missing identities and execution trace references for an explicit baseline, candidate and metric. It requires the same immutable dataset version, exact example/trial plan and selected evaluator binding snapshot through the existing comparison service.

Authentication supplies the organization and workspace scope. Both runs must be visible through the existing dataset scope, including default-workspace legacy nulls; unknown body fields such as organization/workspace overrides are rejected. The read-only POST marker permits authenticated viewers without write permission, and an absent authenticated organization never falls back to a stale user organization.

This additive run endpoint handles immutable run/trial identities that the existing legacy dataset-column comparison routes do not express. The change adds typed request/response serializers, regenerates OpenAPI and frontend contracts, and registers one read-only, idempotent MCP tool (compare_experiment_runs). It adds no models, migrations, warehouse or UI.

Linear: https://linear.app/future-agi/issue/TH-8574
Stack: parent #3471 (feat/TH-8572-run-comparison-evidence).
SDLC: https://github.com/future-agi/internal-docs/pull/70

Validation

Real PostgreSQL, from the worktree's futureagi directory:

pytest model_hub/tests/test_run_comparison_api.py --reuse-db -q
21 passed in 4.23s

Coverage includes response content and serializer validation, real API-key viewer authentication, unauthenticated denial, organization/workspace exclusion, default-workspace legacy nulls, missing-both cases, malformed thresholds, body scope rejection, size refusal, incompatible evaluator bindings, and checked-in schema shape. The parent service already verifies the actual 10,001-row bound; this API suite verifies its error mapping.

pytest mcp_server/tests/test_tool_generation.py mcp_server/tests/test_generated_registry.py mcp_server/tests/test_workflow_catalog.py
50 passed

OpenAPI regenerated with complete EE source pinned at ab169e6: exactly one added route and 11 definitions, with no existing route/definition removals or changes. Frontend contracts:generate and contracts:check passed: 1,434 operations, zero request/response/error schema gaps. MCP generation and --check passed for 93 tools. Environment-only EE and node_modules symlinks were excluded from this commit.

New/changed serializer, view and test files pass Ruff lint/format; the URL module's six pre-existing lint findings are unchanged from the parent. All changed Python files pass format checks. Staged safety and diff checks passed. The normal hook lacks lint-staged in this isolated checkout, so explicit gates ran before HUSKY=0 commit.

FI_SKIP_CH25_MIGRATION=1 and FI_SKIP_CH25_SCHEMA_APPLY=1 keep DB qualification PostgreSQL-only. API/browser deployment, live Observe and model-backed execution remain unverified.

AI use: Codex wrote and reviewed the implementation, contracts and tests; no human approval is claimed.
E2E: exempt (backend-internal)

Rollout and rollback

The endpoint reads persisted evidence and makes no CI decision or task/evaluator invocation. Active runs are allowed; counts describe the evidence read. Public documentation must retain draft availability until rollout. Reverting this PR removes the route/tool without changing stored results.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant