Repository navigation
Conversation
This was referenced Oct 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expose stored run comparisons at
POST /model-hub/experiment-runs/compare/. The endpoint returns per-case outcomes, assessment/completeness counts, missing identities and execution trace references for an explicit baseline, candidate and metric. It requires the same immutable dataset version, exact example/trial plan and selected evaluator binding snapshot through the existing comparison service.Authentication supplies the organization and workspace scope. Both runs must be visible through the existing dataset scope, including default-workspace legacy nulls; unknown body fields such as organization/workspace overrides are rejected. The read-only POST marker permits authenticated viewers without write permission, and an absent authenticated organization never falls back to a stale user organization.
This additive run endpoint handles immutable run/trial identities that the existing legacy dataset-column comparison routes do not express. The change adds typed request/response serializers, regenerates OpenAPI and frontend contracts, and registers one read-only, idempotent MCP tool (
compare_experiment_runs). It adds no models, migrations, warehouse or UI.Linear: https://linear.app/future-agi/issue/TH-8574
Stack: parent #3471 (
feat/TH-8572-run-comparison-evidence).SDLC: https://github.com/future-agi/internal-docs/pull/70
Validation
Real PostgreSQL, from the worktree's
futureagidirectory:Coverage includes response content and serializer validation, real API-key viewer authentication, unauthenticated denial, organization/workspace exclusion, default-workspace legacy nulls, missing-both cases, malformed thresholds, body scope rejection, size refusal, incompatible evaluator bindings, and checked-in schema shape. The parent service already verifies the actual 10,001-row bound; this API suite verifies its error mapping.
OpenAPI regenerated with complete EE source pinned at
ab169e6: exactly one added route and 11 definitions, with no existing route/definition removals or changes. Frontendcontracts:generateandcontracts:checkpassed: 1,434 operations, zero request/response/error schema gaps. MCP generation and--checkpassed for 93 tools. Environment-only EE and node_modules symlinks were excluded from this commit.New/changed serializer, view and test files pass Ruff lint/format; the URL module's six pre-existing lint findings are unchanged from the parent. All changed Python files pass format checks. Staged safety and diff checks passed. The normal hook lacks lint-staged in this isolated checkout, so explicit gates ran before
HUSKY=0commit.FI_SKIP_CH25_MIGRATION=1andFI_SKIP_CH25_SCHEMA_APPLY=1keep DB qualification PostgreSQL-only. API/browser deployment, live Observe and model-backed execution remain unverified.AI use: Codex wrote and reviewed the implementation, contracts and tests; no human approval is claimed.
E2E: exempt (backend-internal)
Rollout and rollback
The endpoint reads persisted evidence and makes no CI decision or task/evaluator invocation. Active runs are allowed; counts describe the evidence read. Public documentation must retain draft availability until rollout. Reverting this PR removes the route/tool without changing stored results.