Skip to content

fix(evaluations): support connected APIs and isolated provider judges - #115

Open
ScriptedAlchemy wants to merge 14 commits into
mainfrom
t3/any-provider-model-evaluations
Open

fix(evaluations): support connected APIs and isolated provider judges#115
ScriptedAlchemy wants to merge 14 commits into
mainfrom
t3/any-provider-model-evaluations

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Why

The evaluation judge picker limits connected models to Claude and Pi. Users need to choose a judge independently of the provider that did the work.

What changed

  • Route configured model APIs through the evaluation service and list them in web, desktop, and mobile settings.
  • Add isolated Codex, Cursor, and local OpenCode judges. Preserve the selected model and reject tool activity or unsupported configuration.
  • Include the verdict schema in the shared prompt so API judges receive the required output structure.

Scope

  • Claude and Pi remain supported. Generic ACP runtimes without an isolation contract remain unavailable as judges.
  • External OpenCode servers and keyring-only Codex authentication remain unsupported. Models that require tools can reject evaluation calls.
  • Provider failures remain failed evaluations and never become poor task scores.

Blast Radius

This changes judge selection and provider execution, including isolated credential access and custom provider routing. Ordinary text generation retains its existing behavior.

Additional audit fixes

  • Match isolated OpenCode judge configuration to the selected OpenCode generation. V1 keeps its native global → explicit file → config-directory → inline precedence; V2 replaces the global directory when configured and lets the explicit file override it. Discovery also respects each generation's supported filenames.
  • Ignore empty config-file, config-directory, and inline-config environment overrides, matching the supported OpenCode servers. Empty values no longer trigger spurious isolation errors or load a working-directory config.
  • Make the command-palette keyboard test explicitly seed and restore its expected remembered project, and assert that Acme is the initially highlighted row. Its ArrowDown, submenu, and focus assertions remain intact. The previous CI failure's exact runtime state was not captured, so its original cause is not proven.
  • Reproduced two config-discovery/precedence failures and six empty-override failures before fixing them. The complete affected OpenCode suites now pass: 27 tests across V1 and V2.
  • Required changed-file validation passed the native addon, formatting, lint, both knip stages, and 29 unit tests across 3 files. The cached typecheck could not start because this workspace lacks the required task-cache sandbox. A direct no-cache provider-opencode typecheck passes. The local browser check stopped before assertions because the required Playwright executable is unavailable.
  • Published commit: e4eebb9. Its Git tree exactly matches the tested local files.
  • Fresh checks for this commit: Fork CI and Agent Context Budget. Both were queued when this section was written; this is not a claim that the full suite has passed.

Initial audit follow-up

  • Merged current main, including the completed removal of new-thread suggestions.
  • Fixed isolated Codex judges rejecting valid unquoted routing overrides and ignoring whitespace around routing keys. Malformed routing arguments still fail explicitly.
  • Made mobile API model discovery independent per API. Fast providers can populate immediately; failed requests remain errors, and the new refresh action bypasses the server model cache.
  • Added regressions that fail against the original behavior. The full affected test selection now passes from the owning packages: 144 tests across 9 server/provider files, plus 2 mobile discovery tests (146 tests / 10 files total). This includes all 36 Codex text-generation tests and the measured context-budget checks. Changed-file formatting, lint, and React compiler checks pass (4 changed files / 9 assertions).
  • The context-budget GitHub workflow passes on the published commit.
  • The required changed-file checker was invoked. This workspace cannot start its task-cache sandbox, and some broader processes exited without diagnostics; a direct typecheck attempt was killed with exit 137 without TypeScript errors. Those are validation limits, not a full-suite pass.
  • Published commit: 5dc36b6. Its Git tree exactly matches the audited local files.
  • Initial workflows: Fork CI and Agent Context Budget.

Original implementation verification

  • vp run check:changed --concurrency 2 passed, including 140 focused tests, affected-package typechecks, lint, formatting, and React compiler checks.
  • The real web scenario selected an API judge, evaluated a Replay task, displayed its verdict, preserved the selection, and restored the default. The same scenario failed on the base with a disabled judge picker and passed on the head.
  • Real API and OpenCode judge calls decoded successfully. Installed Codex and OpenCode request captures showed the selected model and zero tools. The Morph Kimi rubric run passed 9 of 12 samples and 3 of 4 case majorities. The failed-test case remained inconsistent.

Claude live inference was blocked by expired authentication. Codex live inference was blocked by endpoint connectivity. Cursor has deterministic SDK coverage without a live inference pass. Native iOS verification remains incomplete after two Expo module-scan build timeouts.

Browser proof and before/after captures. Download the self-contained HTML file to view its embedded PNGs.

Models and harnesses used were Codex GPT-6 Astra and GPT-6.1 Sol, Claude Agent Opus 5.5, and OpenCode Morph Kimi K3.1 through T3 Code.

ScriptedAlchemy and others added 9 commits November 12, 2026 14:01
@ScriptedAlchemy
test(server): reproduce model API evaluation routing failure
b71c886
@ScriptedAlchemy @claude
fix(server,web,mobile): model APIs can judge model evaluations
928c727 
A judge selection naming a model API failed with "No provider instance
registered", and both settings screens hid model APIs from the judge
picker.

generateModelEvaluation now runs the selected model API directly as a
blind evaluator. It has no tools or workspace, so it never falls back to
another judge. Agent providers keep the exact availability checks.
Web lists model APIs through the existing text generation targets and
treats an enabled API as an available judge per environment. Mobile
lists each enabled API's models through the same guarded command.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ScriptedAlchemy
feat(cursor): support isolated model evaluation judges
3e955db
@ScriptedAlchemy
fix(server): run exact Codex evaluation judges in isolation
a6b126f
@ScriptedAlchemy
fix(opencode): isolate blind evaluation judges
1a8629f
@ScriptedAlchemy
fix(evaluations): preserve judge credentials and validate complete ve…
829c4be 
…rdicts
@ScriptedAlchemy
fix(evaluations): preserve provider routing and reject external judges
bdddbfb
@ScriptedAlchemy
fix(server): narrow selected Codex profile before extracting routing
895471a
@ScriptedAlchemy
test(evaluations): assert connected API appears in judge picker
b905c49
@changeset-bot

changeset-bot Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 906712c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
fix(evaluations): audit provider routing and refresh API model discovery
5dc36b6 
Merge current main, preserve Codex routing overrides in isolated judges, and keep API model discovery failures retryable on mobile. Add focused routing and independent-discovery regressions.
@ScriptedAlchemy
fix(evaluations): preserve native OpenCode configuration
e4eebb9 
Match v1 and v2 configuration precedence and filenames, and ignore empty override flags. Keep keyboard palette coverage deterministic by seeding its remembered-project prerequisite without removing navigation assertions.
@ScriptedAlchemy ScriptedAlchemy mentioned this pull request Nov 13, 2026
Merged
ScriptedAlchemy and others added 2 commits November 13, 2026 15:03
@ScriptedAlchemy
Merge remote-tracking branch 'origin/main' into t3/any-provider-model…
036bac7 
…-evaluations

# Conflicts:
#	apps/web/src/components/CommandPalette.browser.test.tsx
@ScriptedAlchemy @claude
test(web): drop the palette test workaround that main's leak fix repl…
9565de6 
…aces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ScriptedAlchemy ScriptedAlchemy mentioned this pull request Nov 14, 2026
Merged

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ScriptedAlchemy