Skip to content

Commit 606dbcd

Browse files
committed
feat(vision): routed describer backend — options, gates, namespaced ids (lidge-jun#2188 roadmap 170)
Any picker-visible, non-provably-text-only provider row can now be offered as the vision describer through the new 'routed' backend. Adds the VISION_BACKENDS descriptor table, namespaced routed option values, the four-family + namespaced provably-blind probe, PUT coherence rules (namespaced <-> routed), and claude-code override parity. Docs 170/180 revised with three audit rounds folded in.
1 parent 3c2d96f commit 606dbcd

11 files changed

Lines changed: 562 additions & 182 deletions

File tree

Lines changed: 79 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,80 @@
1-
# 170 — Backend union + descriptor table (wp2 implementation cycle)
2-
3-
Depends on: 160. Implements #2188 vision rules for xai/gemini; resolves audit
4-
Blockers A and B.
5-
6-
## Design decision: VISION_BACKENDS descriptor table
7-
8-
Mirror WEB_SEARCH_BACKENDS as a SIBLING table (audit Q2) in a new
9-
`src/vision/backends.ts`:
10-
11-
```ts
12-
interface VisionBackendDescriptor {
13-
backend: VisionSidecarBackend; // "openai" | "anthropic" | "xai" | "gemini"
14-
isActive(auth: SidecarAuthState, config: OcxConfig): boolean;
15-
candidateMatch(candidate: VisionCandidateModel, auth: SidecarAuthState): boolean;
16-
baseline?: string; // only openai/anthropic carry one
17-
rank: number; // stable option ordering
18-
}
19-
```
20-
21-
- openai: isActive = auth.isCodexAuth-shaped predicate already used by
22-
enabledVisionBackends (listOpenAiForwardSidecarCandidates > 0); baseline
23-
gpt-5.6-luna; rank 0.
24-
- anthropic: isActive = anthropicSidecar resolved; candidateMatch =
25-
provider === auth.anthropicProviderName; baseline claude-haiku-4-5; rank 1.
26-
- xai: isActive = same predicate as WEB_SEARCH_BACKENDS xai (enabled oauth
27-
"xai" provider + active account !needsReauth); candidateMatch =
28-
candidate.provider === "xai"; NO baseline; rank 2.
29-
- gemini: isActive = Antigravity OAuth + projectId (same as web-search);
30-
candidateMatch = provider === "google-antigravity"; NO baseline; rank 3.
31-
- Empty-auth fallback (no side active): ["openai","anthropic"] only —
32-
xai/gemini are never offered unauthenticated (audit Q1 gap).
33-
34-
## Blocker A resolution — baselines
35-
36-
`BASELINE_VISION_MODELS` stays a record of exactly the two universal sides:
37-
type becomes `Partial<Record<VisionSidecarBackend, string>>` sourced from
38-
descriptor.baseline. visionEligibleModelOptions iterates descriptors (not the
39-
hardcoded 2-tuple), injecting a baseline row only when descriptor.baseline is
40-
set and that side isActive.
41-
42-
## Blocker B resolution — provably-blind gate
43-
44-
`visionDescriberIsProvablyBlind` widens its vendor probe from
45-
{openai, anthropic} to {openai, anthropic, xai, google} via
46-
resolveMetadataProvider. Collision scan (160) proved no bare id is shared
47-
across the four tables, so "any positive text-only verdict wins" stays sound.
48-
Regression test: PUT model=grok-4 (bare, text-only in xai table, absent from
49-
candidates) must 400; PUT model=grok-4.3 with xai auth must 200.
50-
51-
## Files touched (wp2)
52-
53-
- src/vision/backends.ts (new): descriptor table + sidecarVisionBackends()
54-
helper returning active descriptors.
55-
- src/vision/eligibility.ts: union widens; BASELINE_VISION_MODELS type;
56-
visionBackendForCandidate delegates to descriptor candidateMatch (keeps
57-
signature; gains optional auth arg via new overload consumed by options
58-
path); visionEligibleModelOptions iterates descriptors ranked.
59-
- src/types.ts: OcxVisionSidecarConfig.backend union widens.
60-
- src/server/management/vision-sidecar-options.ts: enabledVisionBackends
61-
delegates to descriptors; visionDescriberIsProvablyBlind four-family probe.
62-
- src/server/management/config-routes.ts: PUT gate literals :594-596, hint
63-
fall-through :623, claude-code override :738-740 — all widen to the union.
64-
- tests: sidecar-settings-vision-filter.test.ts, vision-eligibility.test.ts,
65-
sidecar-settings-vision-controls.test.ts extended; new fixture with xai +
66-
antigravity oauth accounts (pattern from web-search-backend-union.test.ts).
67-
68-
## Not in wp2
69-
70-
Executors (180) — planVisionSidecar keeps its current arms; a persisted
71-
xai/gemini backend without an executor cannot be SELECTED at runtime yet, so
72-
wp2 lands options+gate first with resolveVisionBackend still collapsing
73-
unknown-to-executor backends to the legacy default order. planVisionSidecar
74-
gains its arms in wp3 in the same push train (dev gets both before release).
1+
# 170 — Backend union: "routed" describer (wp2, REVISED)
2+
3+
Depends on: 160. REVISION 2026-08-22: user directive — vision does not need
4+
per-backend executors. Any picker-visible model with image input can describe;
5+
the proxy's own router already speaks every provider wire. The earlier
6+
xai/gemini backend literals were implemented but never released; this revision
7+
replaces them before any push.
8+
9+
## Design
10+
11+
- `VisionSidecarBackend = "openai" | "anthropic" | "routed"`.
12+
- "openai"/"anthropic" arms unchanged (forward Responses / OAuth Messages) —
13+
they carry auth semantics loopback routing cannot replicate (forwarded
14+
headers, OAuth beta fences), and their defaults must not drift.
15+
- "routed": the describer is ANY routed model, dispatched through the proxy's
16+
own /v1/chat/completions on loopback (pattern: src/claude/gateway-cache.ts
17+
self-fetch). One executor, every provider.
18+
19+
## Filter (#2188 rules, unchanged shape)
20+
21+
1. Picker-visible ∪ auth slots (pickerVisibleSidecarCandidates).
22+
2. − provably text-only (modelAcceptsImageInput === false drops the row).
23+
24+
visionBackendForCandidate: native/openai → openai; resolved-OAuth anthropic
25+
row → anthropic; ANY OTHER provider row → "routed". Routed option values are
26+
NAMESPACED ("provider/model") so routeModel is unambiguous; legacy sides keep
27+
bare ids (GUI/current-value compatibility).
28+
29+
## Gate
30+
31+
visionDescriberIsProvablyBlind keeps the four-family probe widening AND
32+
learns namespaced ids: split on first "/", probe that provider's config row +
33+
metadata family. Bare ids keep the existing all-family probe.
34+
35+
## Runtime
36+
37+
- planVisionSidecar routed arm requires: cfg.backend === "routed", explicit
38+
cfg.model, and plan-time modelAcceptsImageInput !== false for the target.
39+
- Recursion safety: the loopback request re-enters the vision planner only if
40+
the routed model is provably text-only; the plan-time check excludes exactly
41+
that set, so describe recursion is structurally impossible.
42+
- resolveVisionBackend: explicit honored; unset default order UNCHANGED.
43+
44+
## Files (wp2 scope, revised)
45+
46+
- src/vision/eligibility.ts: union, visionBackendForCandidate routed arm,
47+
namespaced option values, BASELINE narrow-key record (kept from r1).
48+
- src/vision/backends.ts (r1 descriptor table): SIMPLIFIED — descriptors for
49+
openai/anthropic/routed; xai/gemini entries dropped.
50+
- vision-sidecar-options.ts: enabledVisionBackends offers "routed" whenever
51+
any routed row exists; gate learns namespaced ids.
52+
- config-routes.ts + agent-settings-routes.ts: literal sets accept "routed"
53+
(xai/gemini literals removed).
54+
- types: backend unions.
55+
- tests: vision-backend-union.test.ts rewritten for routed.
56+
57+
58+
## Audit round 2 amendments (2026-08-22, sol-medium)
59+
60+
- **Recursion fence is a MECHANISM, not a predicate claim.** The loopback
61+
describe request carries a terminal marker header
62+
`x-opencodex-vision-describe: 1`. The Responses plan site treats a marked
63+
request as terminal: images are STRIPPED, never described (depth cap 1).
64+
This holds under predicate drift (modelInputModalities is invisible to a
65+
row-less plan-time target) and combo re-resolution (router.ts:625-631 can
66+
land a different sibling). Belt-and-braces: the routed arm also requires
67+
`!isModelTextOnly(resolvedRoute.provider, resolvedRoute.modelId)` at plan
68+
time — the exact re-entry predicate on the resolved route.
69+
- **PUT-gate coherence:** a namespaced model with backend openai/anthropic is
70+
REJECTED (forward executor POSTs the string verbatim — web-search F1
71+
selector/slug failure); backend "routed" REQUIRES a namespaced id.
72+
- **GUI inference:** `value.includes("/") → "routed"` in
73+
visionSidecarBackendForModel's fallback; persisted backend keeps traveling
74+
as currentBackend.
75+
- Known limitation (recorded, not fixed here): a non-loopback-only bindHost
76+
where 127.0.0.1 does not answer — same latent limitation gateway-cache has.
77+
- handleNativeChatCompletions fast path has no vision handling; the marked
78+
describe request must not regress it (marker check lives at the Responses
79+
plan site the bridge replays into).
7580

Lines changed: 72 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,73 @@
1-
# 180 — Describe executors + runtime dispatch (wp3 implementation cycle)
2-
3-
Depends on: 170.
4-
5-
## src/vision/xai-describe.ts (new)
6-
7-
Mirror xai-executor.ts scaffolding: pinned https://api.x.ai origin,
8-
getValidAccessToken("xai"), redirect "manual", fetchWithResetRetry,
9-
signalWithTimeout(settings.timeoutMs) + cancelBodyOnAbort,
10-
sidecarEnter("vision"), redactSecretString on every error path. Body: 160
11-
wire. Non-stream preferred if probe allows (stream:false) — else reduce SSE
12-
output_text deltas. validateImageUrl reused from describe.ts (data: allowed
13-
mimes + 20MB cap, https passthrough). Reasoning: settings.reasoning passes
14-
through as reasoning.effort only for low|medium|high; xhigh/max clamp to high
15-
(xai ladder). Returns DescribeOutcome, never throws.
16-
17-
## src/vision/gemini-describe.ts (new)
18-
19-
Mirror gemini-executor.ts: registry-pinned base, ANTIGRAVITY_REQUEST_UA,
20-
getValidAccessTokenSnapshot (token + projectId), CCA envelope from 160 with
21-
inlineData part; resolveAntigravityEffortWireModel(settings.model,
22-
settings.reasoning, base) for wire model + thinkingLevel; readBoundedResponseBytes;
23-
data: URLs only (https rejected with explicit error, documented delta);
24-
sidecarEnter("vision"); redactSecretString. Returns DescribeOutcome.
25-
26-
## Runtime dispatch (src/vision/index.ts)
27-
28-
- VisionPlan gains backend arms: { backend: "xai", xaiSidecar: {providerName,
29-
provider} } and { backend: "gemini", geminiSidecar: {...} } following the
30-
anthropicSidecar shape.
31-
- planVisionSidecar: after resolving cfg.backend, arms for xai/gemini require
32-
their descriptor isActive (else fall through to legacy resolution — a
33-
persisted xai backend with expired auth degrades exactly like anthropic
34-
without OAuth: sidecar unavailable marker, never a crash).
35-
- resolveVisionBackend: explicit backend honored for all four; DEFAULT
36-
(unset) order unchanged: anthropic-if-auth else openai. No default drift.
37-
- executeDescription: two new arms calling the new executors.
38-
- descriptionIdentity: backend already part of the cache key; reasoning is
39-
keyed only for openai — include it for xai too (effort affects output);
40-
gemini keys thinkingLevel via model+reasoning inputs.
41-
- resolveEffectiveVisionModel: per-backend defaults — xai: grok-4.3,
42-
gemini: gemini-3.7-flash (both text,image in metadata); existing openai/
43-
anthropic defaults unchanged.
44-
45-
## Tests (wp3)
46-
47-
- vision-xai.test.ts, vision-gemini.test.ts (new): executor wire shape
48-
(mocked fetch), error taxonomy, redaction, data-URL validation, effort
49-
clamp/wire-model mapping.
50-
- vision-sidecar-e2e.test.ts: plan arms for xai/gemini with oauth fixtures;
51-
degraded no-auth path.
1+
# 180 — Routed describe executor + dispatch (wp3, REVISED)
2+
3+
Depends on: 170 (revised).
4+
5+
## src/vision/routed-describe.ts (new)
6+
7+
Loopback POST http://127.0.0.1:{config.port}/v1/chat/completions:
8+
9+
```json
10+
{ "model": "<namespaced routed id>", "stream": false,
11+
"messages": [
12+
{ "role": "system", "content": "<describe instruction>" },
13+
{ "role": "user", "content": [
14+
{ "type": "text", "text": "<context>" },
15+
{ "type": "image_url", "image_url": { "url": "<data:/https:>" } }
16+
]}]}
17+
```
18+
19+
- Auth: none on loopback binds (resolveApiAuth admits loopback without a
20+
token); when OPENCODEX_API_AUTH_TOKEN is set, send it as Authorization
21+
bearer (auth-cors.ts:399-400 accepts bearer on /v1/chat/completions).
22+
- signalWithTimeout(settings.timeoutMs) + cancelBodyOnAbort;
23+
sidecarEnter("vision"); redactSecretString on error paths; response text
24+
from choices[0].message.content; DESC clamp caller-side (existing).
25+
- validateImageUrl reused (data: mime allowlist + 20MB, https passthrough).
26+
- The chat inbound translates image_url → input_image and every adapter
27+
compiles its own wire (anthropic blocks, CCA inlineData, xai Responses),
28+
so provider coverage is the router's, not this file's.
29+
30+
## planVisionSidecar routed arm
31+
32+
VisionPlan gains { backend: "routed", routedModel: string }. Arm requires
33+
explicit model + plan-time modelAcceptsImageInput !== false (recursion
34+
fence). executeDescription routed arm calls describeImageRouted.
35+
36+
## Tests
37+
38+
vision-routed.test.ts: wire shape against a mock loopback server; recursion
39+
fence (text-only target never plans routed); timeout/error taxonomy;
40+
redaction. E2E: routed describer via a second mock provider.
41+
42+
43+
## Audit round 2 amendments (2026-08-22)
44+
45+
- **Admission ladder (blocker 2):** token =
46+
configuredApiAuthToken() || loadServiceTokenFromFile(env) || first
47+
config.apiKeys entry; sent as `x-opencodex-api-key` (never Authorization —
48+
gateway-cache.ts:77-86 rule); omitted entirely on loopback binds where
49+
isApiAuthRequired is false.
50+
- **Terminal marker:** executor sets `x-opencodex-vision-describe: 1`; the
51+
core.ts plan site checks it and strips images instead of planning vision.
52+
- Executor also passes stream:false and reads choices[0].message.content;
53+
non-2xx → {error} with redacted body slice.
54+
55+
56+
## Audit round 3 amendment (2026-08-22) — marker propagation
57+
58+
The chat→responses bridge rebuilds headers from the FORWARD_HEADERS allowlist
59+
(chat-completions.ts:198-203, openai-responses.ts:28-36), which would DROP
60+
`x-opencodex-vision-describe` before the plan site — on exactly the one path
61+
recursion lives. Therefore:
62+
63+
- The marker is detected AT THE CHAT SURFACE (raw req.headers before the
64+
bridge) and carried as an explicit option/flag into handleResponses
65+
(`visionDescribeTerminal: true`), not as a header the bridge must
66+
preserve. The Responses surface ALSO honors the raw header directly for
67+
native /v1/responses callers.
68+
- Regression test drives the FULL chat-surface path: marked POST to
69+
/v1/chat/completions with an image + text-only routed model → assert the
70+
plan site STRIPS (no describe dispatch, no recursion), while the same
71+
unmarked POST plans normally. A predicate-only test is insufficient and
72+
would stay green with the marker broken.
5273

‎src/server/management/agent-settings-routes.ts‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1073,13 +1073,14 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise
10731073
const section = body[field];
10741074
if (section === undefined || section === null) continue;
10751075
if (!isPlainObject(section)) return jsonResponse({ error: `${field} must be an object or null` }, 400);
1076-
// The widened union applies to the WEB-SEARCH override only (roadmap 060).
1077-
// Vision keeps its two-backend contract — accepting a wider id there would
1078-
// persist a backend the vision resolver reads as unset, silently activating
1079-
// a backend the operator never chose (review F1).
1076+
// Both overrides now speak their full unions (roadmap 060 web, 170
1077+
// vision revised). Vision's third arm is "routed" (loopback through the
1078+
// proxy's own router), never exa: exa is not an LLM, and accepting an
1079+
// unknown literal would persist a backend the vision resolver reads as
1080+
// unset (review F1's failure mode).
10801081
const allowedBackends = field === "webSearchSidecar"
10811082
? ["openai", "anthropic", "xai", "gemini", "exa"]
1082-
: ["openai", "anthropic"];
1083+
: ["openai", "anthropic", "routed"];
10831084
if (section.backend !== undefined && section.backend !== null
10841085
&& !allowedBackends.includes(section.backend as string)) {
10851086
return jsonResponse({ error: `${field}.backend must be ${allowedBackends.join(", ")}, or null` }, 400);
@@ -1094,8 +1095,18 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise
10941095
const requested = section.model;
10951096
const candidates = await visionCandidateRows(config);
10961097
const hint = section.backend === "anthropic" || section.backend === "openai"
1098+
|| section.backend === "routed"
10971099
? section.backend
10981100
: config.claudeCode?.visionSidecar?.backend;
1101+
// Same coherence rule as /api/sidecar-settings (roadmap 170 r2).
1102+
const effectiveBackend = hint ?? "openai";
1103+
const namespaced = requested.includes("/");
1104+
if (namespaced && effectiveBackend !== "routed") {
1105+
return jsonResponse({ error: `visionSidecar.model "${requested}" is provider-namespaced; it requires backend "routed"` }, 400);
1106+
}
1107+
if (!namespaced && effectiveBackend === "routed") {
1108+
return jsonResponse({ error: `visionSidecar.backend "routed" requires a provider-namespaced model ("provider/model"); got "${requested}"` }, 400);
1109+
}
10991110
if (visionDescriberIsProvablyBlind(config, requested, candidates, hint)) {
11001111
return jsonResponse(visionDescriberRejection("visionSidecar.model", requested, config, candidates), 400);
11011112
}

‎src/server/management/config-routes.ts‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -592,8 +592,9 @@ export async function handleConfigRoutes(ctx: ManagementContext): Promise<Respon
592592
return jsonResponse({ error: "webSearch.streamRoutedModelOutput must be a boolean" }, 400);
593593
}
594594
if (body.vision && body.vision.backend !== undefined
595-
&& body.vision.backend !== null && body.vision.backend !== "openai" && body.vision.backend !== "anthropic") {
596-
return jsonResponse({ error: "vision.backend must be openai, anthropic, or null" }, 400);
595+
&& body.vision.backend !== null && body.vision.backend !== "openai" && body.vision.backend !== "anthropic"
596+
&& body.vision.backend !== "routed") {
597+
return jsonResponse({ error: "vision.backend must be openai, anthropic, routed, or null" }, 400);
597598
}
598599
if (body.vision && body.vision.maxDescriptionsPerTurn !== undefined
599600
&& (typeof body.vision.maxDescriptionsPerTurn !== "number"
@@ -621,8 +622,20 @@ export async function handleConfigRoutes(ctx: ManagementContext): Promise<Respon
621622
const requested = body.vision.model;
622623
const candidates = await visionCandidateRows(config);
623624
const hint = body.vision.backend === "anthropic" || body.vision.backend === "openai"
625+
|| body.vision.backend === "routed"
624626
? body.vision.backend
625627
: config.visionSidecar?.backend;
628+
// Coherence (roadmap 170 r2): the forward/OAuth executors POST the model
629+
// string VERBATIM, so a namespaced id on those backends persists a wire
630+
// id they cannot run; and "routed" without a namespace cannot route.
631+
const effectiveBackend = hint ?? "openai";
632+
const namespaced = requested.includes("/");
633+
if (namespaced && effectiveBackend !== "routed") {
634+
return jsonResponse({ error: `vision.model "${requested}" is provider-namespaced; it requires vision.backend "routed"` }, 400);
635+
}
636+
if (!namespaced && effectiveBackend === "routed") {
637+
return jsonResponse({ error: `vision.backend "routed" requires a provider-namespaced vision.model ("provider/model"); got "${requested}"` }, 400);
638+
}
626639
if (visionDescriberIsProvablyBlind(config, requested, candidates, hint)) {
627640
return jsonResponse(visionDescriberRejection("vision.model", requested, config, candidates), 400);
628641
}
@@ -736,7 +749,8 @@ export async function handleConfigRoutes(ctx: ManagementContext): Promise<Respon
736749
else config.visionSidecar.model = body.vision.model;
737750
}
738751
if (body.vision.backend === null) delete config.visionSidecar.backend;
739-
else if (body.vision.backend === "openai" || body.vision.backend === "anthropic") {
752+
else if (body.vision.backend === "openai" || body.vision.backend === "anthropic"
753+
|| body.vision.backend === "routed") {
740754
config.visionSidecar.backend = body.vision.backend;
741755
}
742756
if (typeof body.vision.maxDescriptionsPerTurn === "number") {

0 commit comments

Comments
 (0)