|
1 | | -# 170 — Backend union + descriptor table (wp2 implementation cycle) |
2 | | - |
3 | | -Depends on: 160. Implements #2188 vision rules for xai/gemini; resolves audit |
4 | | -Blockers A and B. |
5 | | - |
6 | | -## Design decision: VISION_BACKENDS descriptor table |
7 | | - |
8 | | -Mirror WEB_SEARCH_BACKENDS as a SIBLING table (audit Q2) in a new |
9 | | -`src/vision/backends.ts`: |
10 | | - |
11 | | -```ts |
12 | | -interface VisionBackendDescriptor { |
13 | | - backend: VisionSidecarBackend; // "openai" | "anthropic" | "xai" | "gemini" |
14 | | - isActive(auth: SidecarAuthState, config: OcxConfig): boolean; |
15 | | - candidateMatch(candidate: VisionCandidateModel, auth: SidecarAuthState): boolean; |
16 | | - baseline?: string; // only openai/anthropic carry one |
17 | | - rank: number; // stable option ordering |
18 | | -} |
19 | | -``` |
20 | | - |
21 | | -- openai: isActive = auth.isCodexAuth-shaped predicate already used by |
22 | | - enabledVisionBackends (listOpenAiForwardSidecarCandidates > 0); baseline |
23 | | - gpt-5.6-luna; rank 0. |
24 | | -- anthropic: isActive = anthropicSidecar resolved; candidateMatch = |
25 | | - provider === auth.anthropicProviderName; baseline claude-haiku-4-5; rank 1. |
26 | | -- xai: isActive = same predicate as WEB_SEARCH_BACKENDS xai (enabled oauth |
27 | | - "xai" provider + active account !needsReauth); candidateMatch = |
28 | | - candidate.provider === "xai"; NO baseline; rank 2. |
29 | | -- gemini: isActive = Antigravity OAuth + projectId (same as web-search); |
30 | | - candidateMatch = provider === "google-antigravity"; NO baseline; rank 3. |
31 | | -- Empty-auth fallback (no side active): ["openai","anthropic"] only — |
32 | | - xai/gemini are never offered unauthenticated (audit Q1 gap). |
33 | | - |
34 | | -## Blocker A resolution — baselines |
35 | | - |
36 | | -`BASELINE_VISION_MODELS` stays a record of exactly the two universal sides: |
37 | | -type becomes `Partial<Record<VisionSidecarBackend, string>>` sourced from |
38 | | -descriptor.baseline. visionEligibleModelOptions iterates descriptors (not the |
39 | | -hardcoded 2-tuple), injecting a baseline row only when descriptor.baseline is |
40 | | -set and that side isActive. |
41 | | - |
42 | | -## Blocker B resolution — provably-blind gate |
43 | | - |
44 | | -`visionDescriberIsProvablyBlind` widens its vendor probe from |
45 | | -{openai, anthropic} to {openai, anthropic, xai, google} via |
46 | | -resolveMetadataProvider. Collision scan (160) proved no bare id is shared |
47 | | -across the four tables, so "any positive text-only verdict wins" stays sound. |
48 | | -Regression test: PUT model=grok-4 (bare, text-only in xai table, absent from |
49 | | -candidates) must 400; PUT model=grok-4.3 with xai auth must 200. |
50 | | - |
51 | | -## Files touched (wp2) |
52 | | - |
53 | | -- src/vision/backends.ts (new): descriptor table + sidecarVisionBackends() |
54 | | - helper returning active descriptors. |
55 | | -- src/vision/eligibility.ts: union widens; BASELINE_VISION_MODELS type; |
56 | | - visionBackendForCandidate delegates to descriptor candidateMatch (keeps |
57 | | - signature; gains optional auth arg via new overload consumed by options |
58 | | - path); visionEligibleModelOptions iterates descriptors ranked. |
59 | | -- src/types.ts: OcxVisionSidecarConfig.backend union widens. |
60 | | -- src/server/management/vision-sidecar-options.ts: enabledVisionBackends |
61 | | - delegates to descriptors; visionDescriberIsProvablyBlind four-family probe. |
62 | | -- src/server/management/config-routes.ts: PUT gate literals :594-596, hint |
63 | | - fall-through :623, claude-code override :738-740 — all widen to the union. |
64 | | -- tests: sidecar-settings-vision-filter.test.ts, vision-eligibility.test.ts, |
65 | | - sidecar-settings-vision-controls.test.ts extended; new fixture with xai + |
66 | | - antigravity oauth accounts (pattern from web-search-backend-union.test.ts). |
67 | | - |
68 | | -## Not in wp2 |
69 | | - |
70 | | -Executors (180) — planVisionSidecar keeps its current arms; a persisted |
71 | | -xai/gemini backend without an executor cannot be SELECTED at runtime yet, so |
72 | | -wp2 lands options+gate first with resolveVisionBackend still collapsing |
73 | | -unknown-to-executor backends to the legacy default order. planVisionSidecar |
74 | | -gains its arms in wp3 in the same push train (dev gets both before release). |
| 1 | +# 170 — Backend union: "routed" describer (wp2, REVISED) |
| 2 | + |
| 3 | +Depends on: 160. REVISION 2026-08-22: user directive — vision does not need |
| 4 | +per-backend executors. Any picker-visible model with image input can describe; |
| 5 | +the proxy's own router already speaks every provider wire. The earlier |
| 6 | +xai/gemini backend literals were implemented but never released; this revision |
| 7 | +replaces them before any push. |
| 8 | + |
| 9 | +## Design |
| 10 | + |
| 11 | +- `VisionSidecarBackend = "openai" | "anthropic" | "routed"`. |
| 12 | +- "openai"/"anthropic" arms unchanged (forward Responses / OAuth Messages) — |
| 13 | + they carry auth semantics loopback routing cannot replicate (forwarded |
| 14 | + headers, OAuth beta fences), and their defaults must not drift. |
| 15 | +- "routed": the describer is ANY routed model, dispatched through the proxy's |
| 16 | + own /v1/chat/completions on loopback (pattern: src/claude/gateway-cache.ts |
| 17 | + self-fetch). One executor, every provider. |
| 18 | + |
| 19 | +## Filter (#2188 rules, unchanged shape) |
| 20 | + |
| 21 | +1. Picker-visible ∪ auth slots (pickerVisibleSidecarCandidates). |
| 22 | +2. − provably text-only (modelAcceptsImageInput === false drops the row). |
| 23 | + |
| 24 | +visionBackendForCandidate: native/openai → openai; resolved-OAuth anthropic |
| 25 | +row → anthropic; ANY OTHER provider row → "routed". Routed option values are |
| 26 | +NAMESPACED ("provider/model") so routeModel is unambiguous; legacy sides keep |
| 27 | +bare ids (GUI/current-value compatibility). |
| 28 | + |
| 29 | +## Gate |
| 30 | + |
| 31 | +visionDescriberIsProvablyBlind keeps the four-family probe widening AND |
| 32 | +learns namespaced ids: split on first "/", probe that provider's config row + |
| 33 | +metadata family. Bare ids keep the existing all-family probe. |
| 34 | + |
| 35 | +## Runtime |
| 36 | + |
| 37 | +- planVisionSidecar routed arm requires: cfg.backend === "routed", explicit |
| 38 | + cfg.model, and plan-time modelAcceptsImageInput !== false for the target. |
| 39 | +- Recursion safety: the loopback request re-enters the vision planner only if |
| 40 | + the routed model is provably text-only; the plan-time check excludes exactly |
| 41 | + that set, so describe recursion is structurally impossible. |
| 42 | +- resolveVisionBackend: explicit honored; unset default order UNCHANGED. |
| 43 | + |
| 44 | +## Files (wp2 scope, revised) |
| 45 | + |
| 46 | +- src/vision/eligibility.ts: union, visionBackendForCandidate routed arm, |
| 47 | + namespaced option values, BASELINE narrow-key record (kept from r1). |
| 48 | +- src/vision/backends.ts (r1 descriptor table): SIMPLIFIED — descriptors for |
| 49 | + openai/anthropic/routed; xai/gemini entries dropped. |
| 50 | +- vision-sidecar-options.ts: enabledVisionBackends offers "routed" whenever |
| 51 | + any routed row exists; gate learns namespaced ids. |
| 52 | +- config-routes.ts + agent-settings-routes.ts: literal sets accept "routed" |
| 53 | + (xai/gemini literals removed). |
| 54 | +- types: backend unions. |
| 55 | +- tests: vision-backend-union.test.ts rewritten for routed. |
| 56 | + |
| 57 | + |
| 58 | +## Audit round 2 amendments (2026-08-22, sol-medium) |
| 59 | + |
| 60 | +- **Recursion fence is a MECHANISM, not a predicate claim.** The loopback |
| 61 | + describe request carries a terminal marker header |
| 62 | + `x-opencodex-vision-describe: 1`. The Responses plan site treats a marked |
| 63 | + request as terminal: images are STRIPPED, never described (depth cap 1). |
| 64 | + This holds under predicate drift (modelInputModalities is invisible to a |
| 65 | + row-less plan-time target) and combo re-resolution (router.ts:625-631 can |
| 66 | + land a different sibling). Belt-and-braces: the routed arm also requires |
| 67 | + `!isModelTextOnly(resolvedRoute.provider, resolvedRoute.modelId)` at plan |
| 68 | + time — the exact re-entry predicate on the resolved route. |
| 69 | +- **PUT-gate coherence:** a namespaced model with backend openai/anthropic is |
| 70 | + REJECTED (forward executor POSTs the string verbatim — web-search F1 |
| 71 | + selector/slug failure); backend "routed" REQUIRES a namespaced id. |
| 72 | +- **GUI inference:** `value.includes("/") → "routed"` in |
| 73 | + visionSidecarBackendForModel's fallback; persisted backend keeps traveling |
| 74 | + as currentBackend. |
| 75 | +- Known limitation (recorded, not fixed here): a non-loopback-only bindHost |
| 76 | + where 127.0.0.1 does not answer — same latent limitation gateway-cache has. |
| 77 | +- handleNativeChatCompletions fast path has no vision handling; the marked |
| 78 | + describe request must not regress it (marker check lives at the Responses |
| 79 | + plan site the bridge replays into). |
75 | 80 |
|
0 commit comments