|
| 1 | +# 160 — Vision external-backend research (xai Grok / Antigravity Gemini describers) |
| 2 | + |
| 3 | +Continuation of #2188. Web-search shipped four external backends (L6-L9, docs |
| 4 | +060-090); the vision sidecar still dispatches only openai-forward and |
| 5 | +anthropic-OAuth. GUI evidence: the vision dropdown lists only Codex/Claude |
| 6 | +rows while the web-search dropdown already lists Grok/Gemini. |
| 7 | + |
| 8 | +## Current vision dispatch inventory |
| 9 | + |
| 10 | +- Types: `OcxVisionSidecarConfig.backend?: "openai" | "anthropic"` (src/types.ts). |
| 11 | +- Union: `VisionSidecarBackend` (src/vision/eligibility.ts:30) — 2 arms. |
| 12 | +- Candidate mapping: `visionBackendForCandidate` (eligibility.ts:150-165) — |
| 13 | + native/openai → openai; anthropic only via the resolved OAuth provider name. |
| 14 | +- Options: `visionEligibleModelOptions` (eligibility.ts:201+) iterates |
| 15 | + `["openai","anthropic"] as const` and injects `BASELINE_VISION_MODELS`. |
| 16 | +- Enabled backends: `enabledVisionBackends` |
| 17 | + (src/server/management/vision-sidecar-options.ts:31-43); empty-auth fallback |
| 18 | + returns both universal sides. |
| 19 | +- Write gate: `visionDescriberIsProvablyBlind` (vision-sidecar-options.ts:94+) |
| 20 | + probes ONLY the openai/anthropic vendor tables. |
| 21 | +- PUT validation: config-routes.ts:594-596 rejects backends outside the two |
| 22 | + literals; hint fall-through at :623; claude-code override near :738-740. |
| 23 | +- Runtime plan: `planVisionSidecar` (src/vision/index.ts) — anthropic arm and |
| 24 | + openai-forward arm only. `resolveVisionBackend`: explicit > anthropic-if-auth |
| 25 | + > openai. |
| 26 | +- GUI: `SidecarBackend = "openai" | "anthropic"` (gui/src/pages/ |
| 27 | + dashboard-shared.ts:62, claude-manual-env.ts:8). NOTE: this type is shared |
| 28 | + with WebSearchModelOption and is ALREADY stale — the server emits |
| 29 | + xai/gemini/exa web rows today. |
| 30 | + |
| 31 | +## Wire research (from shipped web-search executors, probe-verified 2026-08-20/21) |
| 32 | + |
| 33 | +### xai describe wire |
| 34 | + |
| 35 | +Mirror src/web-search/xai-executor.ts: POST `https://api.x.ai/v1/responses` |
| 36 | +(origin pinned; provider baseUrl honored only on same origin), stored OAuth |
| 37 | +bearer via `getValidAccessToken`, `redirect: "manual"`. Body for describe: |
| 38 | + |
| 39 | +```json |
| 40 | +{ |
| 41 | + "model": "<settings.model>", |
| 42 | + "instructions": "<describe instruction>", |
| 43 | + "input": [{ "role": "user", "content": [ |
| 44 | + { "type": "input_text", "text": "<context>" }, |
| 45 | + { "type": "input_image", "image_url": "<data: or https: url>" } |
| 46 | + ]}], |
| 47 | + "reasoning": { "effort": "<low|medium|high>" }, |
| 48 | + "stream": true |
| 49 | +} |
| 50 | +``` |
| 51 | + |
| 52 | +SSE reduction: reuse the `response.output_text.delta` / `.done` handling |
| 53 | +shape from parseXaiResponsesSSE, without the citation/source machinery. |
| 54 | +Grok Responses accepts `input_image` with data URLs (same shape the OpenAI |
| 55 | +forward describer already posts — describe.ts builds input_image parts). |
| 56 | + |
| 57 | +### Gemini (Antigravity CCA) describe wire |
| 58 | + |
| 59 | +Mirror src/web-search/gemini-executor.ts: POST |
| 60 | +`{registry base}/v1internal:generateContent`, `ANTIGRAVITY_REQUEST_UA`, |
| 61 | +token + projectId via `getValidAccessTokenSnapshot`, envelope: |
| 62 | + |
| 63 | +```json |
| 64 | +{ |
| 65 | + "model": "<wireModelId from resolveAntigravityEffortWireModel>", |
| 66 | + "userAgent": "antigravity", "requestType": "agent", |
| 67 | + "project": "<projectId>", "requestId": "agent-<uuid>", |
| 68 | + "request": { |
| 69 | + "systemInstruction": { "role": "user", "parts": [{ "text": "<describe instruction>" }] }, |
| 70 | + "contents": [{ "role": "user", "parts": [ |
| 71 | + { "text": "<context>" }, |
| 72 | + { "inlineData": { "mimeType": "<mime>", "data": "<base64>" } } |
| 73 | + ]}] |
| 74 | + } |
| 75 | +} |
| 76 | +``` |
| 77 | + |
| 78 | +inlineData shape matches src/adapters/google.ts:972/:1233. Response mapping: |
| 79 | +`candidates[0].content.parts[].text` join (mapCcaGroundedResponse shape, |
| 80 | +minus grounding). https: image URLs cannot be inlined without proxy-side |
| 81 | +fetch — REJECTED for gemini describe (data: URLs only, documented delta, |
| 82 | +same stance as anthropic-describe's stricter base64 rule). |
| 83 | + |
| 84 | +## Metadata facts |
| 85 | + |
| 86 | +- xai vendor table: bare grok-2/grok-3/grok-4 are `text`-only; grok-4.x |
| 87 | + fast/4.3/4.5/4.6 and grok-2-vision are `text,image`. |
| 88 | +- No bare model id collides across the four vendor tables (openai 48, |
| 89 | + anthropic 26, xai 32, google 43; collision scan 2026-08-21: zero) — the |
| 90 | + "vendor tables never disagree" premise of visionDescriberIsProvablyBlind |
| 91 | + survives widening to four families. |
| 92 | + |
| 93 | +## Audit deltas folded into this unit (sol-medium audit, 2026-08-21) |
| 94 | + |
| 95 | +- **Blocker A**: `BASELINE_VISION_MODELS` is a TOTAL |
| 96 | + `Record<VisionSidecarBackend, string>`; widening the union without a |
| 97 | + decision breaks typecheck. Decision → doc 170: baselines become |
| 98 | + descriptor-owned (only openai/anthropic carry one). |
| 99 | +- **Blocker B**: `visionDescriberIsProvablyBlind` collapses non-anthropic |
| 100 | + hints to openai and probes two families; a bare grok id absent from |
| 101 | + candidates would slip the gate. Decision → doc 170: probe all four vendor |
| 102 | + families. |
| 103 | +- Empty-auth fallback stays `["openai","anthropic"]` — never offer |
| 104 | + xai/gemini unauthenticated. |
| 105 | +- GUI shared `SidecarBackend` must split (web-search has exa; vision does |
| 106 | + not). |
| 107 | +- New executors: `sidecarEnter("vision")` (NOT "web-search"), |
| 108 | + `signalWithTimeout` + `cancelBodyOnAbort`, `redactSecretString` on all |
| 109 | + error paths, timeout-bounds.ts as single authority. |
| 110 | + |
0 commit comments