Skip to content

Commit 97bedc4

Browse files
committed
Merge remote-tracking branch 'upstream/dev' into codex/fastwire-b2-xai
2 parents 030cc77 + 401c24f commit 97bedc4

24 files changed

Lines changed: 1376 additions & 80 deletions
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# 160 — Vision external-backend research (xai Grok / Antigravity Gemini describers)
2+
3+
Continuation of #2188. Web-search shipped four external backends (L6-L9, docs
4+
060-090); the vision sidecar still dispatches only openai-forward and
5+
anthropic-OAuth. GUI evidence: the vision dropdown lists only Codex/Claude
6+
rows while the web-search dropdown already lists Grok/Gemini.
7+
8+
## Current vision dispatch inventory
9+
10+
- Types: `OcxVisionSidecarConfig.backend?: "openai" | "anthropic"` (src/types.ts).
11+
- Union: `VisionSidecarBackend` (src/vision/eligibility.ts:30) — 2 arms.
12+
- Candidate mapping: `visionBackendForCandidate` (eligibility.ts:150-165) —
13+
native/openai → openai; anthropic only via the resolved OAuth provider name.
14+
- Options: `visionEligibleModelOptions` (eligibility.ts:201+) iterates
15+
`["openai","anthropic"] as const` and injects `BASELINE_VISION_MODELS`.
16+
- Enabled backends: `enabledVisionBackends`
17+
(src/server/management/vision-sidecar-options.ts:31-43); empty-auth fallback
18+
returns both universal sides.
19+
- Write gate: `visionDescriberIsProvablyBlind` (vision-sidecar-options.ts:94+)
20+
probes ONLY the openai/anthropic vendor tables.
21+
- PUT validation: config-routes.ts:594-596 rejects backends outside the two
22+
literals; hint fall-through at :623; claude-code override near :738-740.
23+
- Runtime plan: `planVisionSidecar` (src/vision/index.ts) — anthropic arm and
24+
openai-forward arm only. `resolveVisionBackend`: explicit > anthropic-if-auth
25+
> openai.
26+
- GUI: `SidecarBackend = "openai" | "anthropic"` (gui/src/pages/
27+
dashboard-shared.ts:62, claude-manual-env.ts:8). NOTE: this type is shared
28+
with WebSearchModelOption and is ALREADY stale — the server emits
29+
xai/gemini/exa web rows today.
30+
31+
## Wire research (from shipped web-search executors, probe-verified 2026-08-20/21)
32+
33+
### xai describe wire
34+
35+
Mirror src/web-search/xai-executor.ts: POST `https://api.x.ai/v1/responses`
36+
(origin pinned; provider baseUrl honored only on same origin), stored OAuth
37+
bearer via `getValidAccessToken`, `redirect: "manual"`. Body for describe:
38+
39+
```json
40+
{
41+
"model": "<settings.model>",
42+
"instructions": "<describe instruction>",
43+
"input": [{ "role": "user", "content": [
44+
{ "type": "input_text", "text": "<context>" },
45+
{ "type": "input_image", "image_url": "<data: or https: url>" }
46+
]}],
47+
"reasoning": { "effort": "<low|medium|high>" },
48+
"stream": true
49+
}
50+
```
51+
52+
SSE reduction: reuse the `response.output_text.delta` / `.done` handling
53+
shape from parseXaiResponsesSSE, without the citation/source machinery.
54+
Grok Responses accepts `input_image` with data URLs (same shape the OpenAI
55+
forward describer already posts — describe.ts builds input_image parts).
56+
57+
### Gemini (Antigravity CCA) describe wire
58+
59+
Mirror src/web-search/gemini-executor.ts: POST
60+
`{registry base}/v1internal:generateContent`, `ANTIGRAVITY_REQUEST_UA`,
61+
token + projectId via `getValidAccessTokenSnapshot`, envelope:
62+
63+
```json
64+
{
65+
"model": "<wireModelId from resolveAntigravityEffortWireModel>",
66+
"userAgent": "antigravity", "requestType": "agent",
67+
"project": "<projectId>", "requestId": "agent-<uuid>",
68+
"request": {
69+
"systemInstruction": { "role": "user", "parts": [{ "text": "<describe instruction>" }] },
70+
"contents": [{ "role": "user", "parts": [
71+
{ "text": "<context>" },
72+
{ "inlineData": { "mimeType": "<mime>", "data": "<base64>" } }
73+
]}]
74+
}
75+
}
76+
```
77+
78+
inlineData shape matches src/adapters/google.ts:972/:1233. Response mapping:
79+
`candidates[0].content.parts[].text` join (mapCcaGroundedResponse shape,
80+
minus grounding). https: image URLs cannot be inlined without proxy-side
81+
fetch — REJECTED for gemini describe (data: URLs only, documented delta,
82+
same stance as anthropic-describe's stricter base64 rule).
83+
84+
## Metadata facts
85+
86+
- xai vendor table: bare grok-2/grok-3/grok-4 are `text`-only; grok-4.x
87+
fast/4.3/4.5/4.6 and grok-2-vision are `text,image`.
88+
- No bare model id collides across the four vendor tables (openai 48,
89+
anthropic 26, xai 32, google 43; collision scan 2026-08-21: zero) — the
90+
"vendor tables never disagree" premise of visionDescriberIsProvablyBlind
91+
survives widening to four families.
92+
93+
## Audit deltas folded into this unit (sol-medium audit, 2026-08-21)
94+
95+
- **Blocker A**: `BASELINE_VISION_MODELS` is a TOTAL
96+
`Record<VisionSidecarBackend, string>`; widening the union without a
97+
decision breaks typecheck. Decision → doc 170: baselines become
98+
descriptor-owned (only openai/anthropic carry one).
99+
- **Blocker B**: `visionDescriberIsProvablyBlind` collapses non-anthropic
100+
hints to openai and probes two families; a bare grok id absent from
101+
candidates would slip the gate. Decision → doc 170: probe all four vendor
102+
families.
103+
- Empty-auth fallback stays `["openai","anthropic"]` — never offer
104+
xai/gemini unauthenticated.
105+
- GUI shared `SidecarBackend` must split (web-search has exa; vision does
106+
not).
107+
- New executors: `sidecarEnter("vision")` (NOT "web-search"),
108+
`signalWithTimeout` + `cancelBodyOnAbort`, `redactSecretString` on all
109+
error paths, timeout-bounds.ts as single authority.
110+
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# 170 — Backend union: "routed" describer (wp2, REVISED)
2+
3+
Depends on: 160. REVISION 2026-08-22: user directive — vision does not need
4+
per-backend executors. Any picker-visible model with image input can describe;
5+
the proxy's own router already speaks every provider wire. The earlier
6+
xai/gemini backend literals were implemented but never released; this revision
7+
replaces them before any push.
8+
9+
## Design
10+
11+
- `VisionSidecarBackend = "openai" | "anthropic" | "routed"`.
12+
- "openai"/"anthropic" arms unchanged (forward Responses / OAuth Messages) —
13+
they carry auth semantics loopback routing cannot replicate (forwarded
14+
headers, OAuth beta fences), and their defaults must not drift.
15+
- "routed": the describer is ANY routed model, dispatched through the proxy's
16+
own /v1/chat/completions on loopback (pattern: src/claude/gateway-cache.ts
17+
self-fetch). One executor, every provider.
18+
19+
## Filter (#2188 rules, unchanged shape)
20+
21+
1. Picker-visible ∪ auth slots (pickerVisibleSidecarCandidates).
22+
2. − provably text-only (modelAcceptsImageInput === false drops the row).
23+
24+
visionBackendForCandidate: native/openai → openai; resolved-OAuth anthropic
25+
row → anthropic; ANY OTHER provider row → "routed". Routed option values are
26+
NAMESPACED ("provider/model") so routeModel is unambiguous; legacy sides keep
27+
bare ids (GUI/current-value compatibility).
28+
29+
## Gate
30+
31+
visionDescriberIsProvablyBlind keeps the four-family probe widening AND
32+
learns namespaced ids: split on first "/", probe that provider's config row +
33+
metadata family. Bare ids keep the existing all-family probe.
34+
35+
## Runtime
36+
37+
- planVisionSidecar routed arm requires: cfg.backend === "routed", explicit
38+
cfg.model, and plan-time modelAcceptsImageInput !== false for the target.
39+
- Recursion safety: the loopback request re-enters the vision planner only if
40+
the routed model is provably text-only; the plan-time check excludes exactly
41+
that set, so describe recursion is structurally impossible.
42+
- resolveVisionBackend: explicit honored; unset default order UNCHANGED.
43+
44+
## Files (wp2 scope, revised)
45+
46+
- src/vision/eligibility.ts: union, visionBackendForCandidate routed arm,
47+
namespaced option values, BASELINE narrow-key record (kept from r1).
48+
- src/vision/backends.ts (r1 descriptor table): SIMPLIFIED — descriptors for
49+
openai/anthropic/routed; xai/gemini entries dropped.
50+
- vision-sidecar-options.ts: enabledVisionBackends offers "routed" whenever
51+
any routed row exists; gate learns namespaced ids.
52+
- config-routes.ts + agent-settings-routes.ts: literal sets accept "routed"
53+
(xai/gemini literals removed).
54+
- types: backend unions.
55+
- tests: vision-backend-union.test.ts rewritten for routed.
56+
57+
58+
## Audit round 2 amendments (2026-08-22, sol-medium)
59+
60+
- **Recursion fence is a MECHANISM, not a predicate claim.** The loopback
61+
describe request carries a terminal marker header
62+
`x-opencodex-vision-describe: 1`. The Responses plan site treats a marked
63+
request as terminal: images are STRIPPED, never described (depth cap 1).
64+
This holds under predicate drift (modelInputModalities is invisible to a
65+
row-less plan-time target) and combo re-resolution (router.ts:625-631 can
66+
land a different sibling). Belt-and-braces: the routed arm also requires
67+
`!isModelTextOnly(resolvedRoute.provider, resolvedRoute.modelId)` at plan
68+
time — the exact re-entry predicate on the resolved route.
69+
- **PUT-gate coherence:** a namespaced model with backend openai/anthropic is
70+
REJECTED (forward executor POSTs the string verbatim — web-search F1
71+
selector/slug failure); backend "routed" REQUIRES a namespaced id.
72+
- **GUI inference:** `value.includes("/") → "routed"` in
73+
visionSidecarBackendForModel's fallback; persisted backend keeps traveling
74+
as currentBackend.
75+
- Known limitation (recorded, not fixed here): a non-loopback-only bindHost
76+
where 127.0.0.1 does not answer — same latent limitation gateway-cache has.
77+
- handleNativeChatCompletions fast path has no vision handling; the marked
78+
describe request must not regress it (marker check lives at the Responses
79+
plan site the bridge replays into).
80+
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# 180 — Routed describe executor + dispatch (wp3, REVISED)
2+
3+
Depends on: 170 (revised).
4+
5+
## src/vision/routed-describe.ts (new)
6+
7+
Loopback POST http://127.0.0.1:{config.port}/v1/chat/completions:
8+
9+
```json
10+
{ "model": "<namespaced routed id>", "stream": false,
11+
"messages": [
12+
{ "role": "system", "content": "<describe instruction>" },
13+
{ "role": "user", "content": [
14+
{ "type": "text", "text": "<context>" },
15+
{ "type": "image_url", "image_url": { "url": "<data:/https:>" } }
16+
]}]}
17+
```
18+
19+
- Auth: none on loopback binds (resolveApiAuth admits loopback without a
20+
token); when OPENCODEX_API_AUTH_TOKEN is set, send it as Authorization
21+
bearer (auth-cors.ts:399-400 accepts bearer on /v1/chat/completions).
22+
- signalWithTimeout(settings.timeoutMs) + cancelBodyOnAbort;
23+
sidecarEnter("vision"); redactSecretString on error paths; response text
24+
from choices[0].message.content; DESC clamp caller-side (existing).
25+
- validateImageUrl reused (data: mime allowlist + 20MB, https passthrough).
26+
- The chat inbound translates image_url → input_image and every adapter
27+
compiles its own wire (anthropic blocks, CCA inlineData, xai Responses),
28+
so provider coverage is the router's, not this file's.
29+
30+
## planVisionSidecar routed arm
31+
32+
VisionPlan gains { backend: "routed", routedModel: string }. Arm requires
33+
explicit model + plan-time modelAcceptsImageInput !== false (recursion
34+
fence). executeDescription routed arm calls describeImageRouted.
35+
36+
## Tests
37+
38+
vision-routed.test.ts: wire shape against a mock loopback server; recursion
39+
fence (text-only target never plans routed); timeout/error taxonomy;
40+
redaction. E2E: routed describer via a second mock provider.
41+
42+
43+
## Audit round 2 amendments (2026-08-22)
44+
45+
- **Admission ladder (blocker 2):** token =
46+
configuredApiAuthToken() || loadServiceTokenFromFile(env) || first
47+
config.apiKeys entry; sent as `x-opencodex-api-key` (never Authorization —
48+
gateway-cache.ts:77-86 rule); omitted entirely on loopback binds where
49+
isApiAuthRequired is false.
50+
- **Terminal marker:** executor sets `x-opencodex-vision-describe: 1`; the
51+
core.ts plan site checks it and strips images instead of planning vision.
52+
- Executor also passes stream:false and reads choices[0].message.content;
53+
non-2xx → {error} with redacted body slice.
54+
55+
56+
## Audit round 3 amendment (2026-08-22) — marker propagation
57+
58+
The chat→responses bridge rebuilds headers from the FORWARD_HEADERS allowlist
59+
(chat-completions.ts:198-203, openai-responses.ts:28-36), which would DROP
60+
`x-opencodex-vision-describe` before the plan site — on exactly the one path
61+
recursion lives. Therefore:
62+
63+
- The marker is detected AT THE CHAT SURFACE (raw req.headers before the
64+
bridge) and carried as an explicit option/flag into handleResponses
65+
(`visionDescribeTerminal: true`), not as a header the bridge must
66+
preserve. The Responses surface ALSO honors the raw header directly for
67+
native /v1/responses callers.
68+
- Regression test drives the FULL chat-surface path: marked POST to
69+
/v1/chat/completions with an image + text-only routed model → assert the
70+
plan site STRIPS (no describe dispatch, no recursion), while the same
71+
unmarked POST plans normally. A predicate-only test is insufficient and
72+
would stay green with the marker broken.
73+
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# 190 — Surfaces, live proof, delivery (wp4 cycle)
2+
3+
Depends on: 180.
4+
5+
## GUI
6+
7+
- Split the shared SidecarBackend (dashboard-shared.ts:62): web-search side
8+
keeps its server-provided backend strings (already emits xai/gemini/exa —
9+
stale type fixed by the split); vision side gets
10+
VisionBackend = "openai" | "anthropic" | "xai" | "gemini".
11+
- visionSidecarBackendForModel fallback stays server-provenance-first;
12+
catalog inference (anthropic-vs-openai guess) only for legacy rows.
13+
- claude-manual-env.ts SidecarOverride backend union widens for vision.
14+
- No new dropdown UI: options arrive from visionModels server list already.
15+
16+
## CLI
17+
18+
- src/cli/agent.ts: usage already names xai|gemini; verify backend values
19+
pass through PUT unvalidated client-side (server gate authoritative);
20+
vision --list renders new backends' rows.
21+
22+
## Live proof (acceptance 3-5)
23+
24+
- GET /api/sidecar-settings on live :10100 shows visionModels containing
25+
xai/gemini rows (auth present on this machine for both — web-search rows
26+
prove it).
27+
- PUT vision {backend:"xai", model:"grok-4.3"} → 200; PUT model grok-4
28+
(bare) → 400 provably-blind; restore original settings after proof.
29+
- GUI screenshot of the vision dropdown listing Grok/Gemini rows.
30+
31+
## Delivery
32+
33+
- Small commits per layer (backends table / eligibility+gate / executors /
34+
GUI+CLI / tests+devlog), full bun run typecheck + bun run test green at
35+
final head, push directly to dev (user-authorized, no PR).
36+
- devlog docs 160-190 land with the same push train; unit stays in _plan
37+
until the release train closes it.
38+
39+
40+
## Delivery evidence (2026-08-22, wp4)
41+
42+
- Live dev server (commit 3ff19c33e, port 11100, copied auth home):
43+
- GET /api/sidecar-settings visionModels: 25 rows — legacy openai/anthropic
44+
sides + 17 namespaced [routed] rows (xai/grok-4.6,
45+
google-antigravity/gemini-3.7-flash, cursor/kimi-k3, zenmux/…,
46+
alibaba…/qwen3.8-max, …). Rule 2 confirmed live: no text-only rows.
47+
- PUT gates live: routed+xai/grok-4.6 → 200; routed+xai/grok-3 →
48+
400 provably-blind; openai+namespaced → 400 coherence.
49+
- GET after PUT reports the routed model verbatim
50+
({"model":"xai/grok-4.6","backend":"routed"}) — fixed the legacy-collapse
51+
display bug found during this verification.
52+
- GUI screenshot: vision dropdown lists namespaced routed rows; current
53+
selection renders as xai/grok-4.6.
54+
- CLI: `ocx agent sidecar vision --list` prints the same 25 rows with
55+
[routed] backend tags (server-computed list, no drift).
56+
- LIVE describe e2e: POST /v1/chat/completions with a 64x64 red PNG to
57+
xai/grok-composer-2.5-fast (noVisionModels) with routed describer
58+
xai/grok-4.6 → main answer "red"; request history shows the inner
59+
grok-4.6 describe call followed by the outer composer call. (A 1x1 probe
60+
earlier failed with xai invalid_image min-8px — upstream constraint, not
61+
a pipeline defect; the graceful degradation path handled it and the main
62+
call still succeeded.)
63+
- Verification-side effect handled: the 11100 dev server rewrote
64+
~/.grok/config.toml to port 11100 during startup sync; restored to 10100
65+
via production `ocx ensure` and confirmed (27x base_url 10100, zero
66+
11100). Temp verify home moved aside (/tmp/trash-ocx-vision-verify-*).
67+
- privacy:scan green; root+gui tsc clean; focused suites green (185 pass).
68+
- Full-suite run at final head queued behind another worktree's runner
69+
(scripts/test.ts exclusive-run queue); recorded separately below when it
70+
lands.
71+
193 KB
Loading

‎gui/src/pages/claude-code-sidecar.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@
44
* trimmed model is present.
55
*/
66

7-
import type { SidecarBackend, SidecarOverride } from "./claude-manual-env";
7+
import type { SidecarOverride, VisionOverrideBackend } from "./claude-manual-env";
88

9-
export type SidecarSelectValue = "inherit" | "auto" | SidecarBackend;
9+
export type SidecarSelectValue = "inherit" | "auto" | VisionOverrideBackend;
1010

1111
export type PersistedSidecarOverride = {
12-
backend: SidecarBackend | null;
12+
backend: VisionOverrideBackend | null;
1313
model: string;
1414
};
1515

‎gui/src/pages/claude-manual-env.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@
66
import { AUTO_COMPACT_WINDOW_DEFAULT } from "./claude-code-types";
77

88
export type SidecarBackend = "openai" | "anthropic";
9-
export interface SidecarOverride { backend?: SidecarBackend; model?: string }
9+
/** Vision override may carry "routed" (proxy-router describer, #2188). */
10+
export type VisionOverrideBackend = SidecarBackend | "routed";
11+
export interface SidecarOverride { backend?: VisionOverrideBackend; model?: string }
1012

1113
export interface ClaudeManualEnvState {
1214
/**

0 commit comments

Comments
 (0)