Skip to content

Commit 495d3e4

Browse files
authored
feat(web-search): exa executor and the non-LLM search lane (lidge-jun#2188 L9) (lidge-jun#2245)
* feat(web-search): exa executor and the non-LLM search lane (lidge-jun#2188 L9) runExaWebSearch POSTs api.exa.ai/search with the operator key and maps ranked results to a digest the routed model synthesizes from. The key never rides the SidecarPlan — core.ts reads it from config at unpack time — and the executor scrubs the literal key from every error string (pattern-based redaction cannot know an arbitrary operator key; canary-tested). Plan, loop, and registry arms fail closed without the key. docs-site gains the explicit-only backend table. * fix(web-search): scrub the exa key before truncating error bodies Reviewer blocker (L9 round 2): error(t.slice(0,200)) truncated before the literal-key scrub, so a key straddling the 200-char boundary left an unscrubbable prefix in the returned tool error. Scrub first, then slice. Adds truncation-boundary and fetch-rejection canaries; 9/9 focused tests, tsc and privacy:scan green. * docs(devlog): integration merge-train roadmap 100-150 (chat default, global order, opt-in switch) Amends the 260820 unit with the audited (3-round sol-medium, round-3 PASS) roadmap: 100 chat-default regression as an atomic lidge-jun#2227+tier-policy unit with a 5-row regression matrix and the E2E reasoning-streaming proof; 110 global cross-train merge order and 21-PR triage matrix (lidge-jun#2072 deferred, lidge-jun#2217 RESHAPE); 120 sidecar L1-L9 merge execution with the fresh blocker inventory; 130 atomic xai Responses opt-in switch (single provider id, auth-mode-scoped sections, virtual PATCH field); 140 release prep; 150 blocking lidge final gate. DeepSeek explicitly out of scope per user decision. * docs(devlog): fold C-gate blockers into roadmap 100-150 Split the opt-in DTO into a write boolean vs read tri-state; record the concrete lidge-jun#2238 (3) and lidge-jun#2242 (5) review blockers in doc 120; recast doc 150 as the final aggregate gate with the full GUI/i18n/docs chain; replace temporal API-key rows with exact wire+tier assertions; state the explicit wp9->wp8->wp11->wp10 execution sequence. * docs(devlog): doc 100 API-key opt-in row preserves current tier forwarding C-gate round 2: current dev forwards caller service_tier verbatim on the API-key + explicit openai-responses route (fastPolicyForModel proof). The tier drop is an OAuth-route policy only; the API-key row now states preserve-current semantics, consistent with doc 130. * docs(devlog): wp9 execution record — all six chain blockers resolved and pushed
1 parent 2bf4e3d commit 495d3e4

13 files changed

Lines changed: 539 additions & 3 deletions

File tree

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# 100 — Chat-default regression for Grok 4.5/4.6 (#2227 integration unit)
2+
3+
Audited: 3-round adversarial plan audit (sol-medium), round-3 PASS. Amendment of this
4+
unit's roadmap for the integration merge-train; consumes user directives from 260821.
5+
6+
## Decision
7+
8+
Third-party Responses APIs are supported only as first-class surfaces. Outside OpenAI,
9+
the default wire is `openai-chat`. xAI's own CLI defaults to Chat; its Responses dialect
10+
rejects opaque reasoning continuation/compaction state on later turns (#2240 regression
11+
axis, 4-layer sanitize chase in #2217). Chat translation structurally filters private
12+
extensions instead of chasing them.
13+
14+
- ADOPT #2227: `modelWireDefaults` for grok-4.6/grok-4.5 flip `openai-responses` ->
15+
`openai-chat` (src/providers/registry.ts:1032/1038 area), structure/04 rewrite, test
16+
conversions to explicit `modelAdapters` opt-in framing.
17+
- DeepSeek is OUT OF SCOPE (user decision 260821): deepseek-v4-flash/pro keep their
18+
Responses defaults (registry.ts:1563-1564). Add a focused non-regression test locking
19+
both V4 entries to `openai-responses` so this train cannot drift them.
20+
- The Responses implementation is NOT deleted; it becomes the opt-in lane (doc 130).
21+
22+
## Atomic merge unit (audit blocker R2-B1)
23+
24+
The #2227 flip and the service_tier policy fix land as ONE merge unit — no intermediate
25+
dev head may exist where OAuth opt-in Responses leaks caller service_tier:
26+
27+
1. Cherry-pick/merge #2227's registry + structure/04 + test changes onto the post-stack
28+
dev head (anchor: doc 110 global order).
29+
2. In the same unit, fix the fastwire.ts:151 bypass: configured `modelAdapters` must not
30+
skip the registry OAuth policy. Per audit round-3 note: make the OAuth registry tier
31+
policy UNCONDITIONAL for the matching xAI route rather than introducing a new config
32+
field — `modelAdapters` values are wire ids only.
33+
3. Regression matrix locked in tests (5 rows):
34+
| route | expectation |
35+
|---|---|
36+
| OAuth default | chat wire |
37+
| OAuth explicit Responses (modelAdapters) | responses wire, caller service_tier dropped |
38+
| API-key default | chat wire; no tier injected; caller service_tier not forwarded unless a capability declares it |
39+
| API-key explicit Responses (modelAdapters) | responses wire; PRESERVE current dev semantics: absent tier stays absent, caller-supplied service_tier forwards verbatim (resolver proof: forwardCallerTier true on this route today; #2072 deferred) |
40+
| DeepSeek V4 flash/pro | responses default unchanged |
41+
42+
## Reasoning-streaming proof (#1886 origin)
43+
44+
#1886 moved grok to Responses because Chat translation showed a blank screen during long
45+
reasoning turns. The regression must prove the Chat path now streams reasoning as an E2E
46+
SSE assertion, not unit-only: an early upstream `reasoning_content` delta must be
47+
observed on Codex's reasoning-summary SSE channel BEFORE the completion is released.
48+
Test shape: mock xAI chat stream emitting reasoning_content deltas first; assert the
49+
bridged Responses SSE emits reasoning summary deltas before `response.completed`.
50+
Follow with one live probe through the running proxy.
51+
52+
## Out of scope
53+
54+
- #2072 API-key Fast/priority policy: DEFERRED (open, conflicting, its own review
55+
cycle). The opt-in switch spec (doc 130) does not depend on it.
56+
- DeepSeek wire changes: none.
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# 110 — Global cross-train triage matrix and merge order
2+
3+
One global order (audit R1-B5): the sidecar stack lands FIRST (9 stacked branches;
4+
cascading a rebase through them is the expensive direction), then the lighter triage
5+
PRs rebase onto the post-stack dev head one at a time. Every individual merge requires
6+
maintainer approval + required CI green (MAINTAINERS.md) — the user's 'CI as lagging
7+
indicator' applies only to repair iterations between pushes, never to the merge click
8+
itself (audit R1-B4).
9+
10+
## Global order
11+
12+
1. Sidecar stack bottom-up: #2203 -> #2204 -> #2206 -> #2209 -> #2211 -> #2238 ->
13+
#2242 -> #2243 -> #2245 (details + blocker inventory: doc 120).
14+
2. #2227 integration unit (Chat default + unconditional OAuth tier policy, doc 100).
15+
3. #2217 RESHAPED (not raw): rebase onto post-#2227 dev, reframe tests/docs as opt-in
16+
hardening, gate the compat rewrite to the opt-in Responses route.
17+
4. Doc-130 atomic opt-in switch (wp11).
18+
5. Remaining Responses fixes: #2237, #2229, #2228 (do not depend on the Responses
19+
default; still valuable for the opt-in lane and other Responses routes).
20+
6. luvs01 / Ingwannu / docs PRs (matrix below).
21+
7. Release prep (doc 140) -> lidge final gate (doc 150).
22+
23+
## Triage matrix (dispositions)
24+
25+
| PR | author | disposition | rationale |
26+
|---|---|---|---|
27+
| #2227 | olddonkey | MERGE (as doc-100 atomic unit) | owns the chat default |
28+
| #2217 | olddonkey | RESHAPE then merge | opt-in-lane hardening; raw form encodes Responses-as-default |
29+
| #2237 | olddonkey | MERGE after #2227 | null reasoning channel drop; wire-agnostic |
30+
| #2229 | olddonkey | MERGE after #2227 | encrypted_content reshape guard; opt-in lane |
31+
| #2228 | olddonkey | MERGE after #2227 | compaction blob provenance; wire-agnostic |
32+
| #2214 | luvs01 | MERGE (address CHANGES_REQUESTED) | continuation binding bug |
33+
| #2236 | luvs01 | MERGE (address CHANGES_REQUESTED) | catalog comment preservation |
34+
| #2226 | luvs01 | MERGE after hygiene unblocked | secret redaction in events |
35+
| #2196 | Ingwannu | MERGE | maintainer chore, privacy-bounded diagnostics |
36+
| #2207 | Ingwannu | MERGE | google tool-result adjacency bug |
37+
| #2202 | Ingwannu | MERGE | claude roster sync bug |
38+
| #2181 | lidge-jun | MERGE (address CHANGES_REQUESTED) | devlog docs |
39+
| #2168 | lidge-jun | MERGE (address CHANGES_REQUESTED) | devlog docs |
40+
| #2235 | umyunsang | REVIEW-ONLY this train | contributor draft gate owns it |
41+
| #2220 | Hylouis233 | REVIEW-ONLY this train | draft, capability sync |
42+
| #2230 | ppvia | OUT (hygiene-blocked draft) | own cycle |
43+
| #2222 | MarcTCruz | OUT (hygiene-blocked draft) | own cycle |
44+
| #2216 | leon80900 | CLOSE-DIRECT (wrong branch, targets main) | ask re-file onto dev |
45+
| #2215 | parkjs101 | OUT (docs draft, changes requested) | own cycle |
46+
| #2213 | louis-tepe | OUT (draft, overlaps doc-130 design) | revisit post-switch |
47+
| #2072 | olddonkey | DEFERRED (audit R2-B2) | Fast policy composes later; must re-verify against both wires |
48+
49+
Rebase anchors are named at execution time in each phase's B (exact dev head SHA),
50+
with cascade verification (typecheck + focused suites) after every land.
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# 120 — Sidecar chain merge execution (L1-L9 into dev)
2+
3+
Order: #2203 -> #2204 -> #2206 -> #2209 -> #2211 -> #2238 -> #2242 -> #2243 -> #2245.
4+
Each merge: resolve CHANGES_REQUESTED, obtain maintainer approval, required CI green,
5+
squash-merge into its base, retarget the next child, cascade-verify (typecheck +
6+
focused suites), then proceed. Mid-stream lidge suites are lagging indicators between
7+
pushes; the merge click itself is gated (MAINTAINERS.md).
8+
9+
## Current blocker inventory (fresh, 260821)
10+
11+
- #2203 (L1, CHANGES_REQUESTED Ingwannu): blocker is the tracked cleanup doc
12+
000_wp0_branch_worktree_cleanup.md — contradictory KEEP/REMOVE entries, no preflight,
13+
incomplete protected set, b2ac2500c preservation, codex/merge-loop-closeout listed
14+
both ways. Fix: rewrite the doc as a non-executable historical record (all deletions
15+
already executed in wp0) with a mechanical protected-set preflight template; or mark
16+
every command block as executed-snapshot. No runtime code change.
17+
- #2204, #2206 (L2, L3): APPROVED. Rebase-carry only.
18+
- #2209 (L4, CHANGES_REQUESTED): runtime blocker — webSearchModelOptionsFrom drops
19+
backend provenance; auth-slot model can persist {backend:'openai',
20+
model:'claude-haiku-4-5'}. Fix: return (backend, model) pairs, validate the pair in
21+
both PUT routes, teach sidecarBackendForModel the auth-slot rows.
22+
- #2211 (L5, CHANGES_REQUESTED): carry the L4 provenance field through the CLI
23+
contract — show backend in `web --list` human output and validate pairs on write.
24+
- #2238 (L6, CHANGES_REQUESTED, head a05f23fa9) — three reviewed blockers:
25+
1. stripOpenAiOnlyWebSearchFields fires for every non-ChatGPT-forward Responses
26+
provider; official OpenAI API-key traffic loses external_web_access /
27+
search_context_size. Gate on xAI-specific provider identity/capability and add
28+
a buildRequest regression proving OpenAI API-key tools retain both fields.
29+
2. English config reference + CLI help still advertise only the old backend pair;
30+
document the xai/gemini/exa arms as explicit-only/inert; keep translations
31+
consistent.
32+
3. exaApiKey only in SENSITIVE_KEY_PATTERN: add it to the shared colon/query/JSON
33+
string-redaction grammar with all three canaries in tests/redact.test.ts.
34+
- #2242 (L7, CHANGES_REQUESTED, head 0f2d670c0) — five reviewed blockers:
35+
1. runXaiWebSearch misses cancelBodyOnAbort after fetchWithResetRetry resolves
36+
(abort-before-reader race).
37+
2. parseXaiResponsesSSE must cancel the upstream body at the byte bound, not just
38+
release the reader lock.
39+
3. the management PUT mutates config.webSearchSidecar before xSearch validation
40+
(400 after live state change) — stage and validate the complete candidate first.
41+
4. malformed xSearch fields are silently omitted — reject invalid handle arrays,
42+
dates, and enabled values instead of broadening the search with a 200.
43+
5. public docs + the type comment still call xai inert; update the English source
44+
and translations. Add no-partial-mutation and oversized-stream regressions.
45+
- #2243 (L8, CHANGES_REQUESTED): three runtime blockers per review + red macOS CI
46+
shard — full RCA in the phase B, fixes + rerun.
47+
- #2245 (L9): reviewer PASS locally; needs maintainer approval; one failing test shard
48+
reported on CI — reproduce, fix, re-push.
49+
50+
Execution sequence (explicit — the wp numbers are not the order): wp9 (this doc,
51+
sidecar chain) -> wp8 (triage PRs per doc 110) -> wp11 (doc 130 switch) -> wp10
52+
(docs 140/150).
53+
54+
## Execution record (wp9 B-phase, 260821)
55+
56+
Every recorded blocker resolved and pushed; Ingwannu re-review re-requested on all
57+
six layers. Worker lanes ran in parallel worktrees under .tmp/ (four sol-medium
58+
subagents + one direct fix):
59+
60+
- #2203 d505dacc7 — cleanup doc recast as an executed historical record; corrected
61+
36-ref remote list (merge-loop-closeout excluded); 5-step preflight template.
62+
- #2209 98eaba601 — options carry (backend, model) pairs; both PUT routes validate
63+
effective pairs; auth-slot Anthropic persists anthropic/claude-haiku-4-5.
64+
Suites 50/0 + 30/0, GUI 9/0, gate 13/0.
65+
- #2211 84357bd2b (parent merge) + 2a610909f — backend-tagged `web --list`,
66+
provenance-aware pair writes, clear rejection errors. CLI suites 401/0.
67+
- #2238 19376f737 — strip gated on supportsOpenAiWebSearchToolFields:false (xAI
68+
registry declares it; OpenAI API-key traffic keeps both fields — regression red
69+
pre-fix); docs/CLI-help union across 8 locales; exaApiKey in the shared
70+
colon/query/JSON redaction grammar, 3 canaries (JSON canary red pre-fix).
71+
- #2242 b2c2054b5 — cancelBodyOnAbort after resolve; byte-bound upstream body
72+
cancel; staged atomic PUT validation (no-partial-mutation); malformed xSearch
73+
rejected with 400; docs/type-comment de-inerted. 47/0 + 12/0.
74+
- #2243 249cc91a3 — atomic token/project snapshot; post-header abort guard;
75+
bounded 64KiB UTF-8 JSON reads. 4 regressions red pre-fix; 50/0, privacy green.
76+
77+
Remaining before merge clicks: Ingwannu approvals + green required CI per layer
78+
(#2245's earlier shard failure not reproduced at the current head — checks green
79+
except queued/pending reruns). Lidge full suite relaunched at stack top
80+
(/tmp/ocx-gate-stack.log) as the lagging indicator.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# 130 — xAI Responses opt-in switch (atomic persistence + GUI)
2+
3+
The Responses implementation survives as an explicit opt-in lane, like the x_search
4+
opt-in block. Default: chat (post-#2227). Surface: xai is ONE provider id
5+
(registry.ts:1000); the 'two pages' are the auth-mode-scoped sections (OAuth account +
6+
API key) of the same provider workspace.
7+
8+
## Contract (audit R1-B2)
9+
10+
- Config truth: `modelAdapters` entries for grok-4.5 + grok-4.6 -> 'openai-responses'.
11+
- Atomic management API: extend the provider PATCH surface (provider-routes.ts:378 area
12+
+ gui provider-workspace DTO types.ts:88) with a split write/read contract:
13+
- WRITE (PATCH input): `xaiResponsesOptIn: boolean` — sets/clears BOTH grok model
14+
entries in one config transaction, preserving unrelated modelAdapters overrides.
15+
- READ (GET/echo DTO): `xaiResponsesOptInState: true | false | "mixed"` — partial
16+
pre-existing state (one model set, one not) reads "mixed"; the first boolean
17+
write normalizes both entries and the echo returns the effective state.
18+
- GUI: one switch rendered in both auth-mode sections; mixed state shows indeterminate.
19+
- Tier policy: the doc-100 unit already made the OAuth registry tier policy
20+
unconditional; the switch adds NO tier behavior. API-key route: opt-in flips wire
21+
only; everything else keeps current dev semantics (#2072 deferred).
22+
- #2217 sanitize layers arm only on this opt-in Responses route (RESHAPE disposition).
23+
- Tests: atomic set/clear, override preservation, mixed-state normalization, effective-
24+
state echo, GUI switch render + PATCH round-trip, opt-in wire selection E2E.
25+
- Docs-site: provider page gains the switch row; structure/04 notes the opt-in lane.
26+
27+
Execution phase: wp11 (after #2227 unit + reshaped #2217; before release prep).
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# 140 — dev release prep
2+
3+
After wp11 lands. Version bump per release train conventions; release notes cover:
4+
sidecar unification (L1-L9), chat-default regression + opt-in switch, responses fixes
5+
(#2237/#2229/#2228, reshaped #2217), maintainer fixes (#2196/#2207/#2202), luvs01
6+
fixes (#2214/#2236/#2226), devlog docs (#2181/#2168). scripts/release.ts is the
7+
authority; no release action without the doc-150 gate green.
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# 150 — lidge final aggregate/release gate
2+
3+
The final AGGREGATE gate before release. It does not replace per-merge gating: docs
4+
110/120 keep approval + required CI blocking for every individual merge; 'lagging
5+
indicator' covers only repair iterations between pushes. At the final dev head:
6+
7+
- lidge: OCX_TEST_NO_QUEUE=1 bun run test — full suite green (baseline 13808+/0 at L9).
8+
- Local: bun run typecheck, bun run privacy:scan, bun run lint:gui, lint:i18n, GUI
9+
tests, bun run build:gui, and the docs-site build (doc 130 touches GUI and
10+
localized copy, so the full GUI/i18n/docs chain is in the gate).
11+
- GitHub Actions: final dev head green on Linux/Windows/macOS (Windows gate is a
12+
standing release requirement).
13+
- Live probes through the running proxy: OAuth chat default turn, opt-in Responses
14+
turn (no caller service_tier upstream), x_search opt-in turn, exa sidecar turn,
15+
reasoning-streaming E2E (doc 100 matrix).
16+
- Release staged on lidge per release-train conventions; promotion remains
17+
maintainer-controlled.

‎docs-site/src/content/docs/guides/sidecars.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@ closed. Explicit `xai` requires a usable stored Grok OAuth account and does not
1818
requires both ChatGPT login auth and an enabled `forward` provider.
1919
:::
2020

21+
### Additional web-search backends (explicit-only)
22+
23+
Three more web-search backends exist beyond the ChatGPT and Claude paths. Each is
24+
**explicit-only** — it never activates from credential presence — and **fails closed**:
25+
a missing credential produces no sidecar plan and the request takes the normal routed path.
26+
27+
| Backend | Runs | Credential | Notes |
28+
| --- | --- | --- | --- |
29+
| `xai` | Grok hosted `web_search` (+ opt-in `x_search`) on `api.x.ai` Responses | Stored Grok OAuth (`ocx login xai`) | `webSearchSidecar.xSearch` enables X search with `allowedXHandles`/`excludedXHandles` (max 20, mutually exclusive) and ISO `fromDate`/`toDate`. Default model `grok-4.6`. |
30+
| `gemini` | `google_search` grounding on the Antigravity transport | Stored Antigravity OAuth with a discovered project (`ocx login google-antigravity`) | Default model `gemini-3.7-flash`; reasoning maps to the tiered thinking level. |
31+
| `exa` | Exa Search API (non-LLM result digest) | `webSearchSidecar.exaApiKey` | The key is write-only through the management API (never echoed, redacted from logs). No sidecar model applies. |
32+
2133
## Web-search sidecar
2234

2335
When Codex requests hosted `web_search` for a non-passthrough routed model, opencodex:

‎src/server/responses/core.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3589,6 +3589,8 @@ async function handleResponsesInner(
35893589
xaiSidecar: wsPlan.xaiSidecar,
35903590
geminiSidecar: wsPlan.geminiSidecar,
35913591
xaiSearchOptions: wsPlan.xaiSearchOptions,
3592+
// The exa key never rides the plan: read it from config at unpack time (L9).
3593+
...(wsPlan.exaConfigured ? { exaApiKey: config.webSearchSidecar?.exaApiKey } : {}),
35923594
hostedTool: wsPlan.hostedTool,
35933595
selectedForwardHeaders: wsPlan.forwardSidecar?.headers ?? selectedForwardHeaders,
35943596
settings: wsPlan.settings,

‎src/web-search/backends.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,13 @@ export const WEB_SEARCH_BACKENDS: readonly WebSearchBackendDescriptor[] = [
7979
},
8080
eligibleModel: candidate => candidate.provider === "google-antigravity",
8181
},
82+
{
83+
backend: "exa",
84+
// Probe = operator key present. Exa is not an LLM: no candidate models ever
85+
// match, so the GUI's model list stays untouched by this backend.
86+
isActive: (_auth, config) => !!config.webSearchSidecar?.exaApiKey,
87+
eligibleModel: () => false,
88+
},
8289
];
8390

8491
/**

0 commit comments

Comments
 (0)