|
| 1 | +# WP2 — #398 sidecar graceful degradation + 2 follow-up issues (010) |
| 2 | + |
| 3 | +Issue #398: web-search/vision sidecar backend is fixed to openai/anthropic; when |
| 4 | +that backend's account limit is exhausted, the turn hangs ~200s then fails whole |
| 5 | +with 499/502; no graceful degradation to "continue this turn without search/vision". |
| 6 | + |
| 7 | +## Findings (Sol explorer, read-only) |
| 8 | + |
| 9 | +The branch ALREADY degrades ordinary sidecar HTTP/connect/timeout failures into |
| 10 | +error markers — executors are "never throws" (`src/web-search/executor.ts:84-100`, |
| 11 | +`anthropic-executor.ts:168-182`); `runSearchCall()` stores the failed outcome and |
| 12 | +forces the routed model to answer without search (`loop.ts:430-457`, |
| 13 | +`format-result.ts:30`). Vision similarly strips images with a marker |
| 14 | +(`vision/index.ts:197-204,334-340`). |
| 15 | + |
| 16 | +**The real defect is latency, not missing degradation.** The default sidecar |
| 17 | +deadline is 200_000ms (`src/web-search/index.ts:134,145-157`). A hung sidecar |
| 18 | +consumes up to 200s; the client cancels first → next routed op maps abort to |
| 19 | +`LoopError(499)` (`loop.ts:320,341,377`), or the forced-answer iteration fails |
| 20 | +as 502 (`loop.ts:343,379-389`, `lib/errors.ts:240`). So 499/502 is a symptom of |
| 21 | +the 200s wait, not a direct sidecar emission. |
| 22 | + |
| 23 | +Also: HTTP status/error-code is flattened into an untyped `error: string`, losing |
| 24 | +429 (exhausted) / 401 / 403 / 5xx and Anthropic `max_uses_exceeded` |
| 25 | +(`anthropic-executor.ts:49-55`), so we can't degrade *immediately* on a known |
| 26 | +exhaustion status. |
| 27 | + |
| 28 | +## Plan (scope boundary IN) |
| 29 | + |
| 30 | +No new `degradeOnSidecarFailure` flag (default-on is already the contract). |
| 31 | + |
| 32 | +MODIFY: |
| 33 | +- `src/web-search/executor.ts` — KEEP the existing sanitized `error: string` |
| 34 | + (consumers depend on it) AND add optional structured metadata |
| 35 | + `failure?: { kind: "http"|"timeout"|"connect"; status?; code?; message }`. |
| 36 | +- `src/web-search/anthropic-executor.ts` — preserve HTTP status + |
| 37 | + `web_search_tool_result_error.error_code`; raw response bodies enter outcomes |
| 38 | + at `anthropic-executor.ts:168-172` — SANITIZE before storing (never persist raw |
| 39 | + body/token into the outcome/tool-result/SSE/log). |
| 40 | +- `src/web-search/loop.ts` — defensive try/catch at `runSearchCall()` dispatch |
| 41 | + (`~427-432`); on 429/401/403/5xx degrade immediately; preserve genuine parent |
| 42 | + abort as 499. |
| 43 | +- `src/web-search/index.ts` — lower default sidecar degradation deadline from |
| 44 | + 200_000ms to a bounded 30_000–45_000ms; keep `webSearchSidecar.timeoutMs` |
| 45 | + override; recompute `stallTimeoutSec` (already at `:152-157`). |
| 46 | +- `src/vision/describe.ts` + `src/vision/anthropic-describe.ts` — structured |
| 47 | + failure metadata; keep marker degradation in `vision/index.ts`. |
| 48 | + |
| 49 | +Activation scenario (C-ACTIVATION-GROUNDING-01): a test where all sidecar queries |
| 50 | +return 429/502/connect/timeout must produce exactly one failed search cell, a |
| 51 | +routed-model answer, `response.completed`, and NO `response.failed`; and an |
| 52 | +internal sidecar timeout must degrade while a genuine parent abort stays 499. |
| 53 | + |
| 54 | +TESTS: |
| 55 | +- `tests/web-search.test.ts` — full-turn all-failed/timeout completion regression. |
| 56 | +- `tests/sidecar-abort.test.ts` — sidecar deadline degrades vs parent abort = 499. |
| 57 | +- `tests/vision-cache.test.ts` — timeout/429 marker, not cached. |
| 58 | +- `tests/web-search-anthropic.test.ts` / `tests/vision-anthropic.test.ts` — |
| 59 | + preserve error_code/type in structured outcomes. |
| 60 | + |
| 61 | +## Consumer compatibility (A-gate blocker #1 fold) |
| 62 | + |
| 63 | +The `error?: string` field is REQUIRED by existing consumers — do NOT replace it: |
| 64 | +- `src/web-search/executor.ts:32-40` (outcome shape) |
| 65 | +- `src/web-search/format-result.ts:27-30` (renders failed result) |
| 66 | +- `src/vision/index.ts:335-342` (catch → marker) |
| 67 | +Add `failure?` alongside; every consumer keeps reading `error`. New structured |
| 68 | +field is additive/optional. |
| 69 | + |
| 70 | +## Secret-safety (A-gate blocker #1 fold, STRICT) |
| 71 | + |
| 72 | +"reuse existing redaction" is NOT automatic on the Anthropic paths |
| 73 | +(`anthropic-executor.ts:168-172`, `vision/anthropic-describe.ts:166-170` put raw |
| 74 | +bodies into outcomes). Requirement: structured `failure.message` and `error` |
| 75 | +MUST be sanitized (status/code/short reason only, no raw body, no headers, no |
| 76 | +token). Add a sentinel-secret test: inject a fake token/secret into a mocked |
| 77 | +error body and assert it never appears in the outcome, tool result, SSE frames, |
| 78 | +or logs. `bun run privacy:scan` must stay green. |
| 79 | + |
| 80 | +## Two follow-up issues to create (per user directive) |
| 81 | + |
| 82 | +1. **feat(sidecar): add exa and other search providers as sidecar backends** — |
| 83 | + today backend ∈ {openai, anthropic} only; add exa (and pluggable search |
| 84 | + providers) so non-OpenAI/Anthropic accounts can back the search sidecar. |
| 85 | +2. **feat(sidecar): investigate/enable search-api-capable providers (e.g. gemini) |
| 86 | + as sidecar backends** — evaluate Gemini (and similar) native search APIs as |
| 87 | + selectable sidecar backends. |
| 88 | + |
| 89 | +Then comment on #398 linking both follow-ups and summarizing the degradation fix. |
| 90 | + |
| 91 | +## Security boundary |
| 92 | + |
| 93 | +Touches sidecar auth-adjacent paths but no credential logging/serialization is |
| 94 | +added. Structured failure must NOT include tokens/bodies with secrets (reuse |
| 95 | +existing redaction). Not a release-workflow change. |
| 96 | + |
| 97 | +Terminal: DONE = degradation code + tests green + 2 issues created + #398 comment. |
| 98 | +If B proves too large for one cycle, land the deadline-lowering + defensive catch |
| 99 | +first (the direct 499/502 fix) and split structured-metadata into a follow-up. |
| 100 | + |
| 101 | +## P stale-check (WP2 cycle, current tree) |
| 102 | + |
| 103 | +Re-read against current `src/web-search/*`: |
| 104 | +- `executor.ts:84-100` and `anthropic-executor.ts` ALREADY degrade HTTP non-2xx |
| 105 | + and timeout/connect into `{ error }` and ALREADY `redactSecretString(...)` the |
| 106 | + body (`executor.ts:88`). So HTTP 429/401/403/5xx already degrade FAST — they are |
| 107 | + not the hang. |
| 108 | +- The ONLY 200s source is the sidecar search deadline default: |
| 109 | + `index.ts:19 DEFAULT_TIMEOUT_MS = 200_000` → `index.ts:145 timeoutMs = |
| 110 | + cfg.timeoutMs ?? DEFAULT_TIMEOUT_MS` → `settings.timeoutMs` → |
| 111 | + `executor.ts:73 signalWithTimeout(settings.timeoutMs)`. Vision already uses |
| 112 | + 45_000 (`vision/index.ts:16`). |
| 113 | +- `#398`'s observed `201,434ms` ≈ 200_000ms confirms the deadline is the cause. |
| 114 | + |
| 115 | +### Scoped B for THIS cycle (direct 499/502 fix) |
| 116 | + |
| 117 | +1. MODIFY `src/web-search/index.ts` — lower `DEFAULT_TIMEOUT_MS` 200_000 → 45_000 |
| 118 | + (sidecar SEARCH deadline only; do NOT touch |
| 119 | + `DEFAULT_ROUTED_MODEL_STALL_TIMEOUT_MS`, which is the routed-model body |
| 120 | + inactivity budget, a different knob). `cfg.timeoutMs` override still honored; |
| 121 | + `webSearchStallTimeoutSec` recomputes from the smaller value automatically. |
| 122 | +2. MODIFY `src/web-search/loop.ts` — defensive try/catch around the |
| 123 | + `runWebSearch`/`runAnthropicWebSearch` dispatch (~:430) so any future executor |
| 124 | + throw becomes a failed `SidecarOutcome` (executors are "never throws" today, |
| 125 | + but this makes the contract enforced, not assumed). Re-raise genuine parent |
| 126 | + aborts (signal.aborted) so real client cancel stays 499. |
| 127 | +3. TEST `tests/web-search.test.ts` — assert the default sidecar deadline is now |
| 128 | + bounded (45s) and a thrown executor degrades to a failed cell + completed turn. |
| 129 | + |
| 130 | +### Deferred to follow-up (NOT this commit) |
| 131 | + |
| 132 | +The structured `failure{ kind,status,code }` metadata refactor (A-gate blocker |
| 133 | +#1) is DEFERRED — since executors already redact+degrade, it is an enhancement, |
| 134 | +not required for the 499/502 fix. Deferring it also removes the secret-safety |
| 135 | +risk from this commit (no new raw-body handling is added). It becomes a THIRD |
| 136 | +follow-up note in the #398 comment. This matches the "if B too large" clause above. |
| 137 | +## A-gate (Halley) folds — GO-WITH-FIXES blockers=3 |
| 138 | + |
| 139 | +1. **Deadline 60_000ms, not 45_000.** Per-search deadline; maxSearches=3 |
| 140 | + sequential; hosted-search p90 ≈ 43s (commit 11cc822e). 45s leaves no tail |
| 141 | + margin → use `DEFAULT_TIMEOUT_MS = 60_000`. `cfg.timeoutMs` override intact; |
| 142 | + stall watchdog still ≥230s (index.ts:52, stall-timeout.ts:8). |
| 143 | +2. **Abort correctness.** Executors already catch parent abort and RETURN |
| 144 | + `{error}` (executor.ts:96, anthropic-executor.ts:179) — so checking |
| 145 | + `signal.aborted` only in a catch misses the fulfilled `{error}`. Check |
| 146 | + `signal.aborted` immediately AFTER the dispatch await (and in catch), and |
| 147 | + throw `LoopError(499, "client closed request during web-search")` to match |
| 148 | + loop.ts:320/341/377. Add a parent-abort-during-sidecar regression. |
| 149 | +3. **Docs sync (SOT-SYNC-01).** 200s default is promised in |
| 150 | + `structure/04_transports-and-sidecars.md:129`, |
| 151 | + `docs-site/.../reference/configuration.md:380` (+ `:450-451` example) and |
| 152 | + translations (`zh-cn/.../configuration.md`). Update the `timeoutMs` default to |
| 153 | + 60000 there or users override the fix back to 200s. |
| 154 | +4. **Bonus security fold (Halley note):** `anthropic-executor.ts:168` places an |
| 155 | + UNSANITIZED HTTP body into `error`. Wrap it with `redactSecretString(...)` in |
| 156 | + THIS commit (small, real secret-leak) — matches `executor.ts:88`. |
| 157 | + |
| 158 | +Final scoped B: `DEFAULT_TIMEOUT_MS 200_000 → 60_000`; loop.ts defensive |
| 159 | +catch + post-await abort→LoopError(499); anthropic-executor body redaction; |
| 160 | +docs sync (en + zh-cn + structure/04); regression tests. |
0 commit comments