Repository navigation
test(worker): pin one assistant turn per observer reply, summary included - #3619
Conversation
df472be to
569ff25
Compare
|
…d request loops Three users were forced onto higher-tier plans by the observer burning their subscription quota. Two independent unbounded loops are responsible. 1. Quadratic conversation growth, then a permanent wedge (thedotmack#3800) The observer keeps one long-lived conversation per session and appends every captured tool call to it. Nothing trimmed it, and OpenAICompatibleProvider re-sent the whole array on every observation, so total input tokens for a session were O(N^2) in the number of tool calls. A single observation prompt is field-truncated to ~32k chars, so ~25-30 observations crossed a 200k ceiling. Past that the provider answers "Prompt is too long" as ordinary assistant text. It was not classified, so it fell through to generic prose, the batch was confirmed and dropped, and the next observation appended yet more — every later tool call paid full freight for a request that could never succeed. A reporter logged 2,264 of these in one day across 28 sessions. Nothing stopped it: consecutiveInvalidOutputs was assigned 0 at all five sites and never incremented, so no breaker keyed on it could trip, and the rejection log line printed the counter it had just zeroed — which read as "breaker fine" on every failure. - Add compactConversationHistory: pins the framing message, keeps the newest turns, drops the middle oldest-first, and announces the elision. Per-request size is now flat in session length instead of linear. - Add isContextOverflowObserverOutput and handle it: preserve the batch, recycle the conversation, force a fresh one, and hard-pause after two consecutive failures instead of retrying forever. - Stop appending rejections to history — a failure must not enlarge the next request. - Stop double-appending assistant replies (provider + ResponseProcessor both pushed the same text, inflating every later request by ~50%) (thedotmack#3619). - Count overflows on their own counter so benign skip acknowledgements keep resetting without rebuilding the respawn debt that reset was added to stop. 2. One doomed request per observation after the allowance is spent (thedotmack#3634) Once a provider reported the inference allowance exhausted, the generator exited and the very next captured tool call started a fresh one that earned the same refusal — for the rest of the billing cycle. 11 capped users produced tens of thousands of cap events in a day, ~100 per successful observation. ensureGeneratorRunning already had this gate for a missing Claude CLI; add the same shape for quota. The breaker arms on a quota exit (classified error or quota prose), withholds requests for a cooldown, lets exactly one probe through after it, and clears immediately on a successful store. Full suite green: 2743 pass, 0 fail across 256 files. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01ERvWLxiY96C7pJrx7hPjh8
|
Routing to #3606 plan-18 (Observer Response Pipeline) — "conversationHistory unbounded / double-appended" is listed in the master, and this is the double-append half. Verified on |
…uded 7a7ada5 removed the provider-side pushes that recorded every reply twice. processAgentResponse now records the observation and summary replies, and handleInitResponse records the init reply, which since a00ce61 no longer reaches processAgentResponse. The observation path is pinned: openai-compatible-init-response.test.ts asserts that roles alternate. The summary path is not. Putting the push back in processSummaryMessage changes no result in the 21 test files that drive the OpenAI-compatible providers or processAgentResponse, and neither does pushing an empty turn when an empty observation reply is not forwarded. Pin both: init, observation and summary each leave exactly one assistant turn and the roles alternate, and an unforwarded empty observation reply adds none.
569ff25 to
59cd021
Compare
|
Rebased onto current |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The test-only changes provide focused regression coverage with no unresolved issues.
Review effort: Lite
Findings: None
What changed in this PR
Adds regression tests for OpenAI-compatible conversation-history bookkeeping.
Changes:
- Verifies replies are appended exactly once across init, observation, and summary turns.
- Verifies unforwarded empty replies add no assistant turn.
| File | Description |
|---|---|
tests/worker/openai-compatible-history-append.test.ts |
Adds conversation-history append regression tests. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
|
Merged, thank you @ntdatt812! These tests pin the rule that every observer reply lands in the conversation history exactly once, covering the summary turn and the empty-reply case that main's existing tests never touched. The duplicate-append fix on main already cited this PR, and now there's a regression net behind it. Your mutation table made the value easy to see. |
… cap configurable, name truncations Trimmed to the three pieces main still lacked, rebuilt on main's request path (buildOpenRouterRequestBody, the thedotmack#4003 max_completion_tokens retry, the trial-expiry fallback clearing and the gateway classification stay as thedotmack#4276 left them): - System anchor: a generation's framing prompt (marked by openObserverGeneration) goes out as the system message on OpenRouter and custom endpoints and as systemInstruction on Gemini. Its user-request block (or a fixed kickoff) stays the first user turn, so every request, the init one included, carries a user message and roles still alternate. Standalone requests (field condensation, the wrap-up) are unchanged. - CLAUDE_MEM_OBSERVER_MAX_OUTPUT_TOKENS (default 4096, 256 to 1,000,000): max_tokens in buildOpenRouterRequestBody, max_completion_tokens on the compatibility retry, Gemini's maxOutputTokens, and the cap thedotmack#3620's cut-off warning names. Validated on the settings API and documented. - finishReason on ProviderQueryResult for OpenRouter and Gemini, a Gemini MAX_TOKENS warning for parity, and finishReason/truncated on the drop log line. The reason is scoped to one reply: processAgentResponse consumes and clears it. The invalid-output counter and the preserve policy stay as they are (C05). Dropped as superseded: the sliding window and its message/char caps (main's generation recycle and window-aware budget, thedotmack#3625), the attempt timeout and timeout:false (CLAUDE_MEM_LLM_TIMEOUT_MS), the per-provider CLAUDE_MEM_OPENROUTER_MAX_TOKENS (one shared setting), the duplicate history test (thedotmack#3619) and the counter change. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01697ztU3scSYRfW49cPr9Du
…put cap configurable (#3868) * fix(openrouter): bound the observer context and anchor the schema in a system message (#3606) The OpenAI-compatible path sent the entire per-session conversation history on every call with no system role. Against a bounded local context (Ollama, llama.cpp) that either fills the window (truncation → empty content) or scrolls the observation schema — stated once in the init prompt — out of view, the model degrades to bare prose. Both end in ResponseProcessor's invalid-parse branch, which confirmed and dropped the batch, and the write-only consecutiveInvalidOutputs counter (always reset to 0) hid how often this was happening. Root cause and fix: - context-window.ts (new): history[0] (the init/continuation prompt, which carries the full <observation> schema) is pinned as `system`; the rest is windowed by turn count and total chars, the final message is always kept, and leading assistant turns are dropped after trimming. - Four settings now declared and honored on the OpenRouter path: CLAUDE_MEM_OPENROUTER_MAX_TOKENS (was hard-coded 4096), _MAX_CONTEXT_MESSAGES, _MAX_CONTEXT_CHARS (0 = unbounded), _ATTEMPT_TIMEOUT_MS (was the hard-coded 30s withRetry default). A corrupt/negative value falls back to the shipped default; max_tokens and the timeout require >= 1, the two context caps keep 0 as "unbounded". - Bun's fetch has a built-in ~300s timeout that overrides withRetry's AbortSignal-based per-attempt timeout, so the request init now sets `timeout: false` explicitly (Bun-specific, not in the DOM RequestInit type — cast through Record<string, unknown>). - finish_reason is read and propagated (ProviderQueryResult.finishReason, session.lastFinishReason). The empty-content branch is now three-way: no choices/message at all is `malformed` (a different failure than a designed empty completion); `finish_reason === 'length'` is a real truncation (error); `=== 'stop'` is the prompt's documented empty-reply skip (info); anything else — content_filter, tool_calls, an absent reason — is neither, and is not a benign skip (warn). - ResponseProcessor.ts's generic non-XML fall-through branch (shared by every provider — Claude/SDK, Gemini, OpenRouter) now: increments consecutiveInvalidOutputs on any real dropped batch instead of leaving it write-only; treats `idle && (finishReason === null || 'stop')` as the only designed skip (`null` covers Claude/Gemini, which never populate lastFinishReason, so their behavior is unchanged); and logs at error (instead of warn) when the batch was truncated or the output was malformed XML. This sits in the fall-through only — the newer overflow/quota/auth rejection branches above it (#3800) are untouched. - OpenAICompatibleProvider.ts: threads finishReason through to session.lastFinishReason before each processAgentResponse call (init, observation, summary). The duplicate assistant-history push this fix originally also targeted was already removed upstream at #3619 (verified by reading OpenAICompatibleProvider.ts pre-change) — no source change needed for that part; tests/worker/openai-compatible-history.test.ts is added as a regression guard. Tests: tsc clean; tests/worker tests/sdk tests/shared = 882 pass, 0 fail (up from 838 pre-change, +44 new). Full `bun test`: 3053-3054 pass, 13 fail, 6 errors — identical failing-test set and tallies on a clean origin/main checkout at the same commit (be44b6c), confirmed by running both full suites side by side; none touch the files this change modifies. The 6 errors are workers/sync-hub's Cloudflare Workers test harness ("Cannot find package 'cloudflare:test'"), unrelated and pre-existing. Refs #3606, #3490 (max_tokens). Co-authored-by: Gwen Ives <[email protected]> Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MwVTb8AbAAQZjxsrewXc83 * fix(openrouter): validate numeric settings as complete integers; scope finish reason to the response that produced it Two review findings on the bounded-observer-context change (#3606): - `parseOpenRouterInt` used `parseInt`, which accepts a numeric prefix, so a setting like `CLAUDE_MEM_OPENROUTER_ATTEMPT_TIMEOUT_MS: "1e3"` resolved to 1 and every attempt timed out after one millisecond. It now accepts only a finite integral number or a full decimal-integer string (optional sign, surrounding whitespace trimmed) within [min, max]; anything else falls back to the documented default. `attemptTimeoutMs` additionally caps at the setTimeout ceiling (2^31-1 ms), which Node otherwise silently treats as ~1 ms. - `session.lastFinishReason` was set by the OpenAI-compatible provider and never cleared, so a later empty Claude reply in the same session could read a stale `length` and be counted as a truncation instead of the designed skip. `processAgentResponse` now consumes and clears it at the top of every call, making the value response-scoped regardless of which branch handles the reply. Tests: strict-integer cases (exponent, decimal, hex, sign, whitespace, empty, non-numeric, non-ASCII digits, overflow, ceiling) and the stale-finish-reason sequence (OpenRouter `length` followed by an empty Claude-path reply). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01RJEKSU5eVwnvBf4irjtP4E * fix(observer): anchor the schema in a system message, make the output cap configurable, name truncations Trimmed to the three pieces main still lacked, rebuilt on main's request path (buildOpenRouterRequestBody, the #4003 max_completion_tokens retry, the trial-expiry fallback clearing and the gateway classification stay as #4276 left them): - System anchor: a generation's framing prompt (marked by openObserverGeneration) goes out as the system message on OpenRouter and custom endpoints and as systemInstruction on Gemini. Its user-request block (or a fixed kickoff) stays the first user turn, so every request, the init one included, carries a user message and roles still alternate. Standalone requests (field condensation, the wrap-up) are unchanged. - CLAUDE_MEM_OBSERVER_MAX_OUTPUT_TOKENS (default 4096, 256 to 1,000,000): max_tokens in buildOpenRouterRequestBody, max_completion_tokens on the compatibility retry, Gemini's maxOutputTokens, and the cap #3620's cut-off warning names. Validated on the settings API and documented. - finishReason on ProviderQueryResult for OpenRouter and Gemini, a Gemini MAX_TOKENS warning for parity, and finishReason/truncated on the drop log line. The reason is scoped to one reply: processAgentResponse consumes and clears it. The invalid-output counter and the preserve policy stay as they are (C05). Dropped as superseded: the sliding window and its message/char caps (main's generation recycle and window-aware budget, #3625), the attempt timeout and timeout:false (CLAUDE_MEM_LLM_TIMEOUT_MS), the per-provider CLAUDE_MEM_OPENROUTER_MAX_TOKENS (one shared setting), the duplicate history test (#3619) and the counter change. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01697ztU3scSYRfW49cPr9Du --------- Co-authored-by: Gwen Ives <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Alex Newman <[email protected]>
Status
The source fix from this PR is on
main. 7a7ada5 ("bound observer conversation growth…", onmainsince 2026-08-31) removed the provider-sideconversationHistory.pushcalls, and its message cites this PR. This branch is now rebased onto currentmainand cut down to the test file. It no longer touchessrc/.The one assistant push left in
OpenAICompatibleProvideris inhandleInitResponse, and it is not a double append. Since a00ce61 the init reply no longer goes throughprocessAgentResponse, so the provider records it itself, once.processAgentResponserecords the observation and summary replies.Why the test is still worth having
mainpins the observation path:tests/worker/openai-compatible-init-response.test.tsasserts that roles alternate after init plus one observation.Two paths have no test:
processSummaryMessagechanges no result in any of the 21 test files that drive the OpenAI-compatible providers orprocessAgentResponse.forwardEmptyMessageResponse = false), an empty observation reply must add no assistant turn. Pushing one there also changes no result in those 21 files.This file pins both:
records each reply exactly once across the init, observation and summary turns: three queries, roles alternateuser, assistantthree times, and the assistant turns are the three replies in order.adds no assistant turn for an empty observation reply when empty replies are not forwarded: roles areuser, assistant, user.Verification
Each mutation below was applied to
main'sOpenAICompatibleProvider.ts. The 21 upstream files were compared with an unmutated run by test name, because this Windows machine has pre-existing failures in that set.processObservationMessageinit-responsefailsprocessSummaryMessageThe new file passes (2 pass, 0 fail). It also type-checks clean against the root
tsconfig.json(strict) when included explicitly;tests/is excluded from the regulartypecheckscript.Refs #3499, #3497. Tracked under #3606 (plan-18).