Repository navigation
Description
[plan-18] Observer Response Pipeline — classify every output, never confirm a batch on failure, bound every history
Tracker: #3606 · Design doc:
plans/18-observer-response-pipeline.md· Series: plan-14 #3602 · plan-15 #3603 · plan-16 #3604 · plan-17 #3605 · plan-18 #3606 · plan-19 #3607 · plan-20 #3608 · plan-21 #3609 · plan-22 #3610 · plan-23 #3611
Defect
ResponseProcessor has effectively one bucket for "the model did not return well-formed <observation>/<summary>/<skip_summary/> XML", and that bucket confirms the claimed batch (confirmClaimedMessages()) — so every non-XML reply is silent data loss, and the pipeline's countermeasures are either dead or self-defeating:
- Skip contract mismatch:
modes/*.jsonskip_guidancesays "return an empty response"; the parser accepts only<skip_summary reason=…/>; empty →idle, "no observations to record" →prose. Both were counted towardconsecutiveInvalidOutputs >= 3→ "SDK session poisoned — killing and respawning" (hundreds of respawns/day,conversationHistorywiped, batches never redelivered). 13.11.0 removed the escalation but left the counter assigned0at five sites and never incremented, and left the silent drop. - Auth failures (
Failed to authenticate 403/401,Not logged in), model errors (There's an issue with the selected model), transport errors (Connection closed mid-response), context overflow (Prompt is too longarriving as assistant text, not a throw), XML schema drift (<kind>/<detail>instead of<type>/<title>), and closed<observation>blocks with prose but no fields are all classifiedprose/xmland dropped; only quota and (recently) auth have preserve-work branches; the malformed turn is pushed intoconversationHistorybefore validation so drift self-reinforces for the rest of the session. - Empty requests: the generator auto-starts with
queueDepth=0, sends an empty INIT turn, gets""back, and bins already-claimed messages;SessionManagerdiscards the storedmemory_session_idwhenever the session is absent from the in-memory map ("Issue [Bug] Stale session fix incomplete: lastPromptNumber loaded from DB bypasses resume guard #817") even when the worker never restarted, so a 180 s idle eviction sends the next batch back through INIT where the model's reply consumes it. - No token budget anywhere:
session.conversationHistoryis push-only (ClaudeProvider.ts:490/528/552,OpenAICompatibleProvider.ts:105/215/265,ResponseProcessor.ts:294) and fix(providers): remove client-side context truncation ("second system") #3096 removed the OpenAI-compatibletruncateHistory(); the OpenRouter copy that remains returns[]when a single message exceedsMAX_TOKENS(400 from the API, observation marked failed); the OpenAI-compatible provider appends each assistant reply twice;max_tokens: 4096is hardcoded andfinish_reason=lengthis never logged (76 % of responses truncated, trailing block lost, degenerate repetition hidden bycontent_hash); base64/image payloads and unbounded per-field tool output go straight into the prompt; the paused-for-auth buffer (buffers=new Map, removed only byconfirm()) grows to GBs; ~10× token amplification is billed to the user's subscription at DEBUG log level. - Output quality is unenforced: the
typeenum is logged as invalid then stored verbatim (or defaulted tobugfix); write-time dedup is byte-identicalcontent_hashonly, so paraphrased restatements flood the recall window; reasoning models' legitimately emptycontentwithreasoning_contentis treated as an init failure; provider settings are validated for all providers regardless of the active one so a retired Gemini id blocks every settings save.
The fix is one explicit classifier with one policy table: every output class maps to {preserve batch | confirm batch | abort session with reason}, and every history/buffer/response has a budget.
Children
- v13.4.0 (Linux): observer still poison-loops on prose-empty responses despite plan-11 recovery — root cause (#2749 expected behavior) not addressed #2817 — v13.4.0: observer still poison-loops on prose-empty responses despite plan-11 recovery
- Observer silently drops observations on "Prompt is too long" — strip image/base64 payloads before sending to the model #2866 — Observer silently drops observations on "Prompt is too long" — strip image/base64 payloads
- Triggering flood queue bug, with over 100 nested observer packages appearing in a single message #2906 — Flood queue: over 100 nested observer packages appearing in a single message
- 用cc switch 切换智谱模型后 只会记录问题不会记录回答 #2915 — After switching Claude Code to Zhipu GLM via cc switch, only prompts are recorded, never answers (model error classified as prose)
- Generator poison loop: prompt says 'return empty response' to skip, but parser only accepts <skip_summary/> — empty/prose replies poison the session #2935 — Generator poison loop: prompt says 'return empty response' but parser only accepts <skip_summary/>
- SDK session poison-loop: generator emits prose "empty observation" instead of <skip_summary/>, triggering repeated kill/respawn #2955 — SDK session poison-loop: generator emits prose "empty observation" instead of <skip_summary/>
- Claude provider (SDKAgent) has no context bound — observer overflows with "Prompt is too long" and saves zero memory on long sessions #2956 — Claude provider (SDKAgent) has no context bound — observer overflows and saves zero memory on long sessions
- tracking: Claude provider observer overflows context window — no bound on history size, zero memory saved on long sessions #2958 — tracking: Claude provider observer overflows context window — no bound on history size
- tracking: observer poison loop — skip-prose counted as invalid, triggering endless kill/respawn cycle #2960 — tracking: observer poison loop — skip-prose counted as invalid
- Feature request: support disabling reasoning for OpenRouter / OpenAI-compatible providers #2995 — Support disabling reasoning for OpenRouter / OpenAI-compatible providers (reasoning eats
max_tokens) - Near-duplicate observations accumulate: content_hash only dedups byte-identical rows (fuzzy write-time dedup?) #3038 — Near-duplicate observations accumulate: content_hash only dedups byte-identical rows
- Windows reliability: poison-loop (#3032) still present in 13.8.1 + worker-lifecycle issues (FD inheritance, stale daemon) #3083 — Windows reliability: poison-loop still present in 13.8.1 (lifecycle half → plan-15)
- Capture-side dedup: title-level boilerplate observations bypass content_hash and flood the recall window #3163 — Capture-side dedup: title-level boilerplate observations bypass content_hash and flood the recall window
- Surface BYOK / custom OpenAI-compatible endpoint (CLAUDE_MEM_OPENROUTER_BASE_URL) in the CLI installer and Settings UI #3188 — Surface BYOK / custom OpenAI-compatible endpoint in the CLI installer and Settings UI
- Observer poisons on correct-empty batches: polite 'ready/waiting' prose counts as 3-strikes invalid output #3193 — Observer poisons on correct-empty batches: polite 'ready/waiting' prose counts as 3-strikes invalid
- Data loss: closed <observation> blocks with unstructured prose are permanently discarded (no salvage, no retry) — ~94 substantive batches/day #3351 — Data loss: closed blocks with unstructured prose are permanently discarded (no salvage, no retry)
- Observer history is unbounded again since #3096: 1M-token single calls on the OpenAI-compatible provider path #3443 — Observer history is unbounded again since fix(providers): remove client-side context truncation ("second system") #3096: 1M-token single calls on the OpenAI-compatible provider path
- Observer discards 63% of turns — 170/day empty (idle) responses, plus dead consecutiveInvalidOutputs counter #3454 — Observer discards 63% of turns — 170/day empty (idle) responses, plus dead consecutiveInvalidOutputs counter
- Viewer settings can never be saved when a retired Gemini model is left in settings.json — and the UI hides the reason #3457 — Viewer settings can never be saved when a retired Gemini model is left in settings.json — and the UI hides the reason
- Observer XML schema drift causes permanent, silent observation loss for the rest of a session #3461 — Observer XML schema drift causes permanent, silent observation loss for the rest of a session
max_tokenshardcoded to 4096 silently truncates 76% of observation responses #3490 —max_tokenshardcoded to 4096 silently truncates 76% of observation responses- OpenRouter truncateHistory returns an empty array when one message exceeds MAX_TOKENS, causing a 400 and a permanently lost observation #3491 — OpenRouter truncateHistory returns an empty array when one message exceeds MAX_TOKENS → 400 and a lost observation
- Observer never leaves the INIT prompt: idle eviction discards memory_session_id on a worker that never restarted — every observation batch dropped (observations=0, Windows) #3492 — Observer never leaves the INIT prompt: idle eviction discards memory_session_id on a worker that never restarted
- Claude-mem logs dead-ends and incorrect memories #3495 — Claude-mem logs dead-ends and incorrect memories
- Observer conversationHistory is never trimmed — ~10x token amplification billed silently to the user's subscription #3497 — Observer conversationHistory is never trimmed — ~10x token amplification billed silently
- OpenAI-compatible provider appends the same assistant response to conversationHistory twice #3499 — OpenAI-compatible provider appends the same assistant response to conversationHistory twice
- Feature request: filter Bash capture by command pattern (SKIP_TOOLS is too coarse) #3560 — Filter Bash capture by command pattern (SKIP_TOOLS is too coarse) — capture gate as token budget
- OpenRouter init fails on empty content from reasoning models (DeepSeek V4 Flash) #3569 — OpenRouter init fails on empty content from reasoning models (DeepSeek V4 Flash)
- Auth failure → buffered observations never drain: single worker heap reaches 5.6 GB (~47 MB/min), 0 drains in 50 h of logs #3588 — Auth failure → buffered observations never drain: single worker heap reaches 5.6 GB
- Observer context overflow is classified as prose and silently discards the batch #3589 — Observer context overflow is classified as prose and silently discards the batch
- Observation
typeenum is unenforced: parser logsInvalid observation typethen stores it verbatim; missing<type>still defaults tobugfix#3592 — Observationtypeenum is unenforced: parser stores invalid types verbatim; missing<type>defaults tobugfix
Related PRs to evaluate/rebase: #3460 (transport-error classifier), #2957 (SDKAgent sliding window), #2884, #2943, #2927, #2857, #2905, #1775 (prose salvage), #3352 (prompt: check prior output), #3207 (auth-prose preserve batch), #3283 (retired model ids).
Fix sequence
- One classifier, one policy table.
sdk/output-classifier.tsreturns a closed enum —valid,skip,empty_request,idle,auth,quota,model_error,transport,overflow,xml_drift,prose_salvageable,prose— andResponseProcessordispatches through a table: preserve batch (resetProcessingToPending) forempty_request|idle|auth|quota|transport|overflow|xml_drift, confirm forvalid|skip, salvage-then-confirm forprose_salvageable(title = first 100 chars, narrative = text, WARN), confirm-with-ERROR for trueprose. Nothing reachesconversationHistorybefore it passes validation;xml_driftpops the bad turn and re-sends the schema.consecutiveInvalidOutputseither counts and aborts with a typedabortReason(drift:*,overflow:*) after 3 or is deleted — not assigned0and logged. - Skip is a sentinel, not silence. Every mode's
skip_guidanceemits<skip_summary reason="…"/>(and<skip_observation/>for observation turns); the parser accepts both; the generator never sends a turn withqueueDepth=0; the INIT prompt is only sent whenbootIdchanged (plan-15 owner record) — a storedmemory_session_idis resumed on the same worker instance instead of being discarded on idle eviction. - Token budget as a first-class object.
ObserverBudget {maxHistoryTokens (fraction of model context), maxFieldChars, maxToolOutputChars, maxOutputTokens}shared by all providers; history is trimmed (compact older half → summary) never emptied (truncateHistorykeeps ≥ 1 message and truncates content instead); the duplicate append is removed; base64/image blobs replaced with[image omitted: type, N KB]at ingest;max_tokensconfigurable and coupled toCLAUDE_MEM_API_TIMEOUT_MS;finish_reason=lengthand repeatedobsIdslogged WARN; cumulative input/output tokens logged INFO and exposed on/api/health+ statusline; the auth-paused buffer is capped with spill-to-disk and stops accepting past the cap. - Provider adapter parity. One
OpenAICompatibleProviderconfig surface: base URL,max_tokens,reasoning: {effort}(only when advertised), empty-content-with-reasoning_contenttreated as no-op,SettingsRoutesvalidates only the active provider's fields and normalizes retired ids at load; the viewer save handler surfaces the JSON error body; installer/Settings UI expose the base URL. - Output quality gates. Unknown
type→ coerced to a neutral type with the raw value inmetadata(enum restated in the per-tool-use template); missing title rejected; write-time (title, project) normalized dedup withoccurrence_count(opt-in fuzzy tier review-only);CLAUDE_MEM_SKIP_BASH_PATTERNSat the same gate asSKIP_TOOLS; a "check your prior output" line in the observation prompt.
Test matrix
| Provider | Output | Required behavior |
|---|---|---|
| claude (SDK), openrouter, gemini, openai-compatible | "" on empty request |
class empty_request; batch preserved; no INIT churn; no log spam |
| all | <skip_summary reason=…/> / <skip_observation/> |
confirmed; zero ERROR lines |
| all | "No observations to record" prose | class idle; batch preserved; counter untouched |
| all | 401/403/Not logged in/OAuth session expired text |
class auth; batch preserved; session paused with ERROR + health flag |
| claude | Prompt is too long as text |
class overflow; batch preserved; history reset; retried once with budget halved |
| all | <observation><kind>… drift |
class xml_drift; turn popped; schema resent; abort after 3 with drift:observer_schema |
| all | closed <observation> with 1,000 chars prose |
salvaged as narrative; WARN |
| all | type=verification (not in enum) |
stored as neutral type + metadata; never bugfix by default |
| openrouter | one message > MAX_TOKENS | request still has ≥ 1 message; content truncated; no 400 |
| openai-compatible | 200 turns | history tokens ≤ budget; each assistant reply present exactly once |
| openrouter/gemini | finish_reason=length |
WARN logged; partial block salvaged; no blind retry |
| all | 50 h paused for auth | buffer ≤ cap; heap flat; observations drain after /login |
| all | model returns paraphrase of previous title | dedup increments occurrence_count, no new row |
| claude, gemini | settings save with retired gemini id while provider=claude | 200; id normalized |
The matrix lives in CI (tests/sdk/output-classifier.test.ts + tests/services/worker/ResponseProcessor.test.ts with recorded provider transcripts). A regression must fail CI before a user can file.
Out of scope
How the observer subprocess is spawned, authenticated, isolated, and where its transcripts live (plan-19). Which project an observation is filed under (plan-20). Queue/session state persistence and retry (plan-21).
Activity
thedotmack commentedon Aug 17, 2026
PR consolidation pass — open pull requests routed to plan-18
Every open PR in the repo (201 as of 2026-08-16) was read in full — body, review threads, files changed, and diffed against
origin/main— and clustered by the root cause it targets. These are the PRs whose fix belongs to [plan-18] Observer Response Pipeline. Sequencing lives here; the design lives inplans/18-*.mdon theoh-my-issuesbranch (#3617).11 bundle candidate · 6 standalone · 10 needs rework · 1 superseded
Bundle candidates — sound fixes to fold into this plan's bundle PR (with author credit), keyed to the fix-sequence step they implement:
↳ step 3 (token budget as first-class object; proactive cap for the SDK path)
↳ step 5 adopts exactly this shape (opt-in normalized-title dedup with occurrence_count; fuzzy tier review-only)
↳ step 3 (history trimmed, never emptied; compact older payloads)
↳ step 5 adopts exactly this ("check your prior output" line in the observation prompt)
↳ step 1 (
prose_salvageable: first line → title, rest → narrative)↳ step 1 (
transportpreserves the batch); already bundled in fix: bundle 8 community root-cause PRs (process ownership, port reclaim, hook prelude, chroma backfill, observer transport, opencode events) #3614; P2: anchor keywords↳ step 1 (
xml_driftpops the bad turn, bounded retry, typed abort reason)↳ step 4 (provider adapter parity: history normalization for OpenRouter); one open thread on leading-assistant turn
↳ step 1 (thinking-only/tool_use-only frames are not
idle; text-block gating)↳ step 5 explicitly (
CLAUDE_MEM_SKIP_BASH_PATTERNSat the same gate asSKIP_TOOLS); P1: documented regex example is over-escaped↳ step 4 exactly (empty content vs missing content; empty init response forwarded)
Standalone / merge-now — small, self-contained; can land ahead of the bundle:
↳ step 4 (OpenAI-compatible config surface: extra request-body fields)
↳ step 5 (missing title rejected at write time)
↳ Stop-hook empty
lastAssistantMessagetruthiness fix; drop the committed .cjs and rebase↳ step 4 (viewer save handler surfaces the JSON error body)
↳ step 3 (auth-paused buffer capped; per-attempt history checkpoint) — independently reproduced 40 GB wedge on 13.15.0
↳ step 5 (unknown/missing
typehandling pinned by tests) — test-onlyNeeds rework — right diagnosis, blocked on the item noted (address it against the plan doc, or it gets absorbed into the bundle):
↳ step 4 (
reasoning:{effort}only when advertised); P1: default 'none' injects reasoning into every request↳ auth-classifier half landed via Auth-failure prose ("Failed to authenticate. API Error: 401") still drops the claimed batch — only quota-limit prose is preserved for retry #3207; remaining:
/api/healthai.lastInteraction wiring + disabled-plugin exit log↳ step 1 (
model_errorfrom body-level errors in 200 envelopes); rebase onto fix(worker): carry the gateway error envelope end-to-end — stop retrying quota, log once, tell the user what to do #3601's classifier rework↳ step 3 (budget object shared by all providers) — apply at OpenAICompatibleProvider level, not OpenRouter-only; drop committed bundle
↳ step 1/2 (
empty_requestpreserves batch; generator never sends with queueDepth=0); P1: whitespace-only content bypasses guard; server half → plan-24↳ step 4 (installer/Settings UI expose the base URL); P1: query/fragment base URLs; installer file rewritten by Installer: 7-day Pro trial funnel (pitch → email → pairing poll → auto-setup) #3524
↳ step 1/2 (
empty_request/idlepreserve; generator never sends with queueDepth=0); CI failing on all jobs at head↳ step 3 (cumulative tokens logged INFO, exposed on health; real usage counts, cache-read tokens); conflicting
↳ step 3 (single push site, compaction re-seed, context-window resolution) — large; conflicting; reconcile with feat: bound observer conversation history for OpenAI-compatible providers #3151
↳ step 1 (
model_errorfingerprints: 400 vs 404, keep body, hostname label); rebase onto fix(worker): carry the gateway error envelope end-to-end — stop retrying quota, log once, tell the user what to do #3601; P1: predicate inconsistencySuperseded — closed in this pass (fix already on main or landed via another PR):
↳ landed as 3bf1914 (fix(worker): carry the gateway error envelope end-to-end — stop retrying quota, log once, tell the user what to do #3601 gateway error envelope end-to-end; classified errors log at debug, quota not retried)
Next step for this plan: cut the bundle PR from the bundle candidates in fix-sequence order (Mode 3 of
oh-my-issues), listing each folded PR withCloses #N/co-author credit; merge the standalone PRs independently once CI is green; leave needs-rework PRs open until their blocker is addressed or the bundle covers them.Generated by the
/oh-my-issuesPR pass; verdicts are the pass's reading of the review threads, not a maintainer decision.ntdatt812 commentedon Aug 18, 2026
Reading the consolidation pass — 201 PRs read in full, diffed against
origin/main, and clustered by root cause. That's the part of this repo that gets harder every time someone like me sends another PR.I did a narrow version of it for plan-18 before sending anything. The children listed here are closed as tracked, not as fixed, so I checked which defects are still live at HEAD instead of assuming. Two were:
OpenAICompatibleProviderrecords each assistant reply intoconversationHistory, thenprocessAgentResponserecords it again (ResponseProcessor.ts:295). Nothing dedupes the history before it becomes themessagesarray, so the wire conversation isuser, assistant, assistant, user, …and the assistant half of every later request is double-billed. → test(worker): pin one assistant turn per observer reply, summary included #3619max_tokenshardcoded to 4096 silently truncates 76% of observation responses #3490 —finish_reasonis declared on the response type and never read. A reply truncated at the hardcoded 4096 cap is logged as the model failing to return XML, and the batch is confirmed and dropped. → fix(openrouter): name the max_tokens cap when it cuts a reply off #3620Both ship with tests that fail on
main. On #3620 the reviewer caught two real defects in my own diagnostic — a capped summary reply reported as a truncated observation, and missing upstream usage rendered asoutputTokens: 0— and I pushed fixes and tests for both the same day.If it's useful, I'd like to do that verification pass for the other trackers — one plan issue at a time: check each listed child against HEAD, mark which are already fixed, which are still live, and which open PRs actually cover them. I'd post it as a comment for you to accept or discard.
Triage only — no merge rights, no write access, nothing lands without you. If applying a
needs-rework/supersededlabel would save you a step I'd use it, but a comment works fine.Easy to say no; the
oh-my-issuespass in #3617 may already cover this. If it isn't covered — which tracker is most in the way right now?rolldav commentedon Aug 19, 2026
Additional 13.15.2 evidence: interactive and print-mode capture diverge after non-XML observer replies
Environment
b0194042d53e618f1eb85a2bdf1f61456f3f49damaininspected atfae697a45d107aae567d605916391ab64d8ecae1Expected
With the same small
Readevent successfully delivered throughPostToolUse, interactive and print-mode invocations should have the same documented capture semantics, or the unsupported lifecycle difference should be documented.Actual, anonymized local evidence
In one controlled interactive session in a fresh directory:
Readon a tiny benign file.PostToolUse: Readand queued an observation batch.No observations to record at this time.In multiple earlier print-mode/one-shot probes, a session was created and
PostToolUse: Readwas logged, but the observer returned either refusal-style prose or an empty/non-XML response; those anonymized sessions persisted zero observations.Minimal comparison
ORACLE.md.claudesession, explicitly request aRead, then exit normally.claude -p ...).PostToolUse, queue state, observer response and Stop/end-of-session with the persisted observation row.The current prompt permits an empty response and says non-XML output is discarded; the current processor then confirms claimed messages after ordinary non-XML output:
claude-mem/src/sdk/prompts.ts
Lines 143 to 154 in fae697a
claude-mem/src/services/worker/agents/ResponseProcessor.ts
Lines 348 to 366 in fae697a
Question / limits
Could maintainers confirm whether print mode is expected to run the same Stop/end-of-session drain semantics as an interactive session, and whether the later XML observation can belong to the earlier claimed batch after summarize is queued?
This sample shows correlation, not a proven concurrency root cause. It does not prove a general Claude Code 2.1.235 incompatibility, a system-prompt cause, or that every non-XML reply should be requeued. Paths, prompts, contents, UUIDs, database identifiers, PIDs and corpus counts are redacted. No downgrade, cache edit, wrapper, database merge or configuration change was used.
rodboev commentedon Aug 20, 2026
Opus 5, is that you?
raka08 commentedon Aug 20, 2026
Independent confirmation on 13.15.2 — the drift is ongoing 4 days after consolidation
Corroborating #3592 (consolidated here) from a separate install. Same symptom, newer version, and the rate has not slowed.
Environment: claude-mem 13.15.2, macOS 15.6 (arm64), plugin install,
CLAUDE_MEM_RUNTIME=worker, stockcodemode (the 9-entry enum), providerclaude/claude-haiku-4-5-20251001. Store: 9,652 observations.Current state
metadata.original_typeThe last row matters: this install has never been remapped, so all 302 are original drift rather than leftovers from a cleanup.
Drift after #3592 was closed (2026-08-16)
286 of the 302 landed on or after the consolidation date.
Log counts over the last 3 days
Both branches described in #3592 are firing on this install — the missing-
<type>branch far more often than the out-of-enum branch.What the emitted types actually look like
Most are not near-miss synonyms such as
progressordebug. They are content-derived slugs, which suggests the observer is treating<type>as a free-text label rather than a closed enum:Downstream effect worth noting
The session-start context header renders a fixed legend for the 9 enum types. Rows carrying one of the 160 drifted types render without a symbol, and any grouping or filtering on
typesilently misses them.The missing-
<type>branch also inflatesbugfixspecifically, since that is the enum's first entry and therefore the default: 703 rows currently carry it, while 355 of the 3-day error lines come from that branch alone.Happy to supply the full 160-value list or per-day breakdowns if that would help sequencing.
ntdatt812 commentedon Aug 20, 2026
@raka08's two log branches are exactly the two branches #3593 pins, so the field data and the test coverage line up one-to-one:
Observation missing type field, using "bugfix"falls back to the first declared type when <type> is absentInvalid observation type: …, preserving emitted typepreserves a type that is outside the enumOne thing worth making explicit, because the log line reads as if it were a decision: the
bugfixin those 355 lines is not a chosen default. It isobservation_types[0]— whatever happens to be first in the active mode's enum. In the bundledcodemode that isbugfix, so every untyped observation is filed as a bug fix by list order rather than by classification. Reordering the enum silently reclassifies all of them, which is why #3593 has a third case that reorders the enum and asserts the fallback follows it torefactor.On this install that is 355 log events in three days landing in one bucket that nothing chose.
What #3593 does not do, to be clear about scope: it is characterization only. It does not enforce the enum and does not stop the drift — it pins today's behaviour so that whichever way enforcement eventually lands, it shows up as an intentional test edit rather than as silent movement. @raka08's numbers are the argument for doing that enforcement; the tests are only the seatbelt for when it happens.
The observation that most drifted types are content-derived slugs rather than near-miss synonyms —
— is the more interesting half, and it is upstream of the parser. Those are not a parser failing to validate a plausible type; they are the observer treating
<type>as a free-text label. 160 distinct invalid types out of 302 rows is close to one new type per two rows, which is what a free-text field looks like, not a mistyped enum. Fixing that is a prompt/contract question rather than a parse-site one, and #3593 deliberately does not touch it.Status on my side: #3593 is not waiting on review. Its workflow runs terminate at
conclusion=action_required— the first-time-contributor approval gate — so CI has never actually run on it. Detail and the affected-PR list are in my comment on the PR. Locally on the current rebased head:npm run typecheckexit 0,bun test ./tests/sdk/parser.test.ts24 pass / 0 fail.raka08 commentedon Aug 22, 2026
Field data on the "classify every output / never confirm a batch on failure" half of this plan — 13.15.3, 6,652 discarded batches in 13 days
Following up my earlier comment on the type-drift half. This install now runs 13.15.3 (macOS 26.6.2, arm64, plugin install,
CLAUDE_MEM_RUNTIME=worker, providerclaudebehind anANTHROPIC_BASE_URLgateway). Store: 11,198 observations. This is the discard side, which #3351 and #3461 were both consolidated here for.Every discard classification, retained logs 2026-08-10 → 2026-08-22
outputClassidleprosexmlAll 6,652 log the same terminal line —
SDK returned non-XML <class> response — ignoring queued batch— and the batch is dropped with no retry, no dead-letter, no salvage.The
xmlclass is the interesting oneoutputClass=xmlmeans the classifier already decided the payload is XML, and the strict parser then rejected it anyway. Breaking those 494 down by the shape of the payload the observer actually returned:<observation>+ prose body, no<type>child```xmlmarkdown fence around otherwise valid XML)<observation> <type>…</type> <title>…— the expected child form<observation type="discovery">— attribute form instead of child elements<observation></observation>— empty element<summary>…484 of the 494 carried substantive content (the log's own
(+N chars)marker), i.e. the discarded payloads were not empty responses — they were written observations that were thrown away.Two shapes deserve calling out specifically:
<observation> <type>bugfix</type> <title>Confirmed: no darwin-arm64 tree-sitter prebuild and no wasm fallback</title> <subtitle>…(+1254 chars). A payload that opens correctly and carries 1.3 KB of body is not a classification problem — the strict parse is failing on something downstream of the opening element. A few others in this bucket are genuinely truncated mid-<title>with no(+N chars)marker, so the bucket is mixed, but not all of it is truncation.preview=```xml <observation> <event>…</event> <timestamp>…</timestamp> …. Nothing is wrong with that XML except the three backticks in front of it.Per-day volume, and what it costs
xml-class discardsOn 2026-08-18 the
xml-class discards alone were ~31% of that day's written observations — and that is before the 1,982proseand 4,176idlediscards.consecutiveInvalidOutputsis still dead on 13.15.3Every one of the 6,652 discard lines carries
consecutiveInvalidOutputs=0:Not one increment across 13 days and 6,652 invalid outputs. Whatever escalation this plan hangs off that counter (#3454's finding, consolidated here) will never fire while the counter is reset on every pass. Worth treating as a precondition for the rest of the sequence rather than a separate step — the "never confirm a batch on failure" work is unverifiable in the field until the counter that records failure actually counts.
Smallest changes that would recover the most, from this data set
```/```xmlfences before parsing — free, and it is currently a total-loss condition.<observation type="…">as equivalent to<observation><type>…</type>— 38 batches here.<observation>with a prose body by filing it under the mode's fallback type and recording the degradation, rather than discarding — 242 batches here. (Note the interaction with the type-drift half of this plan:observation_types[0]as the fallback is what @ntdatt812 flagged above, so the salvage path needs an explicitunclassifiedtype rather than list order.)Happy to pull any further slices from this install's logs or DB if a specific shape would help the fix sequencing.
ntdatt812 commentedon Aug 22, 2026
@raka08 — on point 3, the interaction is worse than list order, and the reason is one line further down the pipeline than either of us was looking.
Where the fallback comes from (
src/sdk/parser.ts:106-116, HEADe2d1df5):Four distinct enums ship, and
bugfixis only the first element of one of them:observation_types[0]— what an untyped observation becomescode--*, which deep-merge the parent's enum)bugfixentitycase-holdingpump-detectedSo in the legal mode an untyped observation is filed as a case holding, and in the crypto mode as a detected pump. Same line, no classification anywhere in it.
The part that changes the fix. The two branches look symmetric — one replaces the type, the other preserves it — but preserving an out-of-enum type is not a safe pass-through. The column has no constraint (
type TEXT NOT NULL,SessionStore.ts:957), so the row is written; then the context query filters on the active mode's enum:An observation whose type is not in that set is stored and then never injected. Your 73× + 11×
preserving emitted typelines are not observations that survived with an odd label — they are observations that are in the database and invisible to every context load.Which is the constraint on
unclassified. Introducing it as the salvage type for those 242<observation>-with-prose-body payloads recovers them from the discard log into exactly that invisible bucket, unlessunclassifiedis declared in each mode'sobservation_types. Declaring it is not free either — the same list drives the type legend inAgentFormatter/HumanFormatterand the<type>[ a | b | c ]line the observer prompt is built from (src/sdk/prompts.ts:28), so adding it also offers it to the model as a choice. Worth deciding deliberately rather than as a side effect: a salvage-only id the prompt does not advertise needs the enum split into "accepted on read" and "offered on write", which is one list today.For scope, since it sits next to this: #3593 is characterization only. It pins the current behaviour of both branches — including a case that reorders the enum and asserts the fallback follows it to
refactor— so that a change here is visible as a diff instead of arriving silently. It does not enforce the enum and does not fix the invisibility above; that belongs to whatever step of this plan takes the salvage path.raka08 commentedon Aug 23, 2026
The 538 discarded batches, classified by output shape — and one of them is a proven two-regex asymmetry
Following up my 13-day batch-discard comment above with the classification this plan's first bullet ("classify every output") actually needs. Same install, 13.15.3, macOS 25.6.0 (arm64), plugin install,
CLAUDE_MEM_RUNTIME=worker, modecode, retained logs 2026-08-14 → 2026-08-23.I took every timestamp-anchored
SDK returned non-XML xml response — ignoring queued batchline (n=538) and bucketed it by the shape of its ownpreview=field, then cross-referenced whether the parser logged any per-observation diagnostic (Skipping empty observation) in the preceding lines — i.e. whetherparseObservationsextracted a block at all.<observation><what_happened>(echo of the input wrapper)<observation><type>…— the canonical child form<observation type="…">— type as an attribute```xmlmarkdown fence<observation>other/prose bodyOnly 11 of 538 (2.0%) were correctly discarded. The other 98% carried substantive observation content — the
(+N chars)markers on these lines run to+3748on a single batch.1. The attribute form is a detector/extractor asymmetry — 89/89, no diagnostics
This one is not a model-behaviour problem, it is two regexes in the shipped bundle disagreeing about what an observation tag looks like.
Detection (
validateResponse) accepts attributes, because of\b:Extraction (
parseObservations) requires the bare tag, no attributes:So
<observation type="discovery">…</observation>is classifiedoutputClass=xml, routed intoparseObservations, matches nothing, returns[], and the batch is dropped through theo.length === 0 → {valid:false}arm. NoSkipping empty observation, noObservation missing type field— nothing is logged about the observation at all, which is exactly why this shape has never shown up in the counters anyone has been quoting. The 89/89 correlation in the table is the whole bucket.Sample preview, verbatim from the log:
Relaxing the extractor to
/<observation\b[^>]*>([\s\S]*?)<\/observation>/gand readingtypefrom the attribute when the child element is absent recovers all 89 — and, more to the point, makes the two regexes agree, which is what stops this bucket from silently reappearing.The same asymmetry explains why
parseObservationsreturning[]is indistinguishable from "the model said nothing useful": both land on{valid:false}. A one-linelogger.warn('PARSER', 'observation tag detected but no block extracted', {…})on that arm would have surfaced 223 of these 538 batches on day one.2.
<what_happened>is the observer echoing the prompt's own input vocabularyThe largest bucket is not invented drift. The tool event handed to the observer is wrapped like this (same bundle):
The observer is being shown
<what_happened>as the schema for its input and then asked to emit<type>/<title>/<narrative>for its output. 120 batches came back in the input's vocabulary. Same for the smaller<event>bucket (11 across the untyped set), which pairs with<occurred_at>.This is worth separating from the
<category>/<phase>shapes (52 combined), which are free invention. The input-echo half is fixable in the prompt rather than in the salvage path — distinct tag names for the input wrapper, or the input rendered as something that is not XML at all.For the
unclassifiedquestion you raised: 120 of these are not "unclassifiable", they are misfiled.<what_happened>maps tonarrativeand<observed_from_primary_session>'s<outcome>tofactsmechanically, no new enum id required.3. The fence stripper is anchored to the whole response
76 batches (14.1%) were fenced.
stripFenceis:^…$over the entire response means one leading sentence before the fence, or one trailing word after it, and the fence survives into the detector — where the literal```characters sit inside whatparseObservationsthen tries to match. Matching the first fenced block anywhere in the response rather than requiring the response to be one fenced block covers all 76.Why this is 98% and not 2%
The three shapes above (attribute form, input echo, fence) are 285 of 538 batches — 53% — and none of the three is the model failing to produce a usable observation. All three are the pipeline failing to accept one it already produced. That seems directly relevant to how this plan sequences "classify every output" against "never confirm a batch on failure": for these 285, classification is not the hard part, and the salvage mapping is mechanical.
Happy to pull the per-shape breakdown per day, or the raw previews for any single bucket, if that is useful for the test matrix in #3593.
ntdatt812 commentedon Aug 24, 2026
@raka08 — I checked all three against the TypeScript source rather than the shipped bundle, since a fix lands there. Two hold exactly. The third does not, and your own table is what shows it.
1. The attribute asymmetry is real, and it is two lines apart.
src/sdk/parser.ts:89/89 with no diagnostics follows directly from that. The
return { valid: false }on line 73 is the only arm in the function that discards a batch without logging why, which is why the bucket is invisible.2.
<what_happened>is the input vocabulary —src/sdk/prompts.ts:144builds the wrapper you quoted, andobservationSkeletonat:24-34asks for<type>/<title>/<subtitle>in the same prompt. Agreed that this half belongs in the prompt, not the salvage path.3. The fence one I do not think is the fix, and "first fence anywhere" would regress a case the anchoring exists for.
The anchoring is deliberate —
parser.ts:8-9: "content that contains internal fenced examples or surrounding prose (CodeRabbit review on PR #2282)". I ran both regexes over four shapes:Two things fall out.
C is the regression. With an outer fence and an inner
```jsexample, matching the first fence anywhere closes on the inner fence and returns a truncated fragment, andparseObservationsthen finds nothing. The current anchoring gets it right. That is #2282's case.More importantly, D says the fence is not what loses those 76. With a leading sentence the fence does survive
stripCodeFences— but the backticks sit outside the tags, so/<observation>…<\/observation>/still matches and extraction succeeds. Your own table agrees: all 76 are in extracted then skipped, none in nothing extracted. And an extracted observation is only skipped here:So for those 76 the block was found and every content field came back empty. Stripping the fence changes nothing about that. Whatever shape the children are in is the cause, and it is probably the same
<what_happened>/<event>vocabulary from your bucket 2 — the previews for the fenced 76 would settle it, if you still have them.That makes it 285 → 209 batches recovered by the two mechanical fixes, with the fenced 76 folded into the input-echo problem rather than being their own bucket. Still by far the largest recoverable share.
One more source for the skip contract, since the issue body names only
modes/*.json. There is a second, hard-coded copy in the per-tool prompt:Three lines apart: return an empty response, and non-XML is discarded.
plugin/modes/code.jsonskip_guidancesays the same thing ("return an empty response only"), and the comment atprompt-builder.ts:24states the intent outright — "or an empty response if the batch should be skipped … parseAgentXml(...) accepts the response unchanged" — which it does not;parseAgentXmlreturns{valid:false}for empty input on its first line. Fixing only the mode file leaves the prompt still asking for it, which is worth knowing before the 4,176idlediscards get attributed to the model.For the #3593 matrix: I would take the attribute form as a case there once the extractor accepts it, since that is the branch where a type arrives from a place the current tests do not cover at all.
sadiash commentedon Aug 24, 2026
Caveat first: this is field data from v12.1.0, not from latest. I only discovered how far behind I was while investigating. I'm posting it because the plan-18 cluster and the issues closed alongside it (#2956, #3344, #2793, #2468) were triaged without field numbers, and these might be useful for judging severity — but they say nothing about whether 13.15.x already fixes this. Happy to re-run on 13.15.3 if that's more useful.
What I measured
Single install,
provider=claude,auth=cli,model=claude-sonnet-4-6, tier routing on.pending_messagesrows, allstatus='failed'retry_counton every failed row0context overflowlog occurrencesStorage at the point I looked:
pending_messagesobservations(the useful table)logs/~/.claude/projects/Log signature, unchanged across the whole four months:
Three things the numbers add
retry_count = 0on all 15,392 rows is consistent with Failed pending_messages structurally can never self-heal: retryMessage() is fully-implemented, fully-dead code (plus prompt-accretion + missing failure-path logging) #3344's finding thatretryMessage()is dead code. Nothing ever attempted a second pass — not once in four months.Nothing surfaces the loss. The worker reports
{"status":"ok"}throughout./contextshows observations being injected. From inside a session, memory looks like it is working; a third of it silently is not. That invisibility seems worth weighing separately from the overflow itself.The provider asymmetry is visible in config.
CLAUDE_MEM_GEMINI_MAX_CONTEXT_MESSAGES/_MAX_TOKENSand the OpenRouter equivalents both exist and default to 20 / 100000. The Claude provider's only setting isCLAUDE_MEM_CLAUDE_AUTH_METHOD— no context bound, no token bound.tool_responseisJSON.parsed and passed through whole, with no truncation. So the default provider is the only one running unbounded. This is Claude provider (SDKAgent) has no context bound — observer overflows with "Prompt is too long" and saves zero memory on long sessions #2956, and the 31.6% figure is what it costs in practice.What I did locally
Cleanup, which reclaimed 1.72 GB (3.0 GB → 1.3 GB) with no loss of real memories:
Mitigation, to reduce what reaches the observer —
CLAUDE_MEM_SKIP_TOOLSextended with the payload-heavy capture tools (screenshots, DOM snapshots, page text, tool-schema search), keeping the action tools so the narrative survives:Worth noting
SKIP_TOOLSis an exact-matchSet, so MCP tools have to be enumerated one by one — a prefix or glob form would make this mitigation practical for anyone with a few MCP servers connected.Question
Does 13.15.3 bound the Claude provider's history, or is that still what plan-18 is for? That determines whether upgrading is the fix or just a prerequisite for a useful bug report.
ntdatt812 commentedon Aug 24, 2026
@sadiash — on your closing question, I can answer the code half of it from
mainate2d1df5(the 13.15.3 changelog commit). No, 13.15.3 does not bound it, for any provider.conversationHistoryis initialized empty and is only ever appended to:No slice, no shift, no splice, no reassignment anywhere in
src/. The array grows for the life of the session and is passed whole to the provider on every turn.The per-provider settings are the same story. Enumerating every
CLAUDE_MEM_{CLAUDE,GEMINI,OPENROUTER}_*reference at HEAD gives:There is no
_MAX_CONTEXT_MESSAGESand no_MAX_TOKENSfor any provider at HEAD.Two limits on what I just said, so you can weigh it:
One thing that does not apply to your install, since it would otherwise look relevant: the double-append I reported in #3619 is in
OpenAICompatibleProvider—ResponseProcessor.ts:295records the assistant reply and that provider records it again, so its history grows at twice the rate. You are onprovider=claude, whereResponseProcessoris the only writer, so your 31.6% is not that. Your numbers stand on the unbounded history alone.On the
SKIP_TOOLSexact-matchSet— that one is small and self-contained enough to be worth its own issue rather than a line in this thread, if you want the prefix form to actually happen.Priscu commentedon Aug 28, 2026
Adding a precise data point from a production fleet (13.16.1, macOS, PROVIDER=claude, MODEL=claude-sonnet-4-6, 25 concurrent sessions).
Symptom: the default
~/.claudeprofile was logged out after a reboot. Every observer batch got the CLI proseNot logged in · Please run /login. Result over ~3 hours: 4,502 messages enqueued, 857 batches logged asSDK returned non-XML prose response — ignoring queued batch, 0 observations written, and the batches were consumed and dropped.Root cause in the bundle: the prose handler ends with
confirmClaimedMessages(sessionDbId)(batch confirmed = lost). The auth guard that would preserve the batch and pause the generator (resetProcessingToPending+abortReason="auth:observer_text", log line "authentication failed; run /login to preserve queued batch") never fires, because its matcher (UZ) only recognisesfailed to authenticate,authentication failed|failure|errorandauth …phrasings - not the CLI's actual wordingNot logged in.Suggested fix: (1) add
/\bnot logged in\b/(andplease run \/login) to the auth matcher; (2) more robustly, as #3606 proposes: never confirm a batch on any unparsed prose - reset to pending and pause. A batch that did not parse was not processed.Happy to test a build against the same setup.
40 remaining items
perf(observer): parse a nested JSON string only when it may hold an i…
fix(observer): keep MCP and data-URL screenshots out of the condense …
Montinou commentedon Oct 7, 2026
Step 5 slice: let the review-only dedup tier record a verdict
Step 5 keeps the fuzzy tier review-only, and #3038 shipped that half.
observation_dedup_candidates(v56) allowsstatus IN ('pending','merged','distinct','dismissed'), but nothing insrc/writes anything except'pending'(dedup-store.ts:215,:268), and the HTTP surface isGET /api/dedup/candidatesplusPOST /api/dedup/scanonly (DedupRoutes.ts:29-30). A reviewer can list candidates but cannot record a decision, so the list only grows and every pass re-reviews the same pairs. Data point from one install with dedup off: 359 project+title duplicate groups, 165 of them created this month.Proposed slice, no new tables and no background work:
POST /api/dedup/candidates/:id/resolvewith{ "status": "distinct" | "dismissed" }: 409dedup_disabledlike the scan route, 404 for an unknown id, 400 for any other status.GET /api/dedup/candidatesaccepts an optionalstatusfilter; the default stays as today.UNIQUE(observation_id, duplicate_of_id)withINSERT OR IGNOREalready guarantees it, and a test pins that.mergedstays out of this slice: merging rewrites rows (and synced columns), so it deserves its own decision.Tests: resolve to each status, rescan keeps the verdict, disabled returns 409, invalid status returns 400, list filter.
Happy to send it as a PR if it fits the plan.
feat(dedup): record a verdict on review-only candidates
feat(dedup): record a verdict on review-only candidates (#4700)
ahmedibrahim085 commentedon Oct 15, 2026
New child for this plan: one busy session's observer backlog grows faster than the Claude observer can drain it, because the Claude provider sends one queued event per prompt. Opt-in fix in #4793.
ClaudeProviderclaims one message from the iterator per prompt. Codex already batches same-context events (CodexProvider.observationTurnPrompt); Claude does not.CLAUDE_MEM_CLAUDE_OBSERVATION_BATCH_SIZE(default 1) that reusesclaimNextObservation+sameObservationContext. A batch stays within two capped fields, and an oversized field ends it.feat(observer): let the Claude observer batch queued events, opt-in
feat(observer): let the Claude observer batch queued events, opt-in (#…
thedotmack commentedon Oct 19, 2026
Round N consolidation (2026-10-19 community sweep)
OBSERVER_CONVERSATION_MAX_CHARS = 400_000(src/shared/observer-recycle.ts:31, fix(worker): bound observer context growth #3625), and a stateless provider resends the whole history each call, so cost grows up to ~100k tokens per call before the generation recycles. Item 3'smaxHistoryTokensbudget needs a much smaller default for stateless providers.maxTokens=4096and comes back empty (4×), and theDiscarding stale memory_session_id ... (Issue #817)line fires on every generator resume, so it no longer signals a restart.fix(server): log Claude replies cut off at max_tokens
fix(server): Claude provider works with Claude Haiku 5.5 (#5048)
feat(server): Claude provider reads a gateway base URL and custom hea…
thedotmack commentedon Oct 29, 2026
Round N consolidation (2026-10-29 community sweep)
<request>, sometimes<notes>) and never regenerated #5184 (@rigel905): summaries with only<request>(sometimes<next_steps>) are stored as complete.src/sdk/parser.ts:223accepts a summary with a single field,ResponseProcessor.ts:982fills the missing ones with'', and nothing stops a<summary>reply to an observation turn from being stored (parser.ts:87). Item 1 (classify every output) should reject a partial summary instead of storing it as complete.tbrandenburger commentedon Oct 31, 2026
Data point for "bound every history", from claude-mem 12.6.2 on macOS (arm64):
claude --model claude-haiku-4-5 --resume <id> --output-format stream-json) kept resuming the same session and appended roughly 100 MB per minute to a single transcript in~/.claude/projects/-Users-<user>--claude-mem-observer-sessions/.Similar to #1937 and #2754 (closed into #2780), but the loop still happened months later. I have now updated to 15.7.0 and will report back if it recurs. A hard size cap per observer transcript (or not resuming a session above a threshold) would have prevented this.
ShitK commentedon Nov 15, 2026
Proposed slice: minimum content for generated summaries (#5184)
I reproduced the minimum-content part of #5184 on main
62ca1eb865ddc08daad6a23e64f1418b928df7d0(15.18.4), using the realSessionManager/RAM buffer,processAgentResponse, and an in-memory SQLiteSessionStore. Replies were deterministic fixtures; no live inference or production data was used.A request-only summary, a next-steps-only summary, their combination, and a request with whitespace-only progress fields all insert a summary row, set
lastSummaryStored=true, confirm the claimed message, and broadcast a summary. The proposed storage expectations fail for all four on unchanged main. The existing parser/notes-only suites pass (10 tests); the separate characterization fixture passes 12 cases. This was run with Bun 1.3.13, not the current CI's 1.4.4.There is an important compatibility constraint:
response-processor-notes-summary.test.tsexplicitly accepts notes-only summaries. I propose a narrow minimum-content rule, rather than requiring every schema field: at least one ofinvestigated,learned,completed, ornotesmust be non-empty after the existing parser's decoding/trimming.requestandnext_stepsalone would not qualify. This cannot validate factual or semantic quality.I would keep XML validity separate from storage eligibility. Simply returning
valid:falseroutes a queued reply into the existing output-retry path (confirmed with an empty-summary control). For these newly rejected, already-answered summaries, the proposal is terminal no-store handling: preserve source evidence and completion deduplication, terminally confirm only the answered batch (without requeueing it), report a content-free diagnostic, and allow fresh work normally. No extra paid retry, fallback, startup replay, or historical regeneration.Scope: shared response/storage boundary, prompt guidance, real SQLite/buffer regressions, and a provider-boundary dispatch test proving no extra send and subsequent fresh-work recovery. No schema migration or deletion of old summaries. A content-bearing summary returned to an observation turn is also reproducible, but request/reply type enforcement would remain a separate slice.
Does that minimum-content rule fit the intended contract, particularly retaining notes-only replies while declining a reply containing only future plans? I have the reproduction and plan ready; implementation has not started.