fix(memory): keep compaction-boundary summaries out of automatic recall#1225
Merged
Aaronontheweb merged 1 commit intoMay 29, 2026
Conversation
Compaction-boundary checkpoints are whole-session summary blobs. They were classed Evidence and so resolved to Searchable recall mode, which puts them in the automatic recall pool (SearchByPlanAsync fetches Auto/Searchable). Because each is a multi-KB session digest, it lexically matched almost any query in that session's topic and sorted to the top of the candidate pool on unrelated queries, crowding out atomic memories. ResolveRecallMode now special-cases IsCompactionBoundary -> Manual (retained but excluded from every recall query). Compaction is unaffected: the summary it relies on lives in the SessionCompacted event / in-session history; this memory is a fire-and-forget side-channel nothing reads back. Adds a regression test asserting a compaction-boundary checkpoint extracts as Evidence with RecallMode == Manual. Fixes netclaw-dev#1224
Aaronontheweb
enabled auto-merge (squash)
May 29, 2026 15:06
Aaronontheweb
added a commit
to Aaronontheweb/netclaw
that referenced
this pull request
Jul 3, 2026
…ion re-tune, injection budget, config-knob wiring Quick-win slice from the July 2026 memory audit (docs/research/memory-audit-2026-07.md): - Curation LLM tier (measured DEAD in production — 0 successful decisions ever): MaxOutputTokens 512→1500 and per-call reasoning suppression (chat_template_kwargs.enable_thinking=false + ollama think=false); new 'Memory Curation LLM' doctor check scans daemon logs and warns on a dead or >20%-failing tier. - Balanced curation prompt (measured Pareto win, May 2026 decider eval): 'same underlying FACT?' framing, dated observations append-only, kills the date/price/status UPDATE-overwrite guidance; content previews 200→700 chars. - Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0 (measured junk-injection vectors), DurableFact RecallRank 120→480, floor 10→14 — durable facts now need two independent lexical matches (or one + facet); zero-injection turns are intended behavior (65% of real queries had nothing relevant to inject). - Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000), whole items dropped in rank order, first-ranked always admitted; memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced. - Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems now actually feed SessionRecallManager (were hardcoded 300ms/3). - Retroactive netclaw-dev#1225 data repair in SQLiteMemoryStore.InitializeAsync: legacy compaction-boundary rows at recall_mode=searchable (still in the automatic pool — measured top polluter) move to manual, idempotently. - Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only recall; netclaw-memory skill 1.7.0 documents selective recall and that zero-injection turns are normal. - Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty under lexical recall; flips back with hybrid recall), evidence-class plumbing test pins policy vs plumbing explicitly, new tests for the char budget and the compaction repair. 4,971 tests green; slopwatch clean.
Aaronontheweb
added a commit
that referenced
this pull request
Jul 4, 2026
…prompt, recall precision re-tune (#1568) * docs(research): July 2026 memory audit + May recall findings Measured audit of the memory system against the live corpus (1,216 docs), all daemon logs since April, and every recall injection from the last 14 days (exhaustively relevance-judged, k=0.754): 46% recall pollution, no relevance signal in the lexical score, LLM curation tier 0-for-6 lifetime, 95% checkpoint drop, 14% corpus redundancy doubling in five weeks, searchable-mode leakage into automatic recall, vestigial Trace/expiry. 19-item defect list with file:line refs; evidence base for the quick-win slice and the memory-core-redesign OpenSpec change. The research tooling (miners, Haiku judge harness, dedup/nominator pipelines, embed bench) operates on real PII corpus data and lives in the operator-local research store — tools/recall-research/ is gitignored so it can never land in the repo. * fix(memory): revive curation LLM tier, balanced prompt, recall precision re-tune, injection budget, config-knob wiring Quick-win slice from the July 2026 memory audit (docs/research/memory-audit-2026-07.md): - Curation LLM tier (measured DEAD in production — 0 successful decisions ever): MaxOutputTokens 512→1500 and per-call reasoning suppression (chat_template_kwargs.enable_thinking=false + ollama think=false); new 'Memory Curation LLM' doctor check scans daemon logs and warns on a dead or >20%-failing tier. - Balanced curation prompt (measured Pareto win, May 2026 decider eval): 'same underlying FACT?' framing, dated observations append-only, kills the date/price/status UPDATE-overwrite guidance; content previews 200→700 chars. - Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0 (measured junk-injection vectors), DurableFact RecallRank 120→480, floor 10→14 — durable facts now need two independent lexical matches (or one + facet); zero-injection turns are intended behavior (65% of real queries had nothing relevant to inject). - Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000), whole items dropped in rank order, first-ranked always admitted; memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced. - Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems now actually feed SessionRecallManager (were hardcoded 300ms/3). - Retroactive #1225 data repair in SQLiteMemoryStore.InitializeAsync: legacy compaction-boundary rows at recall_mode=searchable (still in the automatic pool — measured top polluter) move to manual, idempotently. - Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only recall; netclaw-memory skill 1.7.0 documents selective recall and that zero-injection turns are normal. - Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty under lexical recall; flips back with hybrid recall), evidence-class plumbing test pins policy vs plumbing explicitly, new tests for the char budget and the compaction repair. 4,971 tests green; slopwatch clean. * fix(memory): raise curation LLM token cap 1500->4096 — cap must never bind The cap is the third line of defense after reasoning suppression and the 10s timeout. 1500 was still marginal in the suppression-ignored window (Qwen-class thinking traces run 800-2000 tokens on this comparison task); truncation there reproduces the empty-reply failure the audit measured. Unemitted tokens are free, so the cap is sized to never be the binding constraint. Becomes config (Memory.Curation.LlmMaxOutputTokens) in memory-core-redesign. * fix(providers): map reasoning-suppression intent per provider dialect instead of raw wire fields The curation-actor reasoning suppression shipped earlier in this PR set raw provider-dialect fields (chat_template_kwargs.enable_thinking, think) directly on ChatOptions.AdditionalProperties — a provider-agnostic seam. That was only correct for the pass-through OpenAiCompatibleChatClient, which forwards AdditionalProperties verbatim as top-level request JSON for vLLM/llama.cpp/ SGLang. For every other provider the raw keys were wrong in one of two ways: - Verified by decompiling Microsoft.Extensions.AI.OpenAI 10.6.0 (OpenAIChatClient.ToOpenAIOptions, OpenAIResponsesChatClient) and Anthropic.SDK 12.35.1 (AnthropicChatClient.GetMessageCreateParams): neither adapter ever iterates ChatOptions.AdditionalProperties onto the request body — both only special-case a "strict" flag for JSON-schema response formats. So on OpenAI/Copilot/OpenRouter/VeniceAI/Anthropic the keys were silently inert (dropped, not 400s) — dead weight masquerading as a fix. - Any future adapter that DOES forward extras (or a RawRepresentationFactory path) would push unknown top-level params at strict APIs, which reject them with 400. This change replaces the raw fields with a provider-mapped intent key: - NetclawChatOptionKeys.SuppressReasoning ("netclaw:suppress-reasoning") in Netclaw.Configuration — call sites express intent, never wire dialect. - ILlmProviderPlugin.SuppressionDialect (default None — a genuine domain default: strip-unknown is the only correct behavior for a plugin that has not declared a serving-stack dialect). Overrides: OpenAiCompatibleProviderPlugin → ChatTemplateKwargs, OllamaProviderPlugin → OllamaThink. OpenAI, Anthropic, Copilot, OpenRouter, VeniceAI inherit None (strip only). - ReasoningSuppressionChatClient (Netclaw.Daemon, beside VendorOptionsChatClient) wraps EVERY client ProviderPluginFactory.Create returns: it always removes the intent key, and when true maps it to the declared dialect. Mutates options in place per the established VendorOptionsChatClient pattern (call sites build fresh ChatOptions per call, so no retry hazard). - MemoryCurationActor.TryLlmEvaluationAsync now sets only the intent key. The strip guarantee applies to all roles/providers, so any future call site can use the intent key without knowing which serving stack answers. Tests: ReasoningSuppressionChatClientTests (dialect mapping, unconditional strip, false-value strip-without-map, pass-through untouched, null options, streaming parity), plugin dialect declarations, and a factory regression guard that every created client carries the wrapper. Daemon.Tests 825/825, Actors.Tests 2531/2531 green; slopwatch 0 issues; headers verified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1224.
What
Compaction-boundary session summaries were persisted as
Evidence/Searchablememories, which put them in the automatic recall pool. Each one is a multi-KB digest of a whole session, so it lexically matched almost any query in that session's topic and sorted to the top of the candidate pool on unrelated queries, crowding out atomic memories.Change
One special-case in
MemoryCurationPipeline.ResolveRecallMode: compaction-boundary checkpoints resolve toMemoryRecallMode.Manual(before theEvidence -> Searchablebranch).Manualis excluded from every recall query (auto-recall and explicit search both fetch onlyAuto/Searchable), so the record is retained but never auto-recalled.Compaction is unaffected. The summary it relies on for continuity lives in the
SessionCompactedevent / in-session history (SessionCompactionPipeline+ObservationPromptBuilder.ExtractPriorSummary); this memory is a fire-and-forget side-channel written inLlmSessionActor.HandleCompactionWorkCompletedthat nothing reads back.Test
Adds a regression test asserting a compaction-boundary checkpoint extracts as
EvidencewithRecallMode == Manual(notAuto/Searchable). Existing memory/recall/curation tests still pass.Follow-up (not in this PR)
Existing stores still hold
compaction-boundaryrows withrecall_mode = searchable. A one-time migration ornetclaw doctorpass could declass them; tracked in the issue.