Skip to content

fix(memory): keep compaction-boundary summaries out of automatic recall#1225

Merged
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/1224-compaction-boundary-recall-mode
May 29, 2026
Merged

fix(memory): keep compaction-boundary summaries out of automatic recall#1225
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/1224-compaction-boundary-recall-mode

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Fixes #1224.

What

Compaction-boundary session summaries were persisted as Evidence / Searchable memories, which put them in the automatic recall pool. Each one is a multi-KB digest of a whole session, so it lexically matched almost any query in that session's topic and sorted to the top of the candidate pool on unrelated queries, crowding out atomic memories.

Change

One special-case in MemoryCurationPipeline.ResolveRecallMode: compaction-boundary checkpoints resolve to MemoryRecallMode.Manual (before the Evidence -> Searchable branch). Manual is excluded from every recall query (auto-recall and explicit search both fetch only Auto/Searchable), so the record is retained but never auto-recalled.

Compaction is unaffected. The summary it relies on for continuity lives in the SessionCompacted event / in-session history (SessionCompactionPipeline + ObservationPromptBuilder.ExtractPriorSummary); this memory is a fire-and-forget side-channel written in LlmSessionActor.HandleCompactionWorkCompleted that nothing reads back.

Test

Adds a regression test asserting a compaction-boundary checkpoint extracts as Evidence with RecallMode == Manual (not Auto/Searchable). Existing memory/recall/curation tests still pass.

Follow-up (not in this PR)

Existing stores still hold compaction-boundary rows with recall_mode = searchable. A one-time migration or netclaw doctor pass could declass them; tracked in the issue.

Compaction-boundary checkpoints are whole-session summary blobs. They were classed
Evidence and so resolved to Searchable recall mode, which puts them in the automatic
recall pool (SearchByPlanAsync fetches Auto/Searchable). Because each is a multi-KB
session digest, it lexically matched almost any query in that session's topic and
sorted to the top of the candidate pool on unrelated queries, crowding out atomic
memories.

ResolveRecallMode now special-cases IsCompactionBoundary -> Manual (retained but
excluded from every recall query). Compaction is unaffected: the summary it relies on
lives in the SessionCompacted event / in-session history; this memory is a
fire-and-forget side-channel nothing reads back.

Adds a regression test asserting a compaction-boundary checkpoint extracts as
Evidence with RecallMode == Manual.

Fixes netclaw-dev#1224
@Aaronontheweb Aaronontheweb added bug Something isn't working memory Memory formation, recall, curation pipeline labels May 29, 2026

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) May 29, 2026 15:06
@Aaronontheweb
Aaronontheweb merged commit aa4c7c9 into netclaw-dev:dev May 29, 2026
14 checks passed
Aaronontheweb added a commit to Aaronontheweb/netclaw that referenced this pull request Jul 3, 2026
…ion re-tune, injection budget, config-knob wiring

Quick-win slice from the July 2026 memory audit
(docs/research/memory-audit-2026-07.md):

- Curation LLM tier (measured DEAD in production — 0 successful decisions
  ever): MaxOutputTokens 512→1500 and per-call reasoning suppression
  (chat_template_kwargs.enable_thinking=false + ollama think=false); new
  'Memory Curation LLM' doctor check scans daemon logs and warns on a dead
  or >20%-failing tier.
- Balanced curation prompt (measured Pareto win, May 2026 decider eval):
  'same underlying FACT?' framing, dated observations append-only, kills the
  date/price/status UPDATE-overwrite guidance; content previews 200→700 chars.
- Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0
  (measured junk-injection vectors), DurableFact RecallRank 120→480, floor
  10→14 — durable facts now need two independent lexical matches (or one +
  facet); zero-injection turns are intended behavior (65% of real queries
  had nothing relevant to inject).
- Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000),
  whole items dropped in rank order, first-ranked always admitted;
  memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced.
- Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems
  now actually feed SessionRecallManager (were hardcoded 300ms/3).
- Retroactive netclaw-dev#1225 data repair in SQLiteMemoryStore.InitializeAsync:
  legacy compaction-boundary rows at recall_mode=searchable (still in the
  automatic pool — measured top polluter) move to manual, idempotently.
- Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only
  recall; netclaw-memory skill 1.7.0 documents selective recall and that
  zero-injection turns are normal.
- Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty
  under lexical recall; flips back with hybrid recall), evidence-class
  plumbing test pins policy vs plumbing explicitly, new tests for the char
  budget and the compaction repair. 4,971 tests green; slopwatch clean.
Aaronontheweb added a commit that referenced this pull request Jul 4, 2026
…prompt, recall precision re-tune (#1568)

* docs(research): July 2026 memory audit + May recall findings

Measured audit of the memory system against the live corpus (1,216 docs),
all daemon logs since April, and every recall injection from the last 14
days (exhaustively relevance-judged, k=0.754): 46% recall pollution, no
relevance signal in the lexical score, LLM curation tier 0-for-6 lifetime,
95% checkpoint drop, 14% corpus redundancy doubling in five weeks,
searchable-mode leakage into automatic recall, vestigial Trace/expiry.
19-item defect list with file:line refs; evidence base for the quick-win
slice and the memory-core-redesign OpenSpec change.

The research tooling (miners, Haiku judge harness, dedup/nominator
pipelines, embed bench) operates on real PII corpus data and lives in the
operator-local research store — tools/recall-research/ is gitignored so it
can never land in the repo.

* fix(memory): revive curation LLM tier, balanced prompt, recall precision re-tune, injection budget, config-knob wiring

Quick-win slice from the July 2026 memory audit
(docs/research/memory-audit-2026-07.md):

- Curation LLM tier (measured DEAD in production — 0 successful decisions
  ever): MaxOutputTokens 512→1500 and per-call reasoning suppression
  (chat_template_kwargs.enable_thinking=false + ollama think=false); new
  'Memory Curation LLM' doctor check scans daemon logs and warns on a dead
  or >20%-failing tier.
- Balanced curation prompt (measured Pareto win, May 2026 decider eval):
  'same underlying FACT?' framing, dated observations append-only, kills the
  date/price/status UPDATE-overwrite guidance; content previews 200→700 chars.
- Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0
  (measured junk-injection vectors), DurableFact RecallRank 120→480, floor
  10→14 — durable facts now need two independent lexical matches (or one +
  facet); zero-injection turns are intended behavior (65% of real queries
  had nothing relevant to inject).
- Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000),
  whole items dropped in rank order, first-ranked always admitted;
  memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced.
- Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems
  now actually feed SessionRecallManager (were hardcoded 300ms/3).
- Retroactive #1225 data repair in SQLiteMemoryStore.InitializeAsync:
  legacy compaction-boundary rows at recall_mode=searchable (still in the
  automatic pool — measured top polluter) move to manual, idempotently.
- Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only
  recall; netclaw-memory skill 1.7.0 documents selective recall and that
  zero-injection turns are normal.
- Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty
  under lexical recall; flips back with hybrid recall), evidence-class
  plumbing test pins policy vs plumbing explicitly, new tests for the char
  budget and the compaction repair. 4,971 tests green; slopwatch clean.

* fix(memory): raise curation LLM token cap 1500->4096 — cap must never bind

The cap is the third line of defense after reasoning suppression and the 10s timeout. 1500 was still marginal in the suppression-ignored window (Qwen-class thinking traces run 800-2000 tokens on this comparison task); truncation there reproduces the empty-reply failure the audit measured. Unemitted tokens are free, so the cap is sized to never be the binding constraint. Becomes config (Memory.Curation.LlmMaxOutputTokens) in memory-core-redesign.

* fix(providers): map reasoning-suppression intent per provider dialect instead of raw wire fields

The curation-actor reasoning suppression shipped earlier in this PR set raw
provider-dialect fields (chat_template_kwargs.enable_thinking, think) directly
on ChatOptions.AdditionalProperties — a provider-agnostic seam. That was only
correct for the pass-through OpenAiCompatibleChatClient, which forwards
AdditionalProperties verbatim as top-level request JSON for vLLM/llama.cpp/
SGLang. For every other provider the raw keys were wrong in one of two ways:

- Verified by decompiling Microsoft.Extensions.AI.OpenAI 10.6.0
  (OpenAIChatClient.ToOpenAIOptions, OpenAIResponsesChatClient) and
  Anthropic.SDK 12.35.1 (AnthropicChatClient.GetMessageCreateParams): neither
  adapter ever iterates ChatOptions.AdditionalProperties onto the request
  body — both only special-case a "strict" flag for JSON-schema response
  formats. So on OpenAI/Copilot/OpenRouter/VeniceAI/Anthropic the keys were
  silently inert (dropped, not 400s) — dead weight masquerading as a fix.
- Any future adapter that DOES forward extras (or a RawRepresentationFactory
  path) would push unknown top-level params at strict APIs, which reject them
  with 400.

This change replaces the raw fields with a provider-mapped intent key:

- NetclawChatOptionKeys.SuppressReasoning ("netclaw:suppress-reasoning") in
  Netclaw.Configuration — call sites express intent, never wire dialect.
- ILlmProviderPlugin.SuppressionDialect (default None — a genuine domain
  default: strip-unknown is the only correct behavior for a plugin that has
  not declared a serving-stack dialect). Overrides:
  OpenAiCompatibleProviderPlugin → ChatTemplateKwargs,
  OllamaProviderPlugin → OllamaThink. OpenAI, Anthropic, Copilot, OpenRouter,
  VeniceAI inherit None (strip only).
- ReasoningSuppressionChatClient (Netclaw.Daemon, beside
  VendorOptionsChatClient) wraps EVERY client ProviderPluginFactory.Create
  returns: it always removes the intent key, and when true maps it to the
  declared dialect. Mutates options in place per the established
  VendorOptionsChatClient pattern (call sites build fresh ChatOptions per
  call, so no retry hazard).
- MemoryCurationActor.TryLlmEvaluationAsync now sets only the intent key.

The strip guarantee applies to all roles/providers, so any future call site
can use the intent key without knowing which serving stack answers.

Tests: ReasoningSuppressionChatClientTests (dialect mapping, unconditional
strip, false-value strip-without-map, pass-through untouched, null options,
streaming parity), plugin dialect declarations, and a factory regression
guard that every created client carries the wrapper. Daemon.Tests 825/825,
Actors.Tests 2531/2531 green; slopwatch 0 issues; headers verified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working memory Memory formation, recall, curation pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compaction-boundary session summaries pollute automatic memory recall

1 participant