memory: July 2026 audit + quick wins — revive curation LLM, balanced prompt, recall precision re-tune#1568
Merged
Aaronontheweb merged 7 commits intoJul 4, 2026
Conversation
This was referenced Jul 3, 2026
Measured audit of the memory system against the live corpus (1,216 docs), all daemon logs since April, and every recall injection from the last 14 days (exhaustively relevance-judged, k=0.754): 46% recall pollution, no relevance signal in the lexical score, LLM curation tier 0-for-6 lifetime, 95% checkpoint drop, 14% corpus redundancy doubling in five weeks, searchable-mode leakage into automatic recall, vestigial Trace/expiry. 19-item defect list with file:line refs; evidence base for the quick-win slice and the memory-core-redesign OpenSpec change. The research tooling (miners, Haiku judge harness, dedup/nominator pipelines, embed bench) operates on real PII corpus data and lives in the operator-local research store — tools/recall-research/ is gitignored so it can never land in the repo.
…ion re-tune, injection budget, config-knob wiring Quick-win slice from the July 2026 memory audit (docs/research/memory-audit-2026-07.md): - Curation LLM tier (measured DEAD in production — 0 successful decisions ever): MaxOutputTokens 512→1500 and per-call reasoning suppression (chat_template_kwargs.enable_thinking=false + ollama think=false); new 'Memory Curation LLM' doctor check scans daemon logs and warns on a dead or >20%-failing tier. - Balanced curation prompt (measured Pareto win, May 2026 decider eval): 'same underlying FACT?' framing, dated observations append-only, kills the date/price/status UPDATE-overwrite guidance; content previews 200→700 chars. - Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0 (measured junk-injection vectors), DurableFact RecallRank 120→480, floor 10→14 — durable facts now need two independent lexical matches (or one + facet); zero-injection turns are intended behavior (65% of real queries had nothing relevant to inject). - Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000), whole items dropped in rank order, first-ranked always admitted; memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced. - Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems now actually feed SessionRecallManager (were hardcoded 300ms/3). - Retroactive netclaw-dev#1225 data repair in SQLiteMemoryStore.InitializeAsync: legacy compaction-boundary rows at recall_mode=searchable (still in the automatic pool — measured top polluter) move to manual, idempotently. - Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only recall; netclaw-memory skill 1.7.0 documents selective recall and that zero-injection turns are normal. - Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty under lexical recall; flips back with hybrid recall), evidence-class plumbing test pins policy vs plumbing explicitly, new tests for the char budget and the compaction repair. 4,971 tests green; slopwatch clean.
Aaronontheweb
force-pushed
the
memory-audit-quick-wins
branch
from
July 3, 2026 18:04
21aa0c1 to
f7f2f9a
Compare
Aaronontheweb
added a commit
that referenced
this pull request
Jul 3, 2026
) * opsx: memory-core-redesign — proposal, design, spec deltas, tasks OpenSpec change for the memory-core structural redesign, grounded in the July 2026 audit (PR #1568, tools/recall-research/AUDIT-2026-07.md): - New capabilities: memory-embeddings (in-process ONNX runtime, pinned allowlist provisioning, embed-on-write, loud degradation) and memory-maintenance (backfill, ratified consolidation with plan files, expiry sweep, memory status). - netclaw-agent-memory deltas: hybrid recall with an absolute cosine floor (zero-injection turns are intended), nominate-by-embedding / decide-by-LLM curation through one shared evaluator, lossless merges with a deterministic MergeGuard, recall modes that mean what they say (searchable leaves the automatic pool), short-lived trace revival, tool-use lesson memories with per-tool context injection, checkpoint enqueue gating; the never-used hierarchical anchor graph requirement is removed. - tasks.md: six independently-shippable slices with constitution gates. openspec validate: passing (17 deltas). * opsx(memory-core-redesign): point evidence references at docs/research/ — research tooling is repo-external (tools/recall-research is gitignored)
Aaronontheweb
added a commit
that referenced
this pull request
Jul 3, 2026
…p 3 (#1573) RunResetAsync publishes CurrentProgressStep = 3 ("Purge complete") and then evaluates the Task.Delay(CompletionPause, _timeProvider, ct) that registers the completion-pause timer. Because RunResetAsync runs off the loop, step 3 is the signal observers gate on to know deletion finished — but it fired BEFORE the timer it vouches for existed. On a virtualized clock (FakeTimeProvider in tests, and any consumer advancing the injected TimeProvider) this is a lost wakeup: an Advance that lands between "step 3 published" and "timer registered" is dropped, the delay never fires, and the reset hangs forever awaiting a wakeup that will never come. Fix: create the completion-pause timer before publishing step 3, then await the captured task. This makes "step 3 observed" a happens-before guarantee that the pause timer already exists, so a single clock advance always fires it deterministically. On the real clock this shifts timer registration a few microseconds earlier; the visible "Purge complete" dwell is unchanged. Evidence: hang-dump forensics of CI run 28676578217 (PR #1568, Netclaw.Cli.Tests blame-hang after 300s). Test FullReset_AfterBothConfirmations_DeletesEverything was parked at `await vm.ResetTask!.WaitAsync(ct)`; its CompleteResetAsync helper had exhausted its advance loop (counter i == 10, all virtual-clock advances spent) and FakeTimeProvider was left holding exactly one orphaned waiter — the delay registered after the final Advance. Prior commits 850cf8c (#1525) and eb01b49 (#1536) band-aided the symptom with advance/yield retry loops; this removes the race at the source. The test helper CompleteResetAsync is shared by all three sibling tests (FullReset_AfterBothConfirmations_DeletesEverything, SetupOnlyReset_DeletesConfigButKeepsMemoryAndSessions, DaemonStopFailureResult_ShowsStatusAndContinuesReset); it now waits for step 3 then does exactly one advance, deleting the retry loop that only ever masked the lost wakeup on loaded runners.
… bind The cap is the third line of defense after reasoning suppression and the 10s timeout. 1500 was still marginal in the suppression-ignored window (Qwen-class thinking traces run 800-2000 tokens on this comparison task); truncation there reproduces the empty-reply failure the audit measured. Unemitted tokens are free, so the cap is sized to never be the binding constraint. Becomes config (Memory.Curation.LlmMaxOutputTokens) in memory-core-redesign.
… instead of raw wire fields
The curation-actor reasoning suppression shipped earlier in this PR set raw
provider-dialect fields (chat_template_kwargs.enable_thinking, think) directly
on ChatOptions.AdditionalProperties — a provider-agnostic seam. That was only
correct for the pass-through OpenAiCompatibleChatClient, which forwards
AdditionalProperties verbatim as top-level request JSON for vLLM/llama.cpp/
SGLang. For every other provider the raw keys were wrong in one of two ways:
- Verified by decompiling Microsoft.Extensions.AI.OpenAI 10.6.0
(OpenAIChatClient.ToOpenAIOptions, OpenAIResponsesChatClient) and
Anthropic.SDK 12.35.1 (AnthropicChatClient.GetMessageCreateParams): neither
adapter ever iterates ChatOptions.AdditionalProperties onto the request
body — both only special-case a "strict" flag for JSON-schema response
formats. So on OpenAI/Copilot/OpenRouter/VeniceAI/Anthropic the keys were
silently inert (dropped, not 400s) — dead weight masquerading as a fix.
- Any future adapter that DOES forward extras (or a RawRepresentationFactory
path) would push unknown top-level params at strict APIs, which reject them
with 400.
This change replaces the raw fields with a provider-mapped intent key:
- NetclawChatOptionKeys.SuppressReasoning ("netclaw:suppress-reasoning") in
Netclaw.Configuration — call sites express intent, never wire dialect.
- ILlmProviderPlugin.SuppressionDialect (default None — a genuine domain
default: strip-unknown is the only correct behavior for a plugin that has
not declared a serving-stack dialect). Overrides:
OpenAiCompatibleProviderPlugin → ChatTemplateKwargs,
OllamaProviderPlugin → OllamaThink. OpenAI, Anthropic, Copilot, OpenRouter,
VeniceAI inherit None (strip only).
- ReasoningSuppressionChatClient (Netclaw.Daemon, beside
VendorOptionsChatClient) wraps EVERY client ProviderPluginFactory.Create
returns: it always removes the intent key, and when true maps it to the
declared dialect. Mutates options in place per the established
VendorOptionsChatClient pattern (call sites build fresh ChatOptions per
call, so no retry hazard).
- MemoryCurationActor.TryLlmEvaluationAsync now sets only the intent key.
The strip guarantee applies to all roles/providers, so any future call site
can use the intent key without knowing which serving stack answers.
Tests: ReasoningSuppressionChatClientTests (dialect mapping, unconditional
strip, false-value strip-without-map, pass-through untouched, null options,
streaming parity), plugin dialect declarations, and a factory regression
guard that every created client carries the wrapper. Daemon.Tests 825/825,
Actors.Tests 2531/2531 green; slopwatch 0 issues; headers verified.
Aaronontheweb
commented
Jul 4, 2026
Aaronontheweb
left a comment
Collaborator
Author
There was a problem hiding this comment.
LGTM - should recall way fewer memories this way.
| // ever). Unemitted tokens cost nothing, so size this generously; | ||
| // it becomes the Memory.Curation.LlmMaxOutputTokens config knob in | ||
| // the memory-core-redesign change. | ||
| MaxOutputTokens = 4096, |
Collaborator
Author
There was a problem hiding this comment.
this number was too small originally but it's been improved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Two things, deliberately in one PR because the second is justified line-by-line by the first:
docs/research/memory-audit-2026-07.md) against the live 1,216-document corpus, all daemon logs since April, and every recall event from the last 14 days (exhaustively relevance-judged, κ=0.754 inter-rater agreement).memory-core-redesign).Headline audit findings (19-item defect list in the report)
TurnCompleteNoProjectFact) — the checkpoint lane is busywork; the inline sidecar is the real producer.searchablerecall mode participates in automatic recall (semantic trap), which is how 22 pre-fix(memory): keep compaction-boundary summaries out of automatic recall #1225 compaction summaries remained top polluters.Quick wins in this PR
MaxOutputTokens512→1500 + per-call reasoning suppression (chat_template_kwargs.enable_thinking=false, ollamathink=false); new Memory Curation LLM doctor check warns on a dead/failing tier (this failed silently for 3 months).SessionTuning.MaxRecallInjectedChars, default 2000): items admitted in rank order, dropped whole, never truncated;memory_retrieval_finallogsinjectedChars/droppedByBudget. Config schema synced.Memory.RecallTimeoutMsandMemory.AutoRecallMaxItemsnow actually feedSessionRecallManager(were hardcoded 300ms/3).compaction-boundaryrows out of the automatic pool on next initialization.netclaw-memoryskill 1.7.0 documents selective recall.Validation
memory_identity_preference_routing, 3/5) was already 0/5 on dev on 2026-06-24; every other case 5/5 both runs.memory_recall_filters(which exercises the new floor/budget path) 5/5.Depends on
fix(cli): remote-endpoint preflight) — needed to RUN the eval harness at all; merge that first. This PR does not contain that commit.Deploy note
The #1225 data repair executes on the first daemon start (or
netclaw doctor) with this build — it updates the ~22 legacy compaction rows, which is the intended completion of that fix.