Skip to content

memory: July 2026 audit + quick wins — revive curation LLM, balanced prompt, recall precision re-tune#1568

Merged
Aaronontheweb merged 7 commits into
netclaw-dev:devfrom
Aaronontheweb:memory-audit-quick-wins
Jul 4, 2026
Merged

memory: July 2026 audit + quick wins — revive curation LLM, balanced prompt, recall precision re-tune#1568
Aaronontheweb merged 7 commits into
netclaw-dev:devfrom
Aaronontheweb:memory-audit-quick-wins

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

What this is

Two things, deliberately in one PR because the second is justified line-by-line by the first:

  1. A measured audit of the memory system (docs/research/memory-audit-2026-07.md) against the live 1,216-document corpus, all daemon logs since April, and every recall event from the last 14 days (exhaustively relevance-judged, κ=0.754 inter-rater agreement).
  2. The quick-win fixes for the defects the audit confirmed — the low-risk, already-validated subset. The structural redesign (embeddings, hybrid recall, lossless merge, taxonomy rebalance) is a separate upcoming OpenSpec change (memory-core-redesign).

Headline audit findings (19-item defect list in the report)

  • 46% of auto-injected memories are pollution, 27% marginal, only 26% relevant; 19% of recall events actively misleading. On 65% of real queries nothing relevant existed to inject — but the system always injected its top-3.
  • The lexical composite score carries no relevance signal on real traffic (precision flat ~0.34 across every floor 10→60).
  • The LLM curation tier has never worked: 6 invocations lifetime, 0 successes (3 timeouts, then 3 empty responses — reasoning-model thinking exhausts the 512-token cap).
  • Consolidation count: 2, lifetime. 239 memories (1 in 5) sit in 69 near-duplicate clusters; redundant share doubled in five weeks.
  • 95% of daemon checkpoints are dropped (TurnCompleteNoProjectFact) — the checkpoint lane is busywork; the inline sidecar is the real producer.
  • searchable recall mode participates in automatic recall (semantic trap), which is how 22 pre-fix(memory): keep compaction-boundary summaries out of automatic recall #1225 compaction summaries remained top polluters.
  • Trace class + expiry: fully vestigial (0 traces ever; no purge job; 204/384 evidence records past expiry and still on disk).

Quick wins in this PR

  • Curation LLM revival: MaxOutputTokens 512→1500 + per-call reasoning suppression (chat_template_kwargs.enable_thinking=false, ollama think=false); new Memory Curation LLM doctor check warns on a dead/failing tier (this failed silently for 3 months).
  • Balanced curation prompt (the variant measured as a Pareto win in the May decider eval): 'same underlying FACT?' framing, dated observations append-only, kills the 'date/price/status changed → UPDATE' overwrite guidance; decider previews 200→700 chars.
  • Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0 (measured junk-injection vectors), DurableFact class prior 120→480, floor 10→14 — a durable fact now needs ~2 independent lexical matches; zero-injection turns are intended behavior and documented as such.
  • Injection char budget (SessionTuning.MaxRecallInjectedChars, default 2000): items admitted in rank order, dropped whole, never truncated; memory_retrieval_final logs injectedChars/droppedByBudget. Config schema synced.
  • Dead config knobs wired: Memory.RecallTimeoutMs and Memory.AutoRecallMaxItems now actually feed SessionRecallManager (were hardcoded 300ms/3).
  • Retroactive fix(memory): keep compaction-boundary summaries out of automatic recall #1225 completion: idempotent data repair moves legacy compaction-boundary rows out of the automatic pool on next initialization.
  • Truth-ups: context layer no longer claims durable_fact-only recall; netclaw-memory skill 1.7.0 documents selective recall.
  • Salvaged the operator-local research store from the stale May branch + the new audit tooling (pollution miner, Haiku judge harness, curation health, embed bench, consolidation worklist). All PII stays local (enforced by .gitignore); the report is redacted aggregates.

Validation

  • 4,971 tests green across Actors/Cli/Configuration/Daemon/MemoryRetrievalPoC; slopwatch 0 issues; headers verified; schema parses + round-trips.
  • Memory eval gate: 4/5 (80%), zero regression vs baseline — the one sub-threshold case (memory_identity_preference_routing, 3/5) was already 0/5 on dev on 2026-06-24; every other case 5/5 both runs. memory_recall_filters (which exercises the new floor/budget path) 5/5.
  • Scenario suite updated deliberately: P09 re-pinned as the paraphrase-gap case (expected-empty under lexical recall; flips back when Phase-C hybrid recall lands), evidence-class test pins plumbing-vs-policy explicitly, new tests for the char budget and the compaction repair.

Depends on

Deploy note

The #1225 data repair executes on the first daemon start (or netclaw doctor) with this build — it updates the ~22 legacy compaction rows, which is the intended completion of that fix.

Measured audit of the memory system against the live corpus (1,216 docs),
all daemon logs since April, and every recall injection from the last 14
days (exhaustively relevance-judged, k=0.754): 46% recall pollution, no
relevance signal in the lexical score, LLM curation tier 0-for-6 lifetime,
95% checkpoint drop, 14% corpus redundancy doubling in five weeks,
searchable-mode leakage into automatic recall, vestigial Trace/expiry.
19-item defect list with file:line refs; evidence base for the quick-win
slice and the memory-core-redesign OpenSpec change.

The research tooling (miners, Haiku judge harness, dedup/nominator
pipelines, embed bench) operates on real PII corpus data and lives in the
operator-local research store — tools/recall-research/ is gitignored so it
can never land in the repo.
…ion re-tune, injection budget, config-knob wiring

Quick-win slice from the July 2026 memory audit
(docs/research/memory-audit-2026-07.md):

- Curation LLM tier (measured DEAD in production — 0 successful decisions
  ever): MaxOutputTokens 512→1500 and per-call reasoning suppression
  (chat_template_kwargs.enable_thinking=false + ollama think=false); new
  'Memory Curation LLM' doctor check scans daemon logs and warns on a dead
  or >20%-failing tier.
- Balanced curation prompt (measured Pareto win, May 2026 decider eval):
  'same underlying FACT?' framing, dated observations append-only, kills the
  date/price/status UPDATE-overwrite guidance; content previews 200→700 chars.
- Recall precision re-tune: AnchorMatchWeight 8→2, SoftScopeWeight 3.5→0
  (measured junk-injection vectors), DurableFact RecallRank 120→480, floor
  10→14 — durable facts now need two independent lexical matches (or one +
  facet); zero-injection turns are intended behavior (65% of real queries
  had nothing relevant to inject).
- Injection char budget: SessionTuning.MaxRecallInjectedChars (default 2000),
  whole items dropped in rank order, first-ranked always admitted;
  memory_retrieval_final logs injectedChars + droppedByBudget. Schema synced.
- Wired the dead MemoryConfig knobs: RecallTimeoutMs and AutoRecallMaxItems
  now actually feed SessionRecallManager (were hardcoded 300ms/3).
- Retroactive netclaw-dev#1225 data repair in SQLiteMemoryStore.InitializeAsync:
  legacy compaction-boundary rows at recall_mode=searchable (still in the
  automatic pool — measured top polluter) move to manual, idempotently.
- Truth-ups: MemoryIndexContextLayer no longer claims durable_fact-only
  recall; netclaw-memory skill 1.7.0 documents selective recall and that
  zero-injection turns are normal.
- Tests: scenario P09 re-pinned as the paraphrase-gap case (expected empty
  under lexical recall; flips back with hybrid recall), evidence-class
  plumbing test pins policy vs plumbing explicitly, new tests for the char
  budget and the compaction repair. 4,971 tests green; slopwatch clean.
@Aaronontheweb
Aaronontheweb force-pushed the memory-audit-quick-wins branch from 21aa0c1 to f7f2f9a Compare July 3, 2026 18:04
Aaronontheweb added a commit that referenced this pull request Jul 3, 2026
)

* opsx: memory-core-redesign — proposal, design, spec deltas, tasks

OpenSpec change for the memory-core structural redesign, grounded in the
July 2026 audit (PR #1568, tools/recall-research/AUDIT-2026-07.md):

- New capabilities: memory-embeddings (in-process ONNX runtime, pinned
  allowlist provisioning, embed-on-write, loud degradation) and
  memory-maintenance (backfill, ratified consolidation with plan files,
  expiry sweep, memory status).
- netclaw-agent-memory deltas: hybrid recall with an absolute cosine floor
  (zero-injection turns are intended), nominate-by-embedding /
  decide-by-LLM curation through one shared evaluator, lossless merges
  with a deterministic MergeGuard, recall modes that mean what they say
  (searchable leaves the automatic pool), short-lived trace revival,
  tool-use lesson memories with per-tool context injection, checkpoint
  enqueue gating; the never-used hierarchical anchor graph requirement is
  removed.
- tasks.md: six independently-shippable slices with constitution gates.

openspec validate: passing (17 deltas).

* opsx(memory-core-redesign): point evidence references at docs/research/ — research tooling is repo-external (tools/recall-research is gitignored)
Aaronontheweb added a commit that referenced this pull request Jul 3, 2026
…p 3 (#1573)

RunResetAsync publishes CurrentProgressStep = 3 ("Purge complete") and then
evaluates the Task.Delay(CompletionPause, _timeProvider, ct) that registers the
completion-pause timer. Because RunResetAsync runs off the loop, step 3 is the
signal observers gate on to know deletion finished — but it fired BEFORE the
timer it vouches for existed. On a virtualized clock (FakeTimeProvider in tests,
and any consumer advancing the injected TimeProvider) this is a lost wakeup: an
Advance that lands between "step 3 published" and "timer registered" is dropped,
the delay never fires, and the reset hangs forever awaiting a wakeup that will
never come.

Fix: create the completion-pause timer before publishing step 3, then await the
captured task. This makes "step 3 observed" a happens-before guarantee that the
pause timer already exists, so a single clock advance always fires it
deterministically. On the real clock this shifts timer registration a few
microseconds earlier; the visible "Purge complete" dwell is unchanged.

Evidence: hang-dump forensics of CI run 28676578217 (PR #1568, Netclaw.Cli.Tests
blame-hang after 300s). Test FullReset_AfterBothConfirmations_DeletesEverything
was parked at `await vm.ResetTask!.WaitAsync(ct)`; its CompleteResetAsync helper
had exhausted its advance loop (counter i == 10, all virtual-clock advances
spent) and FakeTimeProvider was left holding exactly one orphaned waiter — the
delay registered after the final Advance. Prior commits 850cf8c (#1525) and
eb01b49 (#1536) band-aided the symptom with advance/yield retry loops; this
removes the race at the source.

The test helper CompleteResetAsync is shared by all three sibling tests
(FullReset_AfterBothConfirmations_DeletesEverything,
SetupOnlyReset_DeletesConfigButKeepsMemoryAndSessions,
DaemonStopFailureResult_ShowsStatusAndContinuesReset); it now waits for step 3
then does exactly one advance, deleting the retry loop that only ever masked the
lost wakeup on loaded runners.
@Aaronontheweb Aaronontheweb added the memory Memory formation, recall, curation pipeline label Jul 3, 2026
… bind

The cap is the third line of defense after reasoning suppression and the 10s timeout. 1500 was still marginal in the suppression-ignored window (Qwen-class thinking traces run 800-2000 tokens on this comparison task); truncation there reproduces the empty-reply failure the audit measured. Unemitted tokens are free, so the cap is sized to never be the binding constraint. Becomes config (Memory.Curation.LlmMaxOutputTokens) in memory-core-redesign.
… instead of raw wire fields

The curation-actor reasoning suppression shipped earlier in this PR set raw
provider-dialect fields (chat_template_kwargs.enable_thinking, think) directly
on ChatOptions.AdditionalProperties — a provider-agnostic seam. That was only
correct for the pass-through OpenAiCompatibleChatClient, which forwards
AdditionalProperties verbatim as top-level request JSON for vLLM/llama.cpp/
SGLang. For every other provider the raw keys were wrong in one of two ways:

- Verified by decompiling Microsoft.Extensions.AI.OpenAI 10.6.0
  (OpenAIChatClient.ToOpenAIOptions, OpenAIResponsesChatClient) and
  Anthropic.SDK 12.35.1 (AnthropicChatClient.GetMessageCreateParams): neither
  adapter ever iterates ChatOptions.AdditionalProperties onto the request
  body — both only special-case a "strict" flag for JSON-schema response
  formats. So on OpenAI/Copilot/OpenRouter/VeniceAI/Anthropic the keys were
  silently inert (dropped, not 400s) — dead weight masquerading as a fix.
- Any future adapter that DOES forward extras (or a RawRepresentationFactory
  path) would push unknown top-level params at strict APIs, which reject them
  with 400.

This change replaces the raw fields with a provider-mapped intent key:

- NetclawChatOptionKeys.SuppressReasoning ("netclaw:suppress-reasoning") in
  Netclaw.Configuration — call sites express intent, never wire dialect.
- ILlmProviderPlugin.SuppressionDialect (default None — a genuine domain
  default: strip-unknown is the only correct behavior for a plugin that has
  not declared a serving-stack dialect). Overrides:
  OpenAiCompatibleProviderPlugin → ChatTemplateKwargs,
  OllamaProviderPlugin → OllamaThink. OpenAI, Anthropic, Copilot, OpenRouter,
  VeniceAI inherit None (strip only).
- ReasoningSuppressionChatClient (Netclaw.Daemon, beside
  VendorOptionsChatClient) wraps EVERY client ProviderPluginFactory.Create
  returns: it always removes the intent key, and when true maps it to the
  declared dialect. Mutates options in place per the established
  VendorOptionsChatClient pattern (call sites build fresh ChatOptions per
  call, so no retry hazard).
- MemoryCurationActor.TryLlmEvaluationAsync now sets only the intent key.

The strip guarantee applies to all roles/providers, so any future call site
can use the intent key without knowing which serving stack answers.

Tests: ReasoningSuppressionChatClientTests (dialect mapping, unconditional
strip, false-value strip-without-map, pass-through untouched, null options,
streaming parity), plugin dialect declarations, and a factory regression
guard that every created client carries the wrapper. Daemon.Tests 825/825,
Actors.Tests 2531/2531 green; slopwatch 0 issues; headers verified.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - should recall way fewer memories this way.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

// ever). Unemitted tokens cost nothing, so size this generously;
// it becomes the Memory.Curation.LlmMaxOutputTokens config knob in
// the memory-core-redesign change.
MaxOutputTokens = 4096,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this number was too small originally but it's been improved.

@Aaronontheweb
Aaronontheweb merged commit 9f51884 into netclaw-dev:dev Jul 4, 2026
15 checks passed
@Aaronontheweb
Aaronontheweb deleted the memory-audit-quick-wins branch July 4, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

memory Memory formation, recall, curation pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant