Skip to content

fix(memory): relevance-gate cold-start — keep-warm ticks, envelope-derived gate budget (canary finding)#1608

Merged
Aaronontheweb merged 1 commit into
netclaw-dev:feature/memory-embeddingsfrom
Aaronontheweb:fix/relevance-gate-cold-start
Jul 9, 2026
Merged

fix(memory): relevance-gate cold-start — keep-warm ticks, envelope-derived gate budget (canary finding)#1608
Aaronontheweb merged 1 commit into
netclaw-dev:feature/memory-embeddingsfrom
Aaronontheweb:fix/relevance-gate-cold-start

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

Production canary (~/.netclaw/logs/daemon-2026-07-09.log) caught two
memory_recall_gate_degraded events with reason
score_failed:TaskCanceledException, both in scheduled-reminder sessions
firing after an idle period:

  • 05:06:52.718 — session D0AC6CKBK5K/1778733767.940429
  • 12:00:00.422 — session reminder/vendor-price-alert/1783598400066

Root cause (verified against the log, not assumed): in both events, the
gap from memory_retrieval_request_plan to memory_recall_gate_degraded
was 318ms and 339ms respectively — already past the entire 300ms
Memory.RecallTimeoutMs envelope before the relevance gate's cross-encoder
even started scoring. Both turns show mode=hybrid (the query embed
succeeded), so the whole pipeline — plan → candidate selection → query embed
→ hybrid fusion → gate — was slow, consistent with a cold ONNX session
(paged-out weights after the idle gap) plus host CPU contention at
reminder-fire time, not a per-call latency regression against design D5's
reference-box measurement (~11ms p50 / ~35ms p95 for 3 pairs), which still
holds on a warm session.

Re-verified prior "3–12% coverage-gap" claim: the two events' own
memory_recall_coverage_gap lines show gapCandidates=3/totalCandidates=61
(4.9%) and gapCandidates=7/totalCandidates=57 (12.3%) — these are the
pre-existing, already-designed-for candidate-pool coverage-gap counters
(documents not yet backfilled with an embedding for the current model,
memory-core-redesign Slice 4 / design D6 gap-repair), not gate failures.
Confirmed unrelated to the timeout finding.

The fix (three parts)

  1. Keep-warmEmbeddingWarmupHostedService now runs a periodic
    keep-warm loop (every 5 minutes, while Memory.Embeddings.Enabled) that
    re-exercises both ONNX sessions with one tiny embed
    (EmbeddingPurpose.RetrievalQuery) and one tiny 1-pair cross-encoder
    score, so an idle gap never lets either session's working set page out
    entirely. Built on PeriodicTimer over the injected TimeProvider — the
    same virtualizable pattern McpReconnectionService already uses — never
    throws out of a tick (rate-limited Debug log on failure), skips whichever
    side is currently unavailable, no-ops when embeddings are disabled, and
    stops cleanly via the existing IHostedService.StopAsync +
    CancellationTokenSource pattern.

  2. BudgetSQLiteMemoryRecallCoordinator's relevance-gate sub-budget
    is now min(RelevanceGateSubBudgetMs, time remaining in the outer RecallTimeoutMs envelope) instead of a fixed value; the ceiling itself
    is raised from 60ms to 120ms. The outer linked CTS remains the hard cap,
    so this can never blow the 300ms envelope — a turn with headroom gets
    more slack than before, while a turn where earlier stages already
    consumed the envelope degrades immediately instead of assuming a fixed
    budget is always affordable. Same degraded-path semantics on
    timeout/failure.

  3. Observabilitymemory_retrieval_final now logs gateElapsedMs
    (gate scoring latency regardless of outcome), and
    memory_recall_gate_degraded now logs elapsedMs too, so soak data can
    quantify margins against the new ceiling.

Also updated openspec/changes/memory-relevance-gate/design.md's 60ms
figures to 120ms (envelope-clamped) with a canary-finding note, and marked
the "combined sub-budget worst-case latency" Open Question as resolved by
this same finding. netclaw-memory skill does not mention the 60ms figure,
so it was left unchanged (checked, no hit).

Test plan

  • dotnet build Netclaw.slnx — clean, 0 warnings/errors
  • dotnet test on Netclaw.Actors.Tests — 2671 passed
  • dotnet test on Netclaw.Daemon.Tests — 848 passed
  • dotnet test on Netclaw.Embeddings.Tests — 48 passed
  • dotnet slopwatch analyze — 0 issues (added a scoped
    [SlopwatchSuppress("SW004", ...)] on the new test-only
    DelayedRelevanceScorer fake's intentional latency simulation)
  • ./scripts/Add-FileHeaders.ps1 -Verify — all files have headers
  • New keep-warm tests: tick fires on schedule and calls embedder+scorer
    exactly once per tick; swallows scorer exceptions without throwing;
    skips whichever side is unavailable; no ticks when embeddings
    disabled; stops cleanly on cancellation — all FakeTimeProvider-driven,
    no Thread.Sleep/Task.Delay in test orchestration
  • New budget tests: an envelope-exhausted RecallTimeoutMs + a
    slower-than-instant-but-under-the-ceiling fake scorer degrades to the
    floor's unfiltered result; the same scorer with a fresh/generous
    envelope runs to completion and its score is honored
  • Updated the existing sub-budget-timeout test's stale "~60ms" comment

…rived gate budget (canary finding)

Production canary caught two memory_recall_gate_degraded events
(score_failed:TaskCanceledException) in scheduled-reminder sessions waking
from an idle period. Root cause: cold ONNX sessions (paged-out weights) plus
host CPU contention at reminder-fire time pushed total turn latency (plan ->
candidate selection -> query embed -> hybrid fusion -> gate) past the entire
300ms RecallTimeoutMs envelope before the gate even started scoring, so its
own sub-budget timer never got a chance to fire on its own terms -- the
OUTER ct was already cancelled by the time ScoreAsync threw. Both events
show mode=hybrid (query embed succeeded, just slow), so this is a cold-start
problem across the whole pipeline, not a per-call latency regression against
design D5's reference-box measurement.

Re-verified the "partial candidate-scoring coverage gaps of 3-12%" prior
claim against the same canary log window: gapCandidates/totalCandidates in
the two events are 3/61 (4.9%) and 7/57 (12.3%) -- these are
memory_recall_coverage_gap counters (pre-existing candidate-pool coverage
gaps for un-backfilled documents, memory-core-redesign Slice 4 gap-repair
design D6), not gate failures. Confirmed: unrelated to this fix.

Three-part fix:

1. Keep-warm: EmbeddingWarmupHostedService now runs a periodic keep-warm
   loop (every 5 minutes, while embeddings are enabled) that re-exercises
   both ONNX sessions with a tiny embed + tiny 1-pair CE score, so an idle
   gap never lets either session's working set page out entirely. Built on
   PeriodicTimer over the injected TimeProvider (same virtualizable pattern
   McpReconnectionService already uses), never throws out of a tick
   (rate-limited Debug log on failure), skips whichever side is unavailable,
   no-ops when embeddings are disabled, and stops cleanly via the existing
   IHostedService.StopAsync/CancellationTokenSource pattern.

2. Budget: SQLiteMemoryRecallCoordinator's relevance-gate sub-budget is now
   min(RelevanceGateSubBudgetMs, time remaining in the outer RecallTimeoutMs
   envelope) instead of a fixed value; the ceiling itself is raised from
   60ms to 120ms. The outer linked CTS remains the hard cap, so a turn with
   headroom gets more slack than before, while a turn where earlier stages
   already consumed the envelope degrades immediately instead of assuming a
   fixed budget is always affordable. Same degraded-path semantics on
   timeout/failure.

3. Observability: memory_retrieval_final now logs gateElapsedMs (gate
   scoring latency regardless of outcome), and memory_recall_gate_degraded
   now logs elapsedMs too, so soak data can quantify margins against the new
   ceiling.

Updated openspec/changes/memory-relevance-gate/design.md's 60ms figures to
120ms (envelope-clamped) with a canary-finding note; the Open Questions
entry about combined sub-budget worst-case latency is marked resolved by
this same finding. netclaw-memory skill does not mention the 60ms figure, so
it is unchanged.

Tests: keep-warm tick fires on schedule / calls embedder+scorer exactly once
per tick / swallows scorer exceptions / skips unavailable sides / no ticks
when disabled / stops cleanly on cancellation (FakeTimeProvider-driven, no
sleeps). Budget: envelope-exhausted vs. envelope-with-headroom behavioral
tests using a real-delay fake scorer, proving the applied sub-budget is
smaller than the fixed ceiling when the outer envelope is nearly spent.
Updated the existing sub-budget-timeout test's stale "~60ms" comment.
@Aaronontheweb
Aaronontheweb merged commit 53fa8e9 into netclaw-dev:feature/memory-embeddings Jul 9, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant