fix(memory): relevance-gate cold-start — keep-warm ticks, envelope-derived gate budget (canary finding)#1608
Merged
Aaronontheweb merged 1 commit intoJul 9, 2026
Conversation
…rived gate budget (canary finding) Production canary caught two memory_recall_gate_degraded events (score_failed:TaskCanceledException) in scheduled-reminder sessions waking from an idle period. Root cause: cold ONNX sessions (paged-out weights) plus host CPU contention at reminder-fire time pushed total turn latency (plan -> candidate selection -> query embed -> hybrid fusion -> gate) past the entire 300ms RecallTimeoutMs envelope before the gate even started scoring, so its own sub-budget timer never got a chance to fire on its own terms -- the OUTER ct was already cancelled by the time ScoreAsync threw. Both events show mode=hybrid (query embed succeeded, just slow), so this is a cold-start problem across the whole pipeline, not a per-call latency regression against design D5's reference-box measurement. Re-verified the "partial candidate-scoring coverage gaps of 3-12%" prior claim against the same canary log window: gapCandidates/totalCandidates in the two events are 3/61 (4.9%) and 7/57 (12.3%) -- these are memory_recall_coverage_gap counters (pre-existing candidate-pool coverage gaps for un-backfilled documents, memory-core-redesign Slice 4 gap-repair design D6), not gate failures. Confirmed: unrelated to this fix. Three-part fix: 1. Keep-warm: EmbeddingWarmupHostedService now runs a periodic keep-warm loop (every 5 minutes, while embeddings are enabled) that re-exercises both ONNX sessions with a tiny embed + tiny 1-pair CE score, so an idle gap never lets either session's working set page out entirely. Built on PeriodicTimer over the injected TimeProvider (same virtualizable pattern McpReconnectionService already uses), never throws out of a tick (rate-limited Debug log on failure), skips whichever side is unavailable, no-ops when embeddings are disabled, and stops cleanly via the existing IHostedService.StopAsync/CancellationTokenSource pattern. 2. Budget: SQLiteMemoryRecallCoordinator's relevance-gate sub-budget is now min(RelevanceGateSubBudgetMs, time remaining in the outer RecallTimeoutMs envelope) instead of a fixed value; the ceiling itself is raised from 60ms to 120ms. The outer linked CTS remains the hard cap, so a turn with headroom gets more slack than before, while a turn where earlier stages already consumed the envelope degrades immediately instead of assuming a fixed budget is always affordable. Same degraded-path semantics on timeout/failure. 3. Observability: memory_retrieval_final now logs gateElapsedMs (gate scoring latency regardless of outcome), and memory_recall_gate_degraded now logs elapsedMs too, so soak data can quantify margins against the new ceiling. Updated openspec/changes/memory-relevance-gate/design.md's 60ms figures to 120ms (envelope-clamped) with a canary-finding note; the Open Questions entry about combined sub-budget worst-case latency is marked resolved by this same finding. netclaw-memory skill does not mention the 60ms figure, so it is unchanged. Tests: keep-warm tick fires on schedule / calls embedder+scorer exactly once per tick / swallows scorer exceptions / skips unavailable sides / no ticks when disabled / stops cleanly on cancellation (FakeTimeProvider-driven, no sleeps). Budget: envelope-exhausted vs. envelope-with-headroom behavioral tests using a real-delay fake scorer, proving the applied sub-budget is smaller than the fixed ceiling when the outer envelope is nearly spent. Updated the existing sub-budget-timeout test's stale "~60ms" comment.
Aaronontheweb
merged commit Jul 9, 2026
53fa8e9
into
netclaw-dev:feature/memory-embeddings
15 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production canary (
~/.netclaw/logs/daemon-2026-07-09.log) caught twomemory_recall_gate_degradedevents with reasonscore_failed:TaskCanceledException, both in scheduled-reminder sessionsfiring after an idle period:
05:06:52.718— sessionD0AC6CKBK5K/1778733767.94042912:00:00.422— sessionreminder/vendor-price-alert/1783598400066Root cause (verified against the log, not assumed): in both events, the
gap from
memory_retrieval_request_plantomemory_recall_gate_degradedwas 318ms and 339ms respectively — already past the entire 300ms
Memory.RecallTimeoutMsenvelope before the relevance gate's cross-encodereven started scoring. Both turns show
mode=hybrid(the query embedsucceeded), so the whole pipeline — plan → candidate selection → query embed
→ hybrid fusion → gate — was slow, consistent with a cold ONNX session
(paged-out weights after the idle gap) plus host CPU contention at
reminder-fire time, not a per-call latency regression against design D5's
reference-box measurement (~11ms p50 / ~35ms p95 for 3 pairs), which still
holds on a warm session.
Re-verified prior "3–12% coverage-gap" claim: the two events' own
memory_recall_coverage_gaplines showgapCandidates=3/totalCandidates=61(4.9%) and
gapCandidates=7/totalCandidates=57(12.3%) — these are thepre-existing, already-designed-for candidate-pool coverage-gap counters
(documents not yet backfilled with an embedding for the current model,
memory-core-redesign Slice 4 / design D6 gap-repair), not gate failures.
Confirmed unrelated to the timeout finding.
The fix (three parts)
Keep-warm —
EmbeddingWarmupHostedServicenow runs a periodickeep-warm loop (every 5 minutes, while
Memory.Embeddings.Enabled) thatre-exercises both ONNX sessions with one tiny embed
(
EmbeddingPurpose.RetrievalQuery) and one tiny 1-pair cross-encoderscore, so an idle gap never lets either session's working set page out
entirely. Built on
PeriodicTimerover the injectedTimeProvider— thesame virtualizable pattern
McpReconnectionServicealready uses — neverthrows out of a tick (rate-limited Debug log on failure), skips whichever
side is currently unavailable, no-ops when embeddings are disabled, and
stops cleanly via the existing
IHostedService.StopAsync+CancellationTokenSourcepattern.Budget —
SQLiteMemoryRecallCoordinator's relevance-gate sub-budgetis now
min(RelevanceGateSubBudgetMs, time remaining in the outer RecallTimeoutMs envelope)instead of a fixed value; the ceiling itselfis raised from 60ms to 120ms. The outer linked CTS remains the hard cap,
so this can never blow the 300ms envelope — a turn with headroom gets
more slack than before, while a turn where earlier stages already
consumed the envelope degrades immediately instead of assuming a fixed
budget is always affordable. Same degraded-path semantics on
timeout/failure.
Observability —
memory_retrieval_finalnow logsgateElapsedMs(gate scoring latency regardless of outcome), and
memory_recall_gate_degradednow logselapsedMstoo, so soak data canquantify margins against the new ceiling.
Also updated
openspec/changes/memory-relevance-gate/design.md's 60msfigures to 120ms (envelope-clamped) with a canary-finding note, and marked
the "combined sub-budget worst-case latency" Open Question as resolved by
this same finding.
netclaw-memoryskill does not mention the 60ms figure,so it was left unchanged (checked, no hit).
Test plan
dotnet build Netclaw.slnx— clean, 0 warnings/errorsdotnet testonNetclaw.Actors.Tests— 2671 passeddotnet testonNetclaw.Daemon.Tests— 848 passeddotnet testonNetclaw.Embeddings.Tests— 48 passeddotnet slopwatch analyze— 0 issues (added a scoped[SlopwatchSuppress("SW004", ...)]on the new test-onlyDelayedRelevanceScorerfake's intentional latency simulation)./scripts/Add-FileHeaders.ps1 -Verify— all files have headersexactly once per tick; swallows scorer exceptions without throwing;
skips whichever side is unavailable; no ticks when embeddings
disabled; stops cleanly on cancellation — all
FakeTimeProvider-driven,no
Thread.Sleep/Task.Delayin test orchestrationRecallTimeoutMs+ aslower-than-instant-but-under-the-ceiling fake scorer degrades to the
floor's unfiltered result; the same scorer with a fresh/generous
envelope runs to completion and its score is honored