Repository navigation
fix(sqlite): preserve bounded matches in topic-focused FTS recall - #4956
Conversation
Signed-off-by: hwan <[email protected]>
[Medium impact] The PR appears safe to merge; the previous Unicode match loss is fixed, and no new blocking issue was found. SummaryThis PR restores bounded matching terms for short and medium SQLite searches when the existing selection has no positive frequency estimate.
Reviews (2) · Last reviewed commit: "fix(sqlite): retain selected terms durin..." · Reviewed by Greptile |
Signed-off-by: hwan <[email protected]>
… in topic-focused FTS recall Squashed from thedotmack/claude-mem pull/4956 (source only, CHANGELOG excluded). Claude-Session: https://claude.ai/code/session_01J69ZRK5a82yX2rX1vVhjeo
|
Merged, thank you @junnhwan! Keyword recall no longer comes back empty in a topic-focused store: when every matching word is common enough to be filtered out and only absent question words remain, the least frequent matching words come back in under a fixed 1024-document budget, so the search stays bounded. The 17 regression tests and the perf evidence made this easy to trust, and keeping the original Unicode selection after Greptile's note was the right call. No changes from our side. |
Fixes #4955.
A natural-language question can stop finding a stored decision after a second same-topic memory is added. On
bbb3379,What was our Redis connection pool limit?changes from one result to zero because the 30% rule discards every indexed content term and keeps only absent question words. The same missing evidence reaches SQLite-only semantic context retrieval. The original report and public-path reproduction are in this #4829 comment.This changes only short/medium any-term selection when the existing selected list contains no token with positive estimated document frequency. It admits excluded matching tokens, least frequent first, while their summed candidate costs stay within 1,024. ASCII tokens use their document-frequency upper bound; non-ASCII tokens are charged the whole table because the existing approximate folding can disagree with FTS5 unicode61 (for example, with Greek accents). The cost sum bounds the union of documents newly admitted by this correction; it is not a latency guarantee. If no token fits, selection stays unchanged. This is deliberately conservative for non-ASCII terms: a store larger than 1,024 rows will not admit one through this correction, even if its true posting list is small. Selections with a positive frequency estimate and long-query handling remain unchanged.
The #4804 performance protections remain: 40/60 term caps; the >=600-character policy; unchanged selections that already contain a token with positive estimated frequency; AND/substring priority; filtering, pagination and FTS ranking; and per-term lookups through the existing temporary vocabulary and query-local cache. No schema, setting, backend, extra search attempt or provider change.
The budget mechanism was explored in the closed, unmerged #4831, including its review about absent terms crowding out matching terms. This is a narrow follow-up to #4829's merged selector, rather than a replacement of the measured policy. The scope and budget are submitted for maintainer review.
Validation on macOS arm64, Bun 1.4.2 / Node 24.19.0:
bbb3379; all 17 new tests pass with the fix. Coverage includes memory growth, short/medium queries, summaries, read-only access, filters, unchanged matching selections, expensive-term refusal and Unicode candidate accounting.bun test tests/services/sqlite tests/sqlite tests/worker/search tests/worker/http/routes tests/context: 1,438 pass, 1 skip, 0 fail (5,032 assertions, 202 files).npm run typecheckandnpm run build: PASS, including worker/MCP bundle-size guardrails.bun test openclaw: 57 pass, 0 fail.python3 -B -m unittest discover -s tests/hermes -v: 36 tests, OK.bun test tests: 9,917 pass, 54 skip, 5 fail (44,440 assertions, 964 files; 836.39 seconds). All five failure signatures reproduce on unchangedbbb3379under the same environment: three failed-turn retry assertions intests/transcripts/watcher-anchor-retry.test.ts(file and isolated runs), and native Enter/Space 30-second timeouts intests/ui/viewer/session-keyboard-actions.browser.test.ts(Enter in the file run, Space in an isolated run). Those paths are unchanged. The full suite is not green; this comparison does not diagnose the underlying failures.17e1775: all six stages retain the original 12-per-process / 48-total supporting text. After growth, search/context counts are 2/2 in the small case and 40/20 in the 100-row case. Keyword and selected-ID controls remain successful. No model, live Chroma/Postgres or real-user database was used.The 1,024 cap is an explicit conservative candidate-work limit: 128 fails to recover the 1,000/400 case, 512 retains one matching term there, and 1,024 permits two (ASCII document-frequency cost sum 800). Representative user stores may justify later tuning; the tests do not establish an optimal production value.
AI assistance: Codex assisted with source/history investigation, implementation, regression tests, synthetic comparison and this description. The reported validation was run locally; remote CI, merge and release are separate outcomes.