Skip to content

fix(sqlite): preserve bounded matches in topic-focused FTS recall - #4956

Merged
thedotmack merged 2 commits into
thedotmack:mainfrom
junnhwan:fix/preserve-matchable-fts-terms
Oct 22, 2026
Merged

thedotmack merged 2 commits into
thedotmack:mainfrom
junnhwan:fix/preserve-matchable-fts-terms

Conversation

@junnhwan

Copy link
Copy Markdown
Contributor

Fixes #4955.

A natural-language question can stop finding a stored decision after a second same-topic memory is added. On bbb3379, What was our Redis connection pool limit? changes from one result to zero because the 30% rule discards every indexed content term and keeps only absent question words. The same missing evidence reaches SQLite-only semantic context retrieval. The original report and public-path reproduction are in this #4829 comment.

This changes only short/medium any-term selection when the existing selected list contains no token with positive estimated document frequency. It admits excluded matching tokens, least frequent first, while their summed candidate costs stay within 1,024. ASCII tokens use their document-frequency upper bound; non-ASCII tokens are charged the whole table because the existing approximate folding can disagree with FTS5 unicode61 (for example, with Greek accents). The cost sum bounds the union of documents newly admitted by this correction; it is not a latency guarantee. If no token fits, selection stays unchanged. This is deliberately conservative for non-ASCII terms: a store larger than 1,024 rows will not admit one through this correction, even if its true posting list is small. Selections with a positive frequency estimate and long-query handling remain unchanged.

The #4804 performance protections remain: 40/60 term caps; the >=600-character policy; unchanged selections that already contain a token with positive estimated frequency; AND/substring priority; filtering, pagination and FTS ranking; and per-term lookups through the existing temporary vocabulary and query-local cache. No schema, setting, backend, extra search attempt or provider change.

The budget mechanism was explored in the closed, unmerged #4831, including its review about absent terms crowding out matching terms. This is a narrow follow-up to #4829's merged selector, rather than a replacement of the measured policy. The scope and budget are submitted for maintainer review.

Validation on macOS arm64, Bun 1.4.2 / Node 24.19.0:

  • Both added test files expose the baseline defect on bbb3379; all 17 new tests pass with the fix. Coverage includes memory growth, short/medium queries, summaries, read-only access, filters, unchanged matching selections, expensive-term refusal and Unicode candidate accounting.
  • bun test tests/services/sqlite tests/sqlite tests/worker/search tests/worker/http/routes tests/context: 1,438 pass, 1 skip, 0 fail (5,032 assertions, 202 files).
  • npm run typecheck and npm run build: PASS, including worker/MCP bundle-size guardrails.
  • bun test openclaw: 57 pass, 0 fail. python3 -B -m unittest discover -s tests/hermes -v: 36 tests, OK.
  • bun test tests: 9,917 pass, 54 skip, 5 fail (44,440 assertions, 964 files; 836.39 seconds). All five failure signatures reproduce on unchanged bbb3379 under the same environment: three failed-turn retry assertions in tests/transcripts/watcher-anchor-retry.test.ts (file and isolated runs), and native Enter/Space 30-second timeouts in tests/ui/viewer/session-keyboard-actions.browser.test.ts (Enter in the file run, Space in an isolated run). Those paths are unchanged. The full suite is not green; this comparison does not diagnose the underlying failures.
  • Repeated the real disk-SQLite/HTTP/MCP-stdio/semantic-context probe against committed 17e1775: all six stages retain the original 12-per-process / 48-total supporting text. After growth, search/context counts are 2/2 in the small case and 40/20 in the 100-row case. Keyword and selected-ID controls remain successful. No model, live Chroma/Postgres or real-user database was used.
  • Equal-workload controlled SQLite comparison: 14 synthetic fixtures, 15 warm repetitions, including 100k-row stores, with execution order rotated. The 1,024 budget restores 2/2, 100/40, 1,000/400 and 1,000/1,000 cases; it refuses a term present in all 100k rows. Existing rare-term and long-query expressions/results stay unchanged. These fixtures are ASCII; Unicode is checked separately by regressions. Public-search medians were 0.422→0.605 ms for 1,000/400 (empty→evidence) and 4.550→4.594 ms for 100k/all-topic (still empty). This spends bounded work for recall; it is not a speedup, production SLA, general recall measurement or model-answer evaluation.

The 1,024 cap is an explicit conservative candidate-work limit: 128 fails to recover the 1,000/400 case, 512 retains one matching term there, and 1,024 permits two (ASCII document-frequency cost sum 800). Representative user stores may justify later tuning; the tests do not establish an optimal production value.

AI assistance: Codex assisted with source/history investigation, implementation, regression tests, synthetic comparison and this description. The reported validation was run locally; remote CI, merge and release are separate outcomes.

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous Unicode match loss is fixed, and no new blocking issue was found.

Summary

This PR restores bounded matching terms for short and medium SQLite searches when the existing selection has no positive frequency estimate.

  • Short and medium fallback searches can add matching topic words when their first selection has no estimated hits.

Reviews (2) · Last reviewed commit: "fix(sqlite): retain selected terms durin..." · Reviewed by Greptile

Comment thread src/services/sqlite/SessionSearch.ts Resolved
Vivek-Yarra added a commit to Vivek-Yarra/claude-mem that referenced this pull request Oct 22, 2026
… in topic-focused FTS recall

Squashed from thedotmack/claude-mem pull/4956 (source only, CHANGELOG excluded).

Claude-Session: https://claude.ai/code/session_01J69ZRK5a82yX2rX1vVhjeo
@thedotmack
thedotmack merged commit e5c8105 into thedotmack:main Oct 22, 2026
8 checks passed
@thedotmack

Copy link
Copy Markdown
Owner

Merged, thank you @junnhwan! Keyword recall no longer comes back empty in a topic-focused store: when every matching word is common enough to be filtered out and only absent question words remain, the least frequent matching words come back in under a fixed 1024-document budget, so the search stays bounded. The 17 regression tests and the perf evidence made this easy to trust, and keeping the original Unicode selection after Greptile's note was the right call. No changes from our side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite recall loses every match when related memories become common in the store

3 participants