Skip to content

fix(search): widen filtered semantic candidates before pagination - #4960

Merged
thedotmack merged 4 commits into
thedotmack:mainfrom
junnhwan:fix/filtered-semantic-candidates
Oct 22, 2026
Merged

thedotmack merged 4 commits into
thedotmack:mainfrom
junnhwan:fix/filtered-semantic-candidates

Conversation

@junnhwan

Copy link
Copy Markdown
Contributor

Closes #4959

Filtered Smart searches can return a single surviving row and lose a recent supporting memory at semantic rank 148, even though that row is fully indexed and retrievable by ID. Both unified implementations stopped at the first 100 candidates before date and SQLite row filters.

Route SearchManager's semantic selection through the existing Chroma strategy and apply #4510's bounded widening mechanism there. For interactive budgets up to 1600, widen through 100/200/400/800/1600 only while the filtered candidate page is insufficient; stop on raw-document exhaustion, cache accepted and rejected SQLite lookups by category, and hydrate in batches of 500. At the work cap, retain semantic matches and fill remaining slots from filtered keyword results without duplicate IDs. Partial-page keyword additions honor the active date window; the existing empty-category fallback policy is preserved.

Keep final pagination, relevance/date ordering, project/source scopes, and larger explicitly requested corpus budgets. No schema, setting, backend or distributed bundle changes. Source-only, following the maintainer release-bundle guidance.

Validation:

  • Eight new desired-behavior regressions fail on clean main and pass with the fix; a compatibility case retains a 1700-row corpus budget.
  • Current manager/search/native group: 166 pass, 0 fail. Broader SQLite/search/knowledge/HTTP/telemetry checks: 1206 pass, 0 fail.
  • Root/viewer typecheck, source build and hard size guards pass. OpenClaw: 57 pass. Hermes: 36 pass.
  • Real local Chroma + disk SQLite + HTTP + MCP/context restores the same 20 result IDs and supporting narrative after adding 140 out-of-window memories; all indexing completes before searching.
  • The initial full run is not green: 9898 pass, 54 skip, 16 fail. Nine stale manager signature/private-helper assertions were then migrated to the public/shared implementation and their group now passes. Of the remaining seven failures in unchanged watcher/signal/browser files, three watcher and two browser test names also fail on clean main; the other watcher and SIGTERM cases pass focused reruns on both revisions. No full-suite rerun after that test-only migration or current-head remote CI is claimed.

The finite work budget does not guarantee semantic matches beyond its prefix or bound backend wall-clock latency; #4938's timeout/timeline issue is separate. The real-backend fixture is synthetic, not a production recall or task-quality benchmark. PostgreSQL and installed user caches were not modified.

AI assistance: investigated, implemented and verified with Codex; commits include DCO sign-off.

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues remain.

Summary

Routes semantic searches through the shared Chroma strategy and widens candidates before pagination.

  • Filtered semantic searches keep looking past early candidates until each requested page is filled.

The earlier finding about full categories hiding sparse categories is fixed: each requested category must now have enough rows before widening stops. No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Semantic search] --> B[Fetch candidate window]
  B --> C[Apply date and row filters]
  C --> D{Every requested category full?}
  D -->|Yes| G[Apply final pagination]
  D -->|No| E{More candidates within budget?}
  E -->|Yes| B
  E -->|Work cap reached| F[Keep semantic rows and add keyword matches]
  E -->|Documents exhausted| H[Keep existing empty-category fallback]
  F --> G
  H --> G
Loading

Reviews (3) · Last reviewed commit: "test(search): keep work-cap fixtures ins..." · Reviewed by Greptile

Vivek-Yarra added a commit to Vivek-Yarra/claude-mem that referenced this pull request Oct 22, 2026
…candidates before pagination

Squashed from thedotmack/claude-mem pull/4960 (source only, CHANGELOG excluded).

Claude-Session: https://claude.ai/code/session_01J69ZRK5a82yX2rX1vVhjeo
Seed 2048 excluded rows instead of 8192: just past the 1600-candidate
semantic work cap, which is all the scenario needs. Give the two
multi-fixture work-cap tests an explicit 20s timeout for slower runners.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QVujqk6gvQYaoymRd7Aynn
@thedotmack
thedotmack merged commit d8efa52 into thedotmack:main Oct 22, 2026
13 of 14 checks passed
@thedotmack

Copy link
Copy Markdown
Owner

Merged, thank you @junnhwan! Filtered semantic searches (date range, type, concepts, files, or the default recency window) no longer lose in-scope matches that ranked past Chroma's first 100 candidates: both search paths now share ChromaSearchStrategy, which widens the window up to 1600 only while a page is short, and at that cap fills the rest of the page from keyword matches in the same window. Removing the duplicate performChromaSemanticSearch along the way was a welcome cleanup.

What we changed:

  • Your new work-cap test was timing out in CI (5008 ms). It now seeds 2048 excluded rows instead of 8192 (just past the 1600 cap, which is all the scenario needs), and the two multi-fixture work-cap tests have an explicit 20 s timeout.
  • Checked the recency window: the partial-page fill and filterByRecency both read the one CLAUDE_MEM_SEARCH_RECENCY_DAYS value that SearchManager passes to the orchestrator and on to the strategy, so they can't drift apart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filtered semantic search loses recent memories behind out-of-scope candidates

2 participants