Skip to content

fix(chroma): keep doc_type out of the semantic search fallback query - #5355

Merged
thedotmack merged 6 commits into
thedotmack:mainfrom
0xABCD01:fix/5354-chroma-doc-type-fallback
Nov 5, 2026
Merged

thedotmack merged 6 commits into
thedotmack:mainfrom
0xABCD01:fix/5354-chroma-doc-type-fallback

Conversation

@0xABCD01

@0xABCD01 0xABCD01 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #5354

When the over-fetch leaves fewer than limit rows and the narrowed row-id fallback cannot run — the scope has more than 5,000 rows, the listing fails, or the caller passes no listing (HybridSearchStrategy) — queryChroma sends the caller's full where to Chroma. Every search clause carries doc_type, and on a large store that member matches most of the collection (2.85M of 3.09M documents in #5354): chromadb evaluates it in full before intersecting, so one such search blocks chroma-mcp for minutes.

The fallback now strips the doc_type clause before sending and applies the full clause client-side to the returned metadata — the same split #4687 uses for fragment lookups. The remaining clauses (project scoping, platform) still go to Chroma. The stripped query over-fetches with the fast path's factor and cap (20×, 2,000); a short window proves the match set complete, and a full window with too few survivors still falls through to the plain where, so the top-limit guarantee is unchanged. A doc_type leaf under an $or is one side of a union and cannot be dropped without changing the match set, so that shape keeps the plain query. The same holds for an $or branch whose conjuncts are all doc_type: emptying it would drop that side of the union, so that shape blocks the rewrite and keeps the plain query too.

Tests: tests/services/sync/chroma-query-postfilter.test.ts gains a #5354 block pinning the sent clauses (no doc_type in any call, residual project/platform clauses preserved, results identical to the plain filter, the $or shapes unchanged), and three #5091 tests that expected the plain fallback now expect the doc_type-free one. bun test tests/services/sync → 235 pass / 0 fail (230 on main); the search suites (tests/worker/search, search-manager, category arrays, corpus scope, scope row ids) → 197 pass / 0 fail, same as main. bun run typecheck:root clean.

One possible follow-up, not included here: the chunked sqlite_id lookup past the 5,000-row cap floated in #5354 ("may hold up better"). It changes the ChromaFallbackRowIds contract and multiplies MCP calls per search; this PR is limited to keeping doc_type out of the query the fallback sends.

The committed bundles are not rebuilt, as in other contributor PRs.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge, with non-blocking extra-query costs remaining.

Findings

  1. P2 Search categories share wrong counts ▶
  2. P2 Searches fetch candidates twice ▶

Summary

Adds a fallback that leaves doc_type out of the Chroma query and applies the full filter to returned metadata.

  • Chroma fallback searches leave safe doc_type filters out of the query.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Unfiltered over-fetch] --> B{Enough matching rows?}
  B -->|Yes| R[Return rows]
  B -->|No| C[Try scoped row IDs]
  C -->|Answered| R
  C -->|Unavailable or insufficient| D{Can remove doc_type safely?}
  D -->|No| P[Run full filter]
  D -->|Yes| E[Query remaining clauses]
  E --> F{Enough rows or short window?}
  F -->|Yes| R
  F -->|No| P
Loading

Reviews (3) · Last reviewed commit: "fix(chroma): keep the doc_type-free fall..." · Reviewed by Greptile

Comment on lines +2068 to +2070
const fallback = await this.queryDocTypeFreeFallback(
runQuery, limit, predicate, stripped.clause, progress
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Searches fetch candidates twice

When a doc_type-only filter misses the fast path, this branch repeats the same unfiltered query and applies the same predicate. For limit <= 2000, the window size is identical too. The added test explicitly expects both calls.

Reuse the first window when stripping leaves no clause. A short window already contains every match; a full window with too few matches already requires the plain query. This avoids an extra Chroma call and embedding request.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

…d exhaustion (thedotmack#5354)

The fallback's null return -- a full stripped window holding too few matches, so only the plain query can prove the top results -- and its progress.exhausted reporting for full and short windows had no test coverage. Logic unchanged.
thedotmack and others added 3 commits November 5, 2026 00:48
…udget

A search widens its limit in steps, and past the over-fetch cap every step
asks the same stripped window. A window that already held too few matches
cost another 2000 fragments per step, which pushed the observation search
past its 13100 fragment budget (14400 in the budget test). Remember those
windows per query, clause and window size and skip straight to the plain
filter.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@thedotmack

Copy link
Copy Markdown
Owner

Hi @0xABCD01, I pushed two commits to your branch so this can land:

  • Merged main.
  • CI failed because the new fallback pushed tests/worker/observation-semantic-filters.test.ts past its work budget: 14400 requested fragments against a limit of 13100. Search widens its limit in steps, and past the over-fetch cap each step asked the same stripped window again. The fallback now remembers a full window that held too few matches, per query, filter and window size, and goes straight to the plain filter. I added a test that fails without this change, and left your fallback logic as it was.

Comment on lines +2195 to +2197
const windowKey = JSON.stringify([query, where, nResults]);
const knownMatches = this.thinDocTypeFreeWindows.get(windowKey);
if (knownMatches !== undefined && knownMatches < limit) return null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Search categories share wrong counts

thinDocTypeFreeWindows stores a count after applying the full predicate, but its key includes only the stripped clause. Summary and observation searches with the same query, project and window size therefore share a count despite keeping different documents.

A sparse summary search can make a later observation search skip a window that would satisfy it. Results remain correct, but the search unnecessarily sends the expensive full doc_type filter. Include the original filter in the cache key.

@thedotmack
thedotmack merged commit f729e47 into thedotmack:main Nov 5, 2026
28 of 35 checks passed
@thedotmack

Copy link
Copy Markdown
Owner

Merged, thank you @0xABCD01! When semantic search falls back on a large project, it no longer sends the broad doc_type clause to Chroma. It filters that clause on the client and only uses the plain filter when the window cannot prove the top matches. Your handling of $or shapes and your tests for what actually gets sent made this easy to check.

This was checked with mocked Chroma in tests and in CI, not against a store of the size in #5354. It does not change the separate sqlite_id lookup limit past 5,000 rows that you called out as a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Semantic search fallback still sends the broad doc_type filter to Chroma; on a 3M-document store each such search blocks chroma-mcp for minutes

2 participants