Repository navigation
perf: skip full-snapshot row-id prefilters and expose loader metrics - #9471
Conversation
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
The fragment-ID guard is limited to unfiltered scopes that exactly cover the snapshot, while explicit fragments remain available for pinned-segment and unindexed fallback and deletion, overlay, external-mask, and predicate filtering stay intact. The residual risk is a deliberate 4-bit IVF_PQ candidate shift: taking the true unfiltered path uses the existing bulk scorer with extra distance-table quantization, so recall-sensitive deployments should benchmark this shape and increase refinement if needed.
lance-gatekeeper
Bot
added
K-approved
yanghua
left a comment
yanghua
left a comment
There was a problem hiding this comment.
The recall regression is not caused by newly introduced scoring logic in this PR. It is an existing difference in the unfiltered batch-scoring path. This PR removes a redundant full-row prefilter, which causes some queries that previously took the filtered path to use that existing batch path instead.
Fixing the scoring discrepancy itself should therefore be handled in the batch/PQ implementation and is likely outside the scope of this PR. However, because this PR expands the set of queries exposed to that behavior, the compatibility impact should still be documented and explicitly accepted here.
LuciferYang
left a comment
LuciferYang
left a comment
There was a problem hiding this comment.
Re-reviewed this against the #9460 review now that it targets main. The release-branch concern from that review is resolved: this is the main counterpart, so nothing has to be forward-ported afterwards.
Three comments inline, all about coverage and instrumentation rather than the optimization itself. The planner change looks sound to me: under full coverage self.fragments only reaches get_fragments_as_bitmap, which is id-equal to the manifest bitmap, so the new branch is the path the pre-existing is_none() case already took, and deletion vectors, overlay masks and index-vs-manifest fragment drift all stay with DatasetPreFilter independently of the loader. The metrics work is also complete on the ANN side now, including the batch node, which was the gap in #9460.
I read code only: no build and no test run, and I did not reproduce the 4/5 → 3/5 recall observation.
| if filter_plan.is_empty() && self.fragments.is_none() { | ||
| log::trace!("no filter plan, no prefilter"); | ||
| return Ok(PreFilterSource::None); | ||
| if filter_plan.is_empty() { |
There was a problem hiding this comment.
This early return moves queries from per-row exact scoring to FlatIndex's bulk path (flat/index.rs:156 and :277 branch on prefilter.is_empty()), and on 4-bit that path quantizes the distance table to 255 steps. That is the recall change you disclose in the body.
Two qualifiers worth adding there: it is 4-bit only (8-bit accumulates in the same order as the per-row path), and prefilter(true) is not required. With the default postfilter the ANN gets ExprFilterPlan::default(), so a filtered postfilter query with a full fragment list newly takes this path too.
I would not gate it behind a flag, since a plain unfiltered query without with_fragments already took the quantized path before this PR. But nothing fires that path today: the 4-bit recall assertion at ivf/v2.rs:3946 uses a fixture of about 128 rows per partition, below FLAT_NUM_4BIT_PQ = 200, so the quantized branch never runs, and this PR's own cases use k=500 over 400 rows, which stays exact. A 4-bit recall case with more than max(200, k) rows in a partition would put something behind the tradeoff.
There was a problem hiding this comment.
Addressed in 583e735. Added an end-to-end 4-bit IVF_PQ recall test with a fixed seed and supplied centroids/codebook: 512 rows in one partition, k=10, and no refinement, so the quantized region beyond max(200, k) is exercised. It covers L2 and dot, both without a predicate and with default postfiltering, checks recall against flat search, and checks explicit full-snapshot scopes against implicit scopes. The PR body now explicitly describes the 4-bit-specific extra distance-table quantization and the postfilter case. No flag or PQ scoring change was added.
| log::trace!("no filter plan, no prefilter"); | ||
| return Ok(PreFilterSource::None); | ||
| if filter_plan.is_empty() { | ||
| let has_full_fragment_coverage = self.fragments.as_ref().is_none_or(|fragments| { |
There was a problem hiding this comment.
prefilter_source has four callers on main and fts (line 4331) is one of them, so this early return also fires for an FTS query with a full fragment list and no filter. The PR body only talks about ANN, so a reader would not notice that FTS plans change too.
I believe it is correct: under full coverage this branch is the path the pre-existing is_none() case already took. What is missing is evidence. Two of the three new cases run ANN queries and the third calls the planner directly; nowhere in the repo does a test put an FTS query together with a complete fragment list.
test_fragment_list_filtering already has four cases ([2], [0], [0,1,2], [0,2]); one more with the complete set would do it, and since both the vector and the FTS variant share that helper, one case covers both.
There was a problem hiding this comment.
Addressed in 583e735. Added the complete-fragment case to the shared test_fragment_list_filtering helper, comparing returned IDs with the no-explicit-scope query. Both ANN and FTS now exercise it, including indexed fragments and unindexed fallback. The PR summary now explicitly includes FTS plan changes.
| } else { | ||
| let stream = src_node.execute(partition, context)?; | ||
| Some(Box::new(FilteredRowIdsToPrefilter(stream)) as Box<dyn FilterLoader>) | ||
| Some(Box::new(FilteredRowIdsToPrefilter::new(stream)) as Box<dyn FilterLoader>) |
There was a problem hiding this comment.
FilteredRowIdsToPrefilter now has three construction sites and only the one in knn.rs gets .with_metrics. The build_prefilter site here is reached from five callers, all in fts.rs, so any filtered FTS query still does the full row-id read while all seven metrics stay absent and the summary reports 0.
Those five are mechanical: each is inside execute(&self, partition, context) with its own metrics set, and build_prefilter already carries partition and masks, so adding a metrics parameter is the same change you already made in knn.rs.
I would not copy it into shared_prefilter_future: N SharedPreFilterExec nodes race for one shared load, so whichever inserts the cache entry records the whole load while the others report zero, and attribution has to be decided first. Separately, multivec_ann clones the source into N per-query nodes, so prefilter_loads reads N there versus the 1 your new test pins for the batch node; worth a note somewhere.
There was a problem hiding this comment.
Addressed in 583e735. Passed each FTS node's metrics set through all five build_prefilter callers and attached the seven loader metrics only on the ordinary FilteredRowIds path. The new filtered FTS execution-summary test first failed with prefilter_loads=0, then passed after the wiring change; it checks all seven metrics and also covers the no-loader case. shared_prefilter_future remains unchanged, and the PR body documents that attribution gap. Added a code comment and PR note that counts represent physical loads: one shared load for batch ANN versus potentially N loads for multi-vector ANN.
|
Regarding the recall risk noted in the review: the additional distance-table quantization is part of Lance's existing 4-bit IVF_PQ bulk distance calculation ( This PR removes a redundant all-row prefilter, allowing some explicitly full-snapshot searches that previously used per-row scoring to reach the existing bulk path. It therefore can expose that recall difference to additional queries; I agree that this observable impact needs to remain documented. If the bulk path's recall tradeoff is considered a defect, would it make sense to address it separately in the bulk distance calculation itself? That would cover searches already using that path as well as those enabled by this optimization. Changing PQ scoring precision or candidate selection would go beyond this PR's scope of eliminating redundant prefilter materialization and adding loader metrics. |
lance-gatekeeper
Bot
removed
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
|
Agreed with the gate, and @yanghua's "documented and explicitly accepted" is the right frame for closing this out. The details are in my inline comment on One thing to add on top of that: the "benchmark this shape and increase refinement if needed" advice belongs in the PR body rather than only in a review thread, since the body is what turns into release notes and what users actually read. |
lance-gatekeeper
Bot
removed
K-approved
There was a problem hiding this comment.
The latest discussion leaves the implementation recommendation unchanged but changes the documentation basis for the accepted risk. Removing the redundant all-row prefilter can expose more searches to the existing 4-bit bulk scorer, changing candidates or lowering recall; the pull-request author has explicitly accepted that consequence.
That discussion adds user-facing context that is not yet in the PR body: the risk is specific to 4-bit scoring, default post-filtered searches can reach it without prefilter(true), the referenced recall case does not exercise the quantized branch, and recall-sensitive deployments should benchmark this shape and increase refinement if needed. These are non-blocking release-note improvements, and maintainers should decide whether to fold them into the body before merge.
lance-gatekeeper
Bot
added
K-approved
lance-gatekeeper
Bot
removed
K-approved
…prefilters (#68408) ### What problem does this PR solve? Related PR: #68375 (branch-4.1). An explicitly full-snapshot Lance search can materialize a redundant row-ID prefilter. Extract the third-party dependency update from #68375 for master, pinning all Lance dependencies in lance-c's Cargo.toml and Cargo.lock to `f202fe41ac18323ca0cd7bc6efd8fd30b8722116` from lance-format/lance#9460. That upstream PR is still open. The pin stays on Lance v11; it does not upgrade to the newer Lance main branch. The new patch follows master's existing PR #83 patch and has its own marker, so both fresh sources and already-patched source caches receive the update exactly once. The patch payload is identical to #68375. This PR only changes `thirdparty/`: Doris reader profile wiring, documentation, and regression fixture changes remain outside this extraction. The C ABI is unchanged; loader metrics are available upstream but this PR does not expose new Doris profile counters. ### Compatibility The upstream change skips redundant row-ID materialization while retaining explicit fragment scope for fallback and the independent deletion/overlay filtering. Removing the all-row prefilter can activate the existing unfiltered bulk scorer. Its 4-bit IVF_PQ distance-table quantization can change candidates and reduce recall at the same refinement budget. #68375 documents a fixture-level recall change and carries a separate regression workaround, which is not included here. This PR does not fix PQ scoring or claim unchanged recall. Recall-sensitive deployments should benchmark the affected query shape; increasing refinement may help but does not guarantee identical results. ### Release note Update the Lance dependency used by lance-c to avoid redundant full-snapshot row-ID prefilters and provide upstream prefilter loader metrics. The existing 4-bit IVF_PQ bulk scoring path may produce different ANN candidates as described above. ### Validation - Verified the lance-c v0.1.9 source archive against the checksum in `thirdparty/vars.sh`. - Executed the actual Lance patch block on fresh sources and on sources prepared by master's previous patch block, with `--fuzz=0`. - Repeated both paths to verify idempotence and compared the resulting lockfiles. - `cargo +1.91.0 metadata --locked --offline --filter-platform x86_64-unknown-linux-gnu`: all 21 Lance-source packages resolve to the intended revision without a lockfile update. - `bash -n thirdparty/download-thirdparty.sh` and `git diff --cached --check` passed. - Full third-party compilation, Doris BE tests, and external regression tests were not run for this extraction; CI validation remains required. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](lance-format/lance#9471), backported into the [v11 PR #9460](lance-format/lance#9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - Test: dependency patch application, cached-source upgrade, rerun idempotence, locked dependency resolution, and shell syntax checks. - Behavior changed: Yes; the dependency removes redundant prefilter work and can select the existing 4-bit PQ bulk scoring path. - Does this need documentation: Compatibility and dependency-only scope are documented above; Doris profile documentation remains with #68375.
…nters (#68375) ### What problem does this PR solve? An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning. Integrate the narrowly scoped Lance v11 fix in lance-format/lance#9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit `f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved. Expose seven profile metrics: `LancePrefilterLoads`, `LancePrefilterInputRows`, `LancePrefilterInputBatches`, `LancePrefilterRowIds`, `LancePrefilterLoadTime`, `LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change. ### ANN result compatibility Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm. As a test-only workaround, the dot-score golden probes all partitions and uses `refine_factor=256`, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries. ### Release note Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters. ### Validation - Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends. - Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests --benches -- -D warnings` passed. - Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed. - Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile. - Test workaround: Groovy compilation and 16 fixture checks against Lance 11.0.0 passed across filtered/unfiltered paths, including all five score goldens and the three single-partition discriminators. The full Doris regression suite was not run locally. - The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](lance-format/lance#9471), backported into the [v11 PR #9460](lance-format/lance#9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - [x] Unit Test: extend the existing Lance reader test and add upstream planner/loader tests. - [x] Behavior changed: redundant full-snapshot prefilter work is skipped; 4-bit PQ ANN candidates may change as described above. - [x] Documentation: `docs/lance-prefilter-profile.md` describes the counters and validation method. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
## Problem Equivalent 4-bit IVF_PQ searches can return different candidates when an all-row prefilter is present versus absent. Removing redundant full-snapshot prefilters in #9471 exposed this existing difference to more queries. The bulk scorer uses f32 for the prefix/tail and a uint8-quantized distance table for the middle of a partition. Besides rounding and saturating accumulation, it subtracts `qmin` for each subvector but adds it back only once. Mixing these scores can invert candidate ordering before refinement. A deterministic 224-row regression has float PQ distances 3.2 and 3.6, but the bulk path changes the latter to 2.6 and selects the wrong top-1. ## Change Use f32 lookup-table accumulation for all 4-bit PQ codes, with the same pairwise addition order as `PQDistCalculator::distance()`. Keep the transposed code layout and the full-snapshot prefilter optimization. Remove the obsolete internal `k_hint` argument and uint8 quantization helpers. There is no index format change or index rebuild requirement; scores remain approximate with respect to original vectors because PQ compression is unchanged. Replace the test that reproduced the quantized implementation with float-score consistency tests covering empty/small partitions, the former 200-row boundary, SIMD tails, positive/negative/constant tables, and a top-1 inversion. Compare the actual bulk/per-row calculators for L2 and Dot at several `k_hint` values. Extend the full-snapshot recall regression to Cosine and compare no-prefilter results with an all-row prefilter. ## Validation - Red: the original kernel failed 16 of the initial 30 focused 4-bit regressions, including the top-1 inversion and actual bulk/per-row calculator comparisons. - `cargo test --profile ci -p lance-index --lib vector::pq -- --nocapture`: 65 passed. - `cargo test --profile ci -p lance --lib full_snapshot_prefilter -- --nocapture`: 22 passed. - The compiled `lance` unit-test binary filtered to `4bit`: 12 passed, including IVF_PQ and IVF_HNSW_PQ with L2/Cosine/Dot (six scanner cases overlap the preceding run). - `cargo fmt --all -- --check`: passed. - `cargo clippy --profile ci --all --tests --benches -- -D warnings`: passed. ## Performance tradeoff This prioritizes consistent float PQ scoring over the previous uint8 FastScan throughput. An isolated microbenchmark of the extracted old/new kernels on synthetic inputs (x86 Haswell target, `release-with-debug`, median of seven rounds, k=100) measured: | Rows | Subvectors | Old kernel | Fixed kernel | Fixed / old | | --- | --- | --- | --- | --- | | 128 | 96 | 8.456 us | 3.584 us | 0.42x | | 512 | 96 | 13.912 us | 14.236 us | 1.02x | | 16,384 | 96 | 98.553 us | 472.215 us | 4.79x | These are isolated kernel measurements, not end-to-end ANN results. Large partitions can incur a substantial CPU cost. The production concurrent workload has not been benchmarked. Further acceleration should preserve float ADC candidate ordering (for example, SIMD float lookup or conservative quantized screening followed by float scoring), rather than only reranking candidates already selected by lossy scores. --------- Co-authored-by: Yang Cen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
…nters (#68375) ### What problem does this PR solve? An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning. Integrate the narrowly scoped Lance v11 fix in lance-format/lance#9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit `f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved. Expose seven profile metrics: `LancePrefilterLoads`, `LancePrefilterInputRows`, `LancePrefilterInputBatches`, `LancePrefilterRowIds`, `LancePrefilterLoadTime`, `LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change. ### ANN result compatibility Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm. As a test-only workaround, the dot-score golden probes all partitions and uses `refine_factor=256`, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries. ### Release note Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters. ### Validation - Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends. - Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests --benches -- -D warnings` passed. - Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed. - Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile. - Test workaround: Groovy compilation and 16 fixture checks against Lance 11.0.0 passed across filtered/unfiltered paths, including all five score goldens and the three single-partition discriminators. The full Doris regression suite was not run locally. - The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](lance-format/lance#9471), backported into the [v11 PR #9460](lance-format/lance#9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - [x] Unit Test: extend the existing Lance reader test and add upstream planner/loader tests. - [x] Behavior changed: redundant full-snapshot prefilter work is skipped; 4-bit PQ ANN candidates may change as described above. - [x] Documentation: `docs/lance-prefilter-profile.md` describes the counters and validation method. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Summary
ANN and FTS queries explicitly scoped to every fragment in a fixed snapshot can still read all indexed row IDs and materialize an allow set. Skip that redundant loader when there is no predicate at the prefilter stage and the selected fragment IDs exactly cover the manifest. Preserve the explicit fragment list for unindexed fallback, deletion/overlay masks, index-segment selection, subset filtering, and actual prefilter predicates.
Expose seven row-ID loader metrics for single/batch ANN and ordinary FTS:
prefilter_loads,prefilter_input_rows,prefilter_input_batches,prefilter_row_ids,prefilter_load_time,prefilter_input_time, andprefilter_build_time. Input rows count non-null IDs; the resulting set count is distinct. Timers operate per batch, and total load time overlaps input/build time.This is the
maincounterpart of #9460 (release/v11.0). Counts represent physical materializations: batch ANN shares one loader, whereas multi-vector ANN can load once per query node. Shared MultiMatch loaders remain uninstrumented because attributing one shared load to racing consumer nodes needs a separate ownership design; missing metrics there do not mean no work occurred.ANN compatibility
Removing an all-row prefilter can select the existing bulk ANN scoring path. For 4-bit IVF_PQ, this path additionally quantizes the distance table for rows beyond the exact prefix; 8-bit PQ does not perform this additional quantization. This also affects default postfilter queries: their predicate is applied after ANN and is not passed into the ANN prefilter, so a complete explicit fragment scope can take the fast path without
prefilter(true).Downstream testing with Lance 11.0.0 found a frozen dot fixture where Top-5 recall changed from 4/5 to 3/5 at the same refinement budget. That observation is scoped to the v11 reproduction, not a benchmark of this main-branch port. This PR does not change PQ scoring or promise identical approximate candidates or unchanged recall. The bulk path already served unfiltered queries without an explicit fragment scope.
Validation
cargo test --profile ci -p lance --lib prefilter: 114 passed, including the new recall and ANN/FTS execution-summary cases, snapshot/deletion coverage, and shared MultiMatch regressions.cargo fmt --all -- --checkpassed.cargo clippy --profile ci --all --tests --benches -- -D warningspassed.The new 4-bit integration test uses a fixed seed, supplied centroids/codebook, 512 rows in one IVF partition, and Top-10 without refinement. This exceeds the exact-prefix threshold of
max(200, k)and exercises quantized bulk scoring. It checks L2 and dot recall against flat search, both with no predicate and with a default postfilter, and compares complete explicit scopes with implicit scopes.The shared fragment-scope helper now checks complete scopes for both ANN and FTS, including indexed hits and unindexed fallback. FTS execution-summary tests check all seven metrics for a real predicate and no loader work for an unfiltered complete scope. The filtered FTS test was observed failing with
prefilter_loads = 0before the wiring fix.The full workspace test suite and a production concurrency benchmark were not run. No production latency reduction is claimed.