fix: skip redundant full-snapshot row-id prefilters and expose loader metrics - #9460

…9471)

## Summary

ANN and FTS queries explicitly scoped to every fragment in a fixed
snapshot can still read all indexed row IDs and materialize an allow
set. Skip that redundant loader when there is no predicate at the
prefilter stage and the selected fragment IDs exactly cover the
manifest. Preserve the explicit fragment list for unindexed fallback,
deletion/overlay masks, index-segment selection, subset filtering, and
actual prefilter predicates.

Expose seven row-ID loader metrics for single/batch ANN and ordinary
FTS: `prefilter_loads`, `prefilter_input_rows`,
`prefilter_input_batches`, `prefilter_row_ids`, `prefilter_load_time`,
`prefilter_input_time`, and `prefilter_build_time`. Input rows count
non-null IDs; the resulting set count is distinct. Timers operate per
batch, and total load time overlaps input/build time.

This is the `main` counterpart of #9460 (`release/v11.0`). Counts
represent physical materializations: batch ANN shares one loader,
whereas multi-vector ANN can load once per query node. Shared MultiMatch
loaders remain uninstrumented because attributing one shared load to
racing consumer nodes needs a separate ownership design; missing metrics
there do not mean no work occurred.

## ANN compatibility

Removing an all-row prefilter can select the existing bulk ANN scoring
path. For 4-bit IVF_PQ, this path additionally quantizes the distance
table for rows beyond the exact prefix; 8-bit PQ does not perform this
additional quantization. This also affects default postfilter queries:
their predicate is applied after ANN and is not passed into the ANN
prefilter, so a complete explicit fragment scope can take the fast path
without `prefilter(true)`.

Downstream testing with Lance 11.0.0 found a frozen dot fixture where
Top-5 recall changed from 4/5 to 3/5 at the same refinement budget. That
observation is scoped to the v11 reproduction, not a benchmark of this
main-branch port. This PR does not change PQ scoring or promise
identical approximate candidates or unchanged recall. The bulk path
already served unfiltered queries without an explicit fragment scope.

## Validation

- `cargo test --profile ci -p lance --lib prefilter`: 114 passed,
including the new recall and ANN/FTS execution-summary cases,
snapshot/deletion coverage, and shared MultiMatch regressions.
- Four fragment-scope/segment-selection tests and all 23 FTS
execution-node tests passed using the compiled Lance test binary.
- `cargo fmt --all -- --check` passed.
- `cargo clippy --profile ci --all --tests --benches -- -D warnings`
passed.

The new 4-bit integration test uses a fixed seed, supplied
centroids/codebook, 512 rows in one IVF partition, and Top-10 without
refinement. This exceeds the exact-prefix threshold of `max(200, k)` and
exercises quantized bulk scoring. It checks L2 and dot recall against
flat search, both with no predicate and with a default postfilter, and
compares complete explicit scopes with implicit scopes.

The shared fragment-scope helper now checks complete scopes for both ANN
and FTS, including indexed hits and unindexed fallback. FTS
execution-summary tests check all seven metrics for a real predicate and
no loader work for an unfiltered complete scope. The filtered FTS test
was observed failing with `prefilter_loads = 0` before the wiring fix.

The full workspace test suite and a production concurrency benchmark
were not run. No production latency reduction is claimed.

…prefilters (#68408)

### What problem does this PR solve?

Related PR: #68375 (branch-4.1).

An explicitly full-snapshot Lance search can materialize a redundant
row-ID prefilter. Extract the third-party dependency update from #68375
for master, pinning all Lance dependencies in lance-c's Cargo.toml and
Cargo.lock to `f202fe41ac18323ca0cd7bc6efd8fd30b8722116` from
lance-format/lance#9460. That upstream PR is
still open. The pin stays on Lance v11; it does not upgrade to the newer
Lance main branch.

The new patch follows master's existing PR #83 patch and has its own
marker, so both fresh sources and already-patched source caches receive
the update exactly once. The patch payload is identical to #68375. This
PR only changes `thirdparty/`: Doris reader profile wiring,
documentation, and regression fixture changes remain outside this
extraction. The C ABI is unchanged; loader metrics are available
upstream but this PR does not expose new Doris profile counters.

### Compatibility

The upstream change skips redundant row-ID materialization while
retaining explicit fragment scope for fallback and the independent
deletion/overlay filtering. Removing the all-row prefilter can activate
the existing unfiltered bulk scorer. Its 4-bit IVF_PQ distance-table
quantization can change candidates and reduce recall at the same
refinement budget. #68375 documents a fixture-level recall change and
carries a separate regression workaround, which is not included here.
This PR does not fix PQ scoring or claim unchanged recall.
Recall-sensitive deployments should benchmark the affected query shape;
increasing refinement may help but does not guarantee identical results.

### Release note

Update the Lance dependency used by lance-c to avoid redundant
full-snapshot row-ID prefilters and provide upstream prefilter loader
metrics. The existing 4-bit IVF_PQ bulk scoring path may produce
different ANN candidates as described above.

### Validation

- Verified the lance-c v0.1.9 source archive against the checksum in
`thirdparty/vars.sh`.
- Executed the actual Lance patch block on fresh sources and on sources
prepared by master's previous patch block, with `--fuzz=0`.
- Repeated both paths to verify idempotence and compared the resulting
lockfiles.
- `cargo +1.91.0 metadata --locked --offline --filter-platform
x86_64-unknown-linux-gnu`: all 21 Lance-source packages resolve to the
intended revision without a lockfile update.
- `bash -n thirdparty/download-thirdparty.sh` and `git diff --cached
--check` passed.
- Full third-party compilation, Doris BE tests, and external regression
tests were not run for this extraction; CI validation remains required.


### Dependency review follow-up

The dependency pin includes the applicable review fixes from [Lance
#9471](lance-format/lance#9471), backported into
the [v11 PR #9460](lance-format/lance#9460). FTS
row-ID materialization now exports the same seven loader metrics as ANN.
Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit
PQ recall fixture that actually exercises bulk distance-table
quantization, including default postfilter queries. PQ scoring itself is
unchanged.

Keep the original dependency patch as the upgrade baseline and apply
`lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing
caches containing the old prefilter patch must receive the new pin;
simply replacing the old patch contents would leave those caches
unchanged. Both Doris PRs carry identical patch payloads.

Follow-up validation:
- Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS
tests passed; workspace rustfmt and Clippy passed.
- Verified the release archive checksum, applied both actual Doris patch
blocks with `--fuzz=0` to fresh and previously patched sources, and
verified identical resulting trees and idempotent reruns.
- Incompatible cached input fails without writing the new success
marker.
- Rust 1.91 locked Linux dependency metadata resolves all 21
Lance-source packages to the new revision without modifying Cargo.lock.
- Shell syntax and whitespace checks passed. This refresh does not
include a full third-party build or Doris regression run; PR CI is
required.

### Check List (For Author)

- Test: dependency patch application, cached-source upgrade, rerun
idempotence, locked dependency resolution, and shell syntax checks.
- Behavior changed: Yes; the dependency removes redundant prefilter work
and can select the existing 4-bit PQ bulk scoring path.
- Does this need documentation: Compatibility and dependency-only scope
are documented above; Doris profile documentation remains with #68375.

…nters (#68375)

### What problem does this PR solve?

An unfiltered Lance vector search explicitly scoped to every fragment in
its snapshot can still scan all indexed row IDs and construct a
redundant allow set. This adds work to every query, particularly under
concurrency, and the existing profile cannot isolate set construction
from input scanning.

Integrate the narrowly scoped Lance v11 fix in
lance-format/lance#9460 through the existing
lance-c patch chain. Pin every Lance crate and the lockfile to commit
`f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from
the official repository. The upstream PR is pending review.
Full-snapshot unfiltered queries omit the row-ID loader; fragment
subsets, real predicates, deletion/overlay masks, indexed segment
selection, and unindexed fallback retain their semantics. The explicit
fragment list is preserved.

Expose seven profile metrics: `LancePrefilterLoads`,
`LancePrefilterInputRows`, `LancePrefilterInputBatches`,
`LancePrefilterRowIds`, `LancePrefilterLoadTime`,
`LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their
scope and overlapping wall-time semantics, and extend indexed
multi-vector reader tests to check filtered/non-filtered metrics after
close. There is no C ABI or FE protocol change.

### ANN result compatibility

Removing an all-row prefilter activates Lance's unfiltered distance
calculation path. For 4-bit IVF_PQ, that path additionally quantizes the
distance table and can change ANN candidates at a fixed refinement
budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5
to 3/5; this PR does not change the underlying PQ algorithm.

As a test-only workaround, the dot-score golden probes all partitions
and uses `refine_factor=256`, covering all 1024 fixture rows before
exact reranking. Update its golden to the exact dot results. Existing
index-selection and single-partition ANN checks retain their original
settings. This workaround does not establish unchanged recall for
production queries.

### Release note

Avoid redundant row-ID prefilter materialization for full-snapshot Lance
vector searches and add detailed prefilter profile counters.

### Validation

- Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed,
including execution-summary export, deleted rows, old snapshots, stable
row IDs, pinned segments, and unindexed appends.
- Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests
--benches -- -D warnings` passed.
- Doris: syntax compilation of both changed C++ files passed with Lance
enabled; clang-format 16 and shell syntax checks passed.
- Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive
with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all
21 Lance crates to the pinned commit without updating the lockfile.
- Test workaround: Groovy compilation and 16 fixture checks against
Lance 11.0.0 passed across filtered/unfiltered paths, including all five
score goldens and the three single-partition discriminators. The full
Doris regression suite was not run locally.
- The full Doris BE unit-test binary and production concurrency
benchmark were not run locally. CI and a benchmark at fixed recall are
still needed; no end-to-end latency reduction is claimed.


### Dependency review follow-up

The dependency pin includes the applicable review fixes from [Lance
#9471](lance-format/lance#9471), backported into
the [v11 PR #9460](lance-format/lance#9460). FTS
row-ID materialization now exports the same seven loader metrics as ANN.
Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit
PQ recall fixture that actually exercises bulk distance-table
quantization, including default postfilter queries. PQ scoring itself is
unchanged.

Keep the original dependency patch as the upgrade baseline and apply
`lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing
caches containing the old prefilter patch must receive the new pin;
simply replacing the old patch contents would leave those caches
unchanged. Both Doris PRs carry identical patch payloads.

Follow-up validation:
- Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS
tests passed; workspace rustfmt and Clippy passed.
- Verified the release archive checksum, applied both actual Doris patch
blocks with `--fuzz=0` to fresh and previously patched sources, and
verified identical resulting trees and idempotent reruns.
- Incompatible cached input fails without writing the new success
marker.
- Rust 1.91 locked Linux dependency metadata resolves all 21
Lance-source packages to the new revision without modifying Cargo.lock.
- Shell syntax and whitespace checks passed. This refresh does not
include a full third-party build or Doris regression run; PR CI is
required.

### Check List (For Author)

- [x] Unit Test: extend the existing Lance reader test and add upstream
planner/loader tests.
- [x] Behavior changed: redundant full-snapshot prefilter work is
skipped; 4-bit PQ ANN candidates may change as described above.
- [x] Documentation: `docs/lance-prefilter-profile.md` describes the
counters and validation method.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…nters (#68375)

### What problem does this PR solve?

An unfiltered Lance vector search explicitly scoped to every fragment in
its snapshot can still scan all indexed row IDs and construct a
redundant allow set. This adds work to every query, particularly under
concurrency, and the existing profile cannot isolate set construction
from input scanning.

Integrate the narrowly scoped Lance v11 fix in
lance-format/lance#9460 through the existing
lance-c patch chain. Pin every Lance crate and the lockfile to commit
`f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from
the official repository. The upstream PR is pending review.
Full-snapshot unfiltered queries omit the row-ID loader; fragment
subsets, real predicates, deletion/overlay masks, indexed segment
selection, and unindexed fallback retain their semantics. The explicit
fragment list is preserved.

Expose seven profile metrics: `LancePrefilterLoads`,
`LancePrefilterInputRows`, `LancePrefilterInputBatches`,
`LancePrefilterRowIds`, `LancePrefilterLoadTime`,
`LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their
scope and overlapping wall-time semantics, and extend indexed
multi-vector reader tests to check filtered/non-filtered metrics after
close. There is no C ABI or FE protocol change.

### ANN result compatibility

Removing an all-row prefilter activates Lance's unfiltered distance
calculation path. For 4-bit IVF_PQ, that path additionally quantizes the
distance table and can change ANN candidates at a fixed refinement
budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5
to 3/5; this PR does not change the underlying PQ algorithm.

As a test-only workaround, the dot-score golden probes all partitions
and uses `refine_factor=256`, covering all 1024 fixture rows before
exact reranking. Update its golden to the exact dot results. Existing
index-selection and single-partition ANN checks retain their original
settings. This workaround does not establish unchanged recall for
production queries.

### Release note

Avoid redundant row-ID prefilter materialization for full-snapshot Lance
vector searches and add detailed prefilter profile counters.

### Validation

- Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed,
including execution-summary export, deleted rows, old snapshots, stable
row IDs, pinned segments, and unindexed appends.
- Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests
--benches -- -D warnings` passed.
- Doris: syntax compilation of both changed C++ files passed with Lance
enabled; clang-format 16 and shell syntax checks passed.
- Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive
with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all
21 Lance crates to the pinned commit without updating the lockfile.
- Test workaround: Groovy compilation and 16 fixture checks against
Lance 11.0.0 passed across filtered/unfiltered paths, including all five
score goldens and the three single-partition discriminators. The full
Doris regression suite was not run locally.
- The full Doris BE unit-test binary and production concurrency
benchmark were not run locally. CI and a benchmark at fixed recall are
still needed; no end-to-end latency reduction is claimed.


### Dependency review follow-up

The dependency pin includes the applicable review fixes from [Lance
#9471](lance-format/lance#9471), backported into
the [v11 PR #9460](lance-format/lance#9460). FTS
row-ID materialization now exports the same seven loader metrics as ANN.
Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit
PQ recall fixture that actually exercises bulk distance-table
quantization, including default postfilter queries. PQ scoring itself is
unchanged.

Keep the original dependency patch as the upgrade baseline and apply
`lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing
caches containing the old prefilter patch must receive the new pin;
simply replacing the old patch contents would leave those caches
unchanged. Both Doris PRs carry identical patch payloads.

Follow-up validation:
- Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS
tests passed; workspace rustfmt and Clippy passed.
- Verified the release archive checksum, applied both actual Doris patch
blocks with `--fuzz=0` to fresh and previously patched sources, and
verified identical resulting trees and idempotent reruns.
- Incompatible cached input fails without writing the new success
marker.
- Rust 1.91 locked Linux dependency metadata resolves all 21
Lance-source packages to the new revision without modifying Cargo.lock.
- Shell syntax and whitespace checks passed. This refresh does not
include a full third-party build or Doris regression run; PR CI is
required.

### Check List (For Author)

- [x] Unit Test: extend the existing Lance reader test and add upstream
planner/loader tests.
- [x] Behavior changed: redundant full-snapshot prefilter work is
skipped; 4-bit PQ ANN candidates may change as described above.
- [x] Documentation: `docs/lance-prefilter-profile.md` describes the
counters and validation method.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

## Summary

Explicitly selecting every fragment in a snapshot should not require
scanning every indexed row ID before ANN/FTS search. Adopt the Lance v11
prefilter optimization and loader statistics from
lance-format/lance#9460, upstreaming the
dependency update used in apache/doris#68408.

Pin all Lance dependencies to
`db211492fc5cd9da5642d7234d9682de9be77c19`. This revision includes
lance-c's existing `356acb0` dependency, preserving vector segment
compatibility validation. The lockfile changes only the 21 Lance source
revisions; package versions, Arrow 58 dependencies, and the C ABI are
unchanged.

Add a C API regression test with two fragments and both stable and
physical row IDs. It checks exact IVF_FLAT results, verifies that full
explicit and implicit scopes scan the same number of rows without
loading an allow set, and checks that strict fragment subsets still load
and enforce their row-ID prefilter.

## Dependency and compatibility

lance-format/lance#9460 is still open; this pins its tested integration
revision. The optimization can select the existing bulk scorer: 4-bit
IVF_PQ candidate recall at a fixed refinement budget may change. The
separate scoring fix in lance-format/lance#9537 on main is not included
in this v11 update. See the upstream PR for the detailed compatibility
discussion.

## Validation

- The new C API test fails against the previous `356acb0` pin: an
explicit full scope scans **69 rows** versus **5** for the implicit
scope. The same test passes with the new pin, including both row-ID
modes and strict-subset counters.
- `cargo fmt --all -- --check`: passed.
- `cargo clippy --locked --all-targets -- -D warnings`: passed with Rust
1.91.0.
- `cargo test --locked`: 441 passed.
- `cargo test --locked --test compile_and_run_test -- --ignored`: all
three passed (C, C++, and static-library OSS transport).
- Upstream integration: 84 prefilter tests and 18 segment
commit/compatibility tests passed. Its workspace Clippy run is blocked
by an existing diagnostic in an unchanged file; details are recorded in
lance-format/lance#9460.

)

Refreshes and repairs `release/v11.0` so its validation can pass again.
This branch does not carry the #9460 code change; it is only the
release-branch maintenance that #9460's normal merge depends on.

## Base validation failures fixed

- `cargo-deny` after newly published RustSec advisories: update
`ctutils` to 0.4.6, `rustls` to 0.23.46, `rustls-webpki` to 0.103.16,
and `webpki-roots` to 1.0.10 in the root, Python, and Java JNI
lockfiles. Move `lance-encoding` to zstd 0.14.3 / zstd-safe 8.2.0 /
zstd-sys 2.1.2; Arrow 58 keeps zstd 0.13.1 / zstd-safe 7.2.1 for
`arrow-ipc`.
- `vec_to_table` doctest under pandas 3: apply upstream #10241 so the
`to_pandas()` example uses `+ELLIPSIS`.
- Lindera dictionary tests racing under pytest 9: apply upstream #10389
by pinning `pytest==8.4.2` and `pytest-xdist==3.8.0` in the Python
`tests` extra.
- HNSW coverage `test_distance_range_prefilter_dispatch`: backport the
bounded stranded-node repair from main #9053 so every level-0 node keeps
a path from the entry point after parallel construction. This is a
behavior fix, unlike #10294 which only adjusted test call sites.

## Advisory exception evidence

- RUSTSEC-2024-0449: local RustSec advisory source lists `patched = []`
and `unaffected = []`; the crate is archived with a final 0.2.11
release. The dependency graph is `unsafe-libyaml -> serde_yaml
(datafusion-substrait build dep via substrait)`. `cargo-deny` reports no
safe upgrade, so the ignore is a no-fix record, not a policy bypass.
- RUSTSEC-2026-0403 affects only `zstd_safe::DCtx::decompress_stream`.
Release's zstd consumers use `zstd::bulk::{Compressor,
decompress_to_buffer}`, `zstd::stream::copy_decode` (a standard
`read::Decoder`), and `arrow-ipc`'s `zstd::bulk::Decompressor`; none
call the affected streaming API. The dated ignore expires with the
Arrow/DataFusion upgrade that can move `arrow-ipc` to zstd 0.14.

## Validation actually run

- `cargo-deny 0.19.0 check` (CI-pinned): advisories ok, bans ok,
licenses ok, sources ok.
- `cargo fmt --all -- --check`: pass.
- Full workspace `cargo clippy --profile ci --locked --features
<aggregate release features> --all-targets -- -D warnings`: pass after
the HNSW change.
- `cargo check --locked --profile ci -p lance-encoding --all-features
--lib`: pass.
- Locked metadata checks pass for root, Python, and Java JNI manifests.
- `lance-index` lib tests after the HNSW backport, with real cargo exit
codes: `test_distance_range_prefilter_dispatch` (2/2),
`test_connect_stranded_nodes_links_from_nearby_reachable_node` (1/1),
`test_connect_stranded_nodes_anchors_each_node_once` (1/1),
`test_degenerate_vectors_stay_reachable_within_degree_bound` (2/2),
`test_minimum_params_reachability` (1/1). Eight pre-patch loop runs (16
cases) also passed and are recorded, but they do not prove the
concurrency race is gone; the deterministic regression tests are the
meaningful coverage.
- Python 3.11 `uv run make lint`: pass. Targeted `uv run pytest
--doctest-modules python/lance/vector.py -q`: 1 passed. Remote PR head
`70352c699` lint job (pandas 3.1.0, pytest 9.2.0) reported `42 passed, 1
skipped`, which is the original pandas-3 doctest path.
- Full local doctest collection was not used as evidence: the first
Python 3.14t environment lacked the optional `torch` extra and failed
during collection before reaching assertions. The original vector
doctest was verified in both Python 3.11 locally and the pandas-3 CI
lint path above.

## Scope and known limits

- No unrelated main API/format changes are backported; the HNSW fix is
restricted to `connect_stranded_nodes` and its regression tests in
`rust/lance-index/src/vector/hnsw/builder.rs`.
- The HNSW fix addresses the identified reachability defect; the job
that failed did not capture graph structure, so the base-branch
attribution is code/history evidence, not an instrumentation reproducer.
- This PR is authored by Xuanwo and still requires an independent
qualified reviewer before merge.

---------

Co-authored-by: YueZhang <[email protected]>