Repository navigation
[opt](lance) skip full-snapshot row-id prefilters and add profile counters - #68375
Merged
yiguolei merged 3 commits intoSep 26, 2026
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 28th, 2026 2:03 AM. The selected account is excluded until 2026-09-28T02:03:00Z. Please trigger /review again; another configured account may be available. |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Related PR: apache#68375, apache#68408 Refresh the pinned Lance v11 dependency with the FTS metric wiring and full-snapshot test coverage backported from the main-branch review. Apply the follow-up pin independently because existing source caches already carry the original prefilter patch marker. ### Release note Include FTS row-ID prefilter metrics in the Lance dependency and upgrade previously patched source caches to the same revision as fresh builds. ### Check List (For Author) - Test: v11 prefilter and FTS tests, Rust formatting and workspace Clippy, archive checksum, fresh/cached patch application, idempotence, rejection without a success marker, locked dependency metadata, shell syntax. - Behavior changed: FTS loader metrics are recorded; cached dependencies receive the updated pin. PQ scoring is unchanged. - Does this need documentation: Dependency scope and validation are recorded in the pull request description.
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Sep 23, 2026
### What problem does this PR solve? Related PR: apache#68375, apache#68408 Refresh the pinned Lance v11 dependency with the FTS metric wiring and full-snapshot test coverage backported from the main-branch review. Apply the follow-up pin independently because existing source caches already carry the original prefilter patch marker. ### Release note Include FTS row-ID prefilter metrics in the Lance dependency and upgrade previously patched source caches to the same revision as fresh builds. ### Check List (For Author) - Test: v11 prefilter and FTS tests, Rust formatting and workspace Clippy, archive checksum, fresh/cached patch application, idempotence, rejection without a success marker, locked dependency metadata, shell syntax. - Behavior changed: FTS loader metrics are recorded; cached dependencies receive the updated pin. PQ scoring is unchanged. - Does this need documentation: Dependency scope and validation are recorded in the pull request description.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
One documentation correction is needed before this profile surface is reliable.
Critical checkpoints:
- Correctness and filtering: the full-snapshot optimization compares the explicit fragment scope with the opened snapshot and still leaves deletion, overlay, index-coverage, and fallback filtering in place. I found no new result-domain bug in that path. The 4-bit IVF-PQ candidate change is an acknowledged approximate-recall tradeoff; the changed Doris query is an exact-score oracle, while approximate execution remains separately covered upstream.
- Metrics and lifecycle: names, units, per-plan aggregation, and additive accumulation are coherent for scans that reach successful EOF. The inline comment identifies the remaining contract mismatch for cancellation, abort, error, and other non-EOF exits.
- Reader boundaries: the change is confined to Lance scanner profiling and dependency behavior; it does not alter schema mapping, materialization, split ownership, or the TableReader/FileReader boundary.
- Third-party integration: the pins form the direct ab6b5bbe -> f75f3343 -> f202fe41 ancestry, Cargo.toml and Cargo.lock pins advance together, and fresh/intermediate/cache-marker paths either reach the final pin or fail before recording success.
- Tests and validation: I statically checked the changed unit, regression query/golden, all eight changed files, and the exact upstream changes. No local build or test was run, as required. At review time formatting, license, dependency review, and Thirdparty Script Test were green; Linux, macOS, and macOS-arm64 third-party builds were still in progress.
- User focus: no additional focus points were provided.
This is a static review of head 71e92eb.
| The timers overlap: do not add LoadTime to InputTime or BuildTime. They are not CPU | ||
| timers. Across multiple loaders or scanners they accumulate and can exceed query | ||
| wall time. RowIds is not peak resident memory and can count the same ID again when | ||
| separate loaders build separate sets. An interrupted or failed load may contribute |
Contributor
There was a problem hiding this comment.
[P2] Document the EOF publication boundary. The pinned lance-c statistics callback fires only when the stream reaches successful EOF; its FinallyStream does not publish on drop or Some(Err). Doris closes or returns without another poll on cancellation, abort_split, and scanner/conversion errors, so an interrupted or failed prefilter normally leaves all seven profile counters at zero even after doing work. This sentence instead implies that such a load may publish partial input counts. Please state that partial counts are observable only when an execution summary is published at EOF, and that non-EOF exits can leave the entire metric set unavailable.
Gabriel39
added a commit
that referenced
this pull request
Sep 24, 2026
…prefilters (#68408) ### What problem does this PR solve? Related PR: #68375 (branch-4.1). An explicitly full-snapshot Lance search can materialize a redundant row-ID prefilter. Extract the third-party dependency update from #68375 for master, pinning all Lance dependencies in lance-c's Cargo.toml and Cargo.lock to `f202fe41ac18323ca0cd7bc6efd8fd30b8722116` from lance-format/lance#9460. That upstream PR is still open. The pin stays on Lance v11; it does not upgrade to the newer Lance main branch. The new patch follows master's existing PR #83 patch and has its own marker, so both fresh sources and already-patched source caches receive the update exactly once. The patch payload is identical to #68375. This PR only changes `thirdparty/`: Doris reader profile wiring, documentation, and regression fixture changes remain outside this extraction. The C ABI is unchanged; loader metrics are available upstream but this PR does not expose new Doris profile counters. ### Compatibility The upstream change skips redundant row-ID materialization while retaining explicit fragment scope for fallback and the independent deletion/overlay filtering. Removing the all-row prefilter can activate the existing unfiltered bulk scorer. Its 4-bit IVF_PQ distance-table quantization can change candidates and reduce recall at the same refinement budget. #68375 documents a fixture-level recall change and carries a separate regression workaround, which is not included here. This PR does not fix PQ scoring or claim unchanged recall. Recall-sensitive deployments should benchmark the affected query shape; increasing refinement may help but does not guarantee identical results. ### Release note Update the Lance dependency used by lance-c to avoid redundant full-snapshot row-ID prefilters and provide upstream prefilter loader metrics. The existing 4-bit IVF_PQ bulk scoring path may produce different ANN candidates as described above. ### Validation - Verified the lance-c v0.1.9 source archive against the checksum in `thirdparty/vars.sh`. - Executed the actual Lance patch block on fresh sources and on sources prepared by master's previous patch block, with `--fuzz=0`. - Repeated both paths to verify idempotence and compared the resulting lockfiles. - `cargo +1.91.0 metadata --locked --offline --filter-platform x86_64-unknown-linux-gnu`: all 21 Lance-source packages resolve to the intended revision without a lockfile update. - `bash -n thirdparty/download-thirdparty.sh` and `git diff --cached --check` passed. - Full third-party compilation, Doris BE tests, and external regression tests were not run for this extraction; CI validation remains required. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](lance-format/lance#9471), backported into the [v11 PR #9460](lance-format/lance#9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - Test: dependency patch application, cached-source upgrade, rerun idempotence, locked dependency resolution, and shell syntax checks. - Behavior changed: Yes; the dependency removes redundant prefilter work and can select the existing 4-bit PQ bulk scoring path. - Does this need documentation: Compatibility and dependency-only scope are documented above; Doris profile documentation remains with #68375.
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Two documentation corrections are needed before the new profile counters are reliable to interpret. The inline comments cover (1) the counters' shared ANN/FTS scope and (2) the difference between per-scanner all-manifest elision and a full-snapshot query composed of strict-subset splits. I did not duplicate the existing line-46 EOF-publication thread.
Critical checkpoints:
- Correctness and filtering: the full-manifest optimization uses exact fragment-ID set equality against the opened fixed snapshot, retains explicit fragments for pinned-index and unindexed fallback planning, and leaves deletion, stable-row-ID, external, and overlay masks independent. I found no result-domain defect.
- Metrics and lifecycle: count/time kinds, nanosecond units, successful-EOF callback lifetime, and additive aggregation across loaders/scanners are coherent. The accepted issues are semantic attribution and scanner-versus-query scope; non-EOF publication is already covered by the existing thread.
- Reader and split boundaries: Doris can create one ANN split per physical index segment plus uncovered-fragment splits, and the BE reopens a scanner with each split's fragment subset. That production path is why a full-snapshot query is not itself a zero-counter guarantee. No schema-mapping or materialization boundary regression was found.
- Third-party integration: the direct ab6b5bbe -> f75f3343 -> f202fe41 ancestry is consistent; all 11 manifest pins and all 21 Lance lockfile sources converge on f202 under cargo --locked. Fresh, intermediate, stale-cache, and interruption marker states either reach the final pin or fail before recording success.
- Performance and compatibility: removing the redundant all-manifest loader preserves index-segment selection, unindexed fallback, and visibility semantics. The changed exact-refinement golden does not by itself validate approximate ranking, but adjacent single-probe Doris coverage and upstream L2/Dot prefilter/postfilter recall coverage still exercise the 4-bit IVF_PQ path.
- Tests and validation: I performed static inspection of all eight changed paths, their Doris callers, and the exact upstream commits, plus a static application of the full patch chain to the MD5-matching v0.1.9 archive. No local build or test was run, as required. At submission time, formatting/license/dependency checks, third-party script and Linux/macOS/macOS-arm64 builds, compile, FE/Cloud UT, performance, and vault_p0 were green; BE UT, External/NonConcurrent/P0/cloud_p0 regressions, and code-review checks were still pending. Those are CI results, not independent execution here.
- User focus: no additional focus points were provided.
This is a converged static review of head 71e92eb.
| handling keep their existing semantics. A strict fragment subset or an actual filter | ||
| continues to use the normal prefilter path. | ||
|
|
||
| The following Doris counters describe Lance's ANN **row-ID prefilter loader**, not |
Contributor
There was a problem hiding this comment.
[P2] Include FTS in the documented metric scope. The second pin advances to f202fe41, whose filtered FTS execution nodes use the same FilteredRowIdsToPrefilter metrics, and Doris maps those names without checking _search_kind. A successfully exhausted MATCH/PHRASE scan can therefore increment these counters even though this paragraph attributes them to ANN, which can mislead profile diagnosis. Please describe them as shared ANN/FTS row-ID loader metrics and update the adjacent C++ ANN row-id loader comment (or export distinct names by search kind).
| separate loaders build separate sets. An interrupted or failed load may contribute | ||
| partial input counts without a completed set cardinality. | ||
|
|
||
| For a full-snapshot, unfiltered ANN query, no row-ID loader is needed and these |
Contributor
There was a problem hiding this comment.
[P2] Scope this zero-counter claim to a single all-fragment scanner. The f75 optimization compares each scanner's fragment set with the opened manifest, but Doris plans one split per physical vector-index segment plus strict-subset uncovered-fragment splits. A full-snapshot unfiltered query over multiple segments therefore creates subset scanners that still build FilteredRowIds, and their successful callbacks can make these profile counters positive. Please say zero is expected only when a scanner/split itself contains the whole manifest, and note that segment-sharded full-snapshot queries may legitimately be nonzero.
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Sep 26, 2026
yiguolei
pushed a commit
that referenced
this pull request
Sep 29, 2026
…nters (#68375) ### What problem does this PR solve? An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning. Integrate the narrowly scoped Lance v11 fix in lance-format/lance#9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit `f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved. Expose seven profile metrics: `LancePrefilterLoads`, `LancePrefilterInputRows`, `LancePrefilterInputBatches`, `LancePrefilterRowIds`, `LancePrefilterLoadTime`, `LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change. ### ANN result compatibility Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm. As a test-only workaround, the dot-score golden probes all partitions and uses `refine_factor=256`, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries. ### Release note Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters. ### Validation - Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends. - Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests --benches -- -D warnings` passed. - Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed. - Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile. - Test workaround: Groovy compilation and 16 fixture checks against Lance 11.0.0 passed across filtered/unfiltered paths, including all five score goldens and the three single-partition discriminators. The full Doris regression suite was not run locally. - The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](lance-format/lance#9471), backported into the [v11 PR #9460](lance-format/lance#9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - [x] Unit Test: extend the existing Lance reader test and add upstream planner/loader tests. - [x] Behavior changed: redundant full-snapshot prefilter work is skipped; 4-bit PQ ANN candidates may change as described above. - [x] Documentation: `docs/lance-prefilter-profile.md` describes the counters and validation method. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
3 of 12 tasks
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 26, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 27, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 28, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 29, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 31, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Nov 2, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Nov 5, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
yiguolei
pushed a commit
that referenced
this pull request
Nov 6, 2026
### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: zy-kkk <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning.
Integrate the narrowly scoped Lance v11 fix in lance-format/lance#9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit
f202fe41ac18323ca0cd7bc6efd8fd30b8722116; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved.Expose seven profile metrics:
LancePrefilterLoads,LancePrefilterInputRows,LancePrefilterInputBatches,LancePrefilterRowIds,LancePrefilterLoadTime,LancePrefilterInputTime, andLancePrefilterBuildTime. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change.ANN result compatibility
Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm.
As a test-only workaround, the dot-score golden probes all partitions and uses
refine_factor=256, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries.Release note
Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters.
Validation
cargo clippy --profile ci --all --tests --benches -- -D warningspassed.--fuzz=0; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile.Dependency review follow-up
The dependency pin includes the applicable review fixes from Lance #9471, backported into the v11 PR #9460. FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged.
Keep the original dependency patch as the upgrade baseline and apply
lance-c-0.1.9-prefilter-fts.patchwith an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads.Follow-up validation:
--fuzz=0to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns.Check List (For Author)
docs/lance-prefilter-profile.mddescribes the counters and validation method.Check List (For Reviewer who merge this PR)