[Feature](lance) Support multi-vector columns and queries by Gabriel39 · Pull Request #68028 · apache/doris

Gabriel39 · 2026-09-15T14:55:52Z

What problem does this PR solve?

Issue Number: close #66494

vector_search accepts one query matrix for a Lance List<FixedSizeList<T, D>> column on branch-4.1, supporting Float16, Float32 and Float64. A matrix such as [[1,0],[0,1]] ranks logical rows containing variable numbers of subvectors. The row score is sum(q in query, min(v in row, distance(q, v))); the TVF filter precedes candidate selection, and TopK/offset count rows.

The Lance-C adapter computes scores before candidate truncation, preserving small L2 differences and aggregating each ANN query child across all output batches. Indexed candidates undergo exact refinement, and fragment-scoped results regain distance order before limit/offset. ANN candidate selection remains approximate. Omitted metrics resolve consistently to L2 on every split, including partially indexed datasets. The pinned Lance version supports cosine multi-vector indexes; L2/dot use exact search when no compatible index exists.

FE, BE and C API checks bound query width to 128 subvectors and candidate work to 100,000 for both num_vectors * (top_k + offset) and refine_factor * (top_k + offset). Actual null/non-finite stored elements encountered by scoring fail execution; empty/null outer rows do not rank. Undefined cosine pairs from zero-norm vectors are ignored, and rows without a defined match for any query subvector are filtered. The adapter resolves nested vector paths and backtick-quoted child names consistently with Lance. The shared result-window code also preserves Lance's strict batching stage after sorting and slicing; the older Doris C binding does not expose a strict-batching setter, so that path is covered by a native execution-plan test without adding an API. Requests retain schema version 1; the optional num_vectors field distinguishes matrices from ordinary single-vector payloads.

The vendored patch adds the matrix API without changing existing C signatures. Following #67741, the third-party build now applies the community patch chain on Lance-C 0.1.9, including the Lance v11 upgrade and merged lance-format/lance-c#83. PR #83's patch records its upstream commit and adapts overlapping context to retain PR #79's scalar-segment execution path; its multi-vector implementation and integration tests match upstream. Rebuild Lance-C: previously installed libraries lack the required API. Re-extract sources carrying the older patch chain; the build rejects stale patch markers instead of silently retaining an old library.

Validation

Completed a full local self-review and corrected the issues found before committing.

53 Lance BE reader tests passed with the reader/test sources and generated Thrift linked to the newly patched Lance-C and existing BE test dependencies. Segment-scoped TopK is compared with an independent global scoring oracle across two index segments and an unindexed fragment.
The integrated Lance v11 patch chain passed 410 Rust tests, including 14 multi-vector integration tests. All three opt-in C/C++ compile/run and static OSS transport tests also passed.
88 additional native query combinations passed independent score, ID, window and full nested-payload checks against the same integrated library.
31 focused FE tests and FE Checkstyle passed after retaining schema version 1 and removing the multi-vector upgrade fence.
clang-format 16, Groovy syntax compilation, standalone Groovy scoring-oracle checks, fixture self-checks, shell syntax and zero-fuzz patch application passed. The actual patch driver was checked for clean application, repeat execution and rejection of stale source caches.
Added SQL regression coverage with 768-row, 128-dimensional IVF_FLAT and IVF_PQ fixtures: four partitions, two index segments and one unindexed fragment. EXPLAIN assertions verify index/split planning; independent expected results cover global TopK/offset, filtering, empty windows, and full vector/nullable payload retrieval with lazy materialization enabled and disabled.
Expanded type/dimension/scoring coverage to Float16/Float32/Float64 at dimensions 1, 3, 8 and 128, all three metrics, repeated query subvectors, empty/null outer rows, unsupported schemas, invalid query elements, Float16 overflow, and actual null/NaN/infinite stored elements for every element type. Existing regressions retain nested-path, precision, batch-size, default-metric and work-budget checks.

The complete test_lance_multivector_coverage and test_lance_vector_search_index_types suites passed locally (2 suites, 0 failures, 0 skips) against the CI build of d38fba4. The schema-version change was compiled and tested separately in the focused FE/BE harnesses; those results are not a full Doris build of the final revision. The complete external regression set remains for CI. Exact equality in the representative index fixtures uses all partitions and sufficient candidate overfetch; it is not a general ANN recall guarantee.

Release note

Support multi-vector Lance columns and query matrices in vector_search, with documented row-level distance, filtering and TopK semantics.

Check List (For Author)

Test
- Regression test added
- Unit Test
Behavior changed
- Multi-vector columns accept matrix queries; omitted metrics consistently use L2.
Documentation
- Added docs/lance-multivector-search.md.
- Website documentation: apache/doris-website#4148, with English and Chinese usage examples, parameter defaults and limits, scoring, null handling, and index/refinement guidance. Merge the documentation with or after this PR.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-09-15T14:55:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2026-09-15T14:56:09Z

run buildall

Gabriel39 · 2026-09-15T14:56:11Z

/review

github-actions

Static review of exact head f1d9951b8729aee5d8158e3e9d626183a32e55ac against base d9570d1ea9f7744c08df6a74a188c8619aa37ac2 found five blocking issues: precision is lost before exact TopK; indexed scoring depends on RecordBatch fragmentation; an omitted metric can mix cosine and L2 across splits; unbounded query-subvector count expands into unbounded ANN plan width; and unsupported stored null/non-finite elements can be scored plausibly instead of rejected.

I reviewed the complete authoritative 53-file change set, applicable AGENTS/guides, current thread/comment history, and pinned Lance/DataFusion production paths through two bounded convergence rounds. The final live head/base and duplicate fences were rechecked before submission. No additional review focus was supplied. This was static analysis only: I did not run builds or tests or edit repository source. The PR description reports focused tests passed and says the full SQL regression suite was not run.

hello-stephen · 2026-09-15T16:34:27Z

FE UT Coverage Report

Increment line coverage 80.49% (33/41) 🎉
Increment coverage report
Complete coverage report

Gabriel39 · 2026-09-16T01:45:57Z

run buildall

Gabriel39 · 2026-09-16T01:45:59Z

/review

github-actions

Requesting changes for two remaining correctness issues on exact head cd8255c.

Critical checkpoints:

Scoring correctness: direct distance accumulation, complete indexed-child reduction, null/empty handling, refinement ordering, filters, split offsets, and final global TopN were traced. Finite zero-norm cosine data still turns one derived NaN into a whole-stream failure (inline P1).
Schema/identity: FE JSON -> Thrift v2 -> BE -> Lance-C shape, type, metric, row identity, and work budgets are consistent. The new public C API nevertheless accepts nested Lance field paths that its replacement scorer cannot resolve (inline P2).
Compatibility/lifecycle: schema-version fencing, smooth-upgrade scheduling, scanner ownership, error/cancellation propagation, patch/cache order, C/C++ ABI, and pinned fixture formats were checked; no additional blocker remained.
Index and resource safety: mixed indexed/unindexed fragments, metric mismatch fallback, candidate/refinement caps, and offset expansion converge correctly after the existing fixes.
Coverage/artifacts: duplicate fixtures and catalog compatibility-mode discovery were checked. Existing threads for normalization, batch fragmentation, default metrics, fan-out, and actual null/non-finite primitives were treated as hard duplicate fences and revalidated at this head.

No additional user review focus was supplied. This was a static-only review: the task contract prohibited builds and tests. Two convergence rounds completed, all final-round subagents returned NO_NEW_VALUABLE_FINDINGS, and every candidate was accepted, deduplicated, or dismissed with evidence.

hello-stephen · 2026-09-16T02:38:11Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.06% (32824/53753)
Region Coverage	48.68% (31098/63884)
Branch Coverage	51.60% (9650/18700)

Gabriel39 · 2026-09-16T02:40:15Z

run buildall

Gabriel39 · 2026-09-16T02:40:17Z

/review

Gabriel39 · 2026-09-16T02:55:38Z

run buildall

Gabriel39 · 2026-09-16T02:55:40Z

/review

Gabriel39 · 2026-09-16T02:55:42Z

Synchronized the follow-up from lance-format/lance-c#83 in ee71f0e:

Keep Lance's strict batch-sizing node outside the final sorted result window, preserving its resolved configuration. The older Doris C binding has no strict-batching setter; the shared execution-plan regression exercises the fix without adding a new API.
Add 32 nested/quoted projection combinations covering exact, indexed and hybrid search, checking schema, payload values and distances.

Local validation: 61 native library tests, 13 native multi-vector integration tests and 52 BE reader tests passed against the rebuilt library. The patch applies cleanly with zero fuzz; shared implementation/test equivalence with the upstream PR was checked. Full local self-review completed before committing. Full SQL regression remains for CI.

github-actions · 2026-09-16T03:13:57Z

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/35048879007

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hello-stephen · 2026-09-16T03:14:17Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.15% (32868/53753)
Region Coverage	48.71% (31119/63884)
Branch Coverage	51.61% (9652/18700)

github-actions

Static review of exact head ee71f0e67229922e28c5a2ee9725306cb532316a against base d9570d1ea9f7744c08df6a74a188c8619aa37ac2 found one remaining correctness issue: documented-valid finite L2/dot operands can overflow the pinned f32 distance kernels, and the replacement scorer turns that row into a whole-batch execution error.

Correctness: traced the FE JSON matrix through Thrift, split planning, BE validation, the C ABI, and Lance exact/indexed/refined/hybrid plans. The prior fixes for score precision, child-stream batching, metric consistency, work bounds, actual invalid stored values, zero-norm cosine, and nested paths are present. The inline P2 below remains distinct.

Compatibility: the optional schema-versioned Thrift fields, direct/stale-request revalidation, resolved metric propagation, and smooth-upgrade source fence are aligned across FE and BE.

Lifecycle and safety: checked scanner ownership, typed query-buffer lifetime, checked multiplication/narrowing, cancellation/error propagation, result materialization, and scan-stat callback wiring; no additional issue was substantiated.

Performance and observability: the 128-subvector and 100000 candidate/refinement budgets are enforced at FE, BE, and C boundaries; indexed child streams are fully reduced before final ranking, and the scanner callback remains connected.

Tests and packaging: reviewed the native, FE, BE, and SQL cases, fixture generator/promotion, patch ordering, and all 91 changed paths. The three BE/Docker fixture pairs are byte-identical. Per the review environment, this was static-only: no builds or tests were run.

Convergence: two normal full-review passes and the separate risk pass returned NO_NEW_VALUABLE_FINDINGS in Round 2. One provisional nullable-Struct concern was dismissed after verifying that pinned Lance pushes ancestor validity into persisted children. The user supplied no additional focus points.

## Summary Expose one multi-vector query through `lance_scanner_nearest_multivector` and the C++ `Scanner::nearest_multivector` wrapper. The API copies a row-major matrix for a `List<FixedSizeList<float16|float32|float64>>` column, preserving the distinction between one multi-vector query and independent batch queries without changing existing API signatures. Rows rank by `sum(query_vector, min(stored_vector, distance))`. L2 is the default across indexed and appended fragments; cosine indexes use exact candidate refinement by default. Null/empty outer rows do not rank. Undefined cosine pairs from zero-norm vectors are ignored; rows without a defined match for any query subvector do not rank. Nested vector columns retain Lance field-path parsing, including backtick-quoted child names. Shape/type checks, finite-value validation and limits on query-vector count and candidate work reject unsupported inputs before expensive plan expansion. The pinned Lance execution plan also needs corrections before its TopK operators: accumulating distances directly preserves small L2 differences, and reducing each ANN child across its complete stream prevents batch boundaries from changing scores and dropping the winning row. A scoped scoring adapter implements those corrections, validates actual stored elements, and restores final distance order before limit/offset after fragment-scoped payload reads. Strict row batching remains outside the final sort and result window, using the batch size resolved by Lance. ANN candidate selection remains approximate. The adapter is isolated so it can be removed when equivalent corrections are available in the Lance dependency. ## Validation - Latest full Rust test run: 389 tests passed; the two opt-in C/C++ compile/run tests were ignored in this run. They passed during earlier feature validation, and this follow-up does not change the headers or ABI. - 14 multi-vector integration tests cover types, metrics, malformed/invalid values, work limits, small-distance Top1, batch-size-independent indexed Top1, partial index coverage, zero-norm cosine rows/queries, and nested/quoted paths. - Strict batching reproducer returned `[1, 2, 1]` before the fix and now returns `[2, 2]`. Added offset/limit, empty/tail windows, default sizing and a plan-level test preserving the resolved strict size. - Nested projection tests check schema, payload values, row counts and distance order across 32 combinations of ordinary/quoted paths, projections, index use and appended fragments. - `cargo fmt --check` and `cargo clippy --all-targets -- -D warnings` passed. - Completed full local self-review before committing. Related consumer feature: apache/doris#68028.

Gabriel39 · 2026-09-16T06:08:20Z

run buildall

Gabriel39 · 2026-09-16T06:08:22Z

/review

hello-stephen · 2026-09-16T06:26:22Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.08% (32833/53753)
Region Coverage	48.70% (31109/63884)
Branch Coverage	51.59% (9648/18700)

github-actions · 2026-09-16T06:28:31Z

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 19th, 2026 8:24 AM.
Workflow run: https://github.com/apache/doris/actions/runs/35062385187

The selected account is excluded until 2026-09-19T08:24:00Z. Please trigger /review again; another configured account may be available.

zhangstar333 · 2026-09-16T07:22:54Z

+        }
        TExternalSearchRequest searchRequest = new TExternalSearchRequest()
-                .setSchemaVersion(1)
+                .setSchemaVersion(queryVector.isSetNumVectors() ? 2 : 1)


no need change to version to 2 now

Updated in 80a6aab. Both ordinary and multi-vector requests now use schema version 1; the optional num_vectors field identifies the matrix payload. Removed the multi-vector-specific upgrade fence and its compatibility-only test/documentation. Existing additional-type checks are unchanged. Validation: 31 focused FE tests, 53 Lance BE tests, FE Checkstyle and clang-format 16 passed. The two previously failing SQL suites also passed locally against the CI build of d38fba4; the protocol change was compiled and tested separately with the current FE/BE sources.

zhangstar333 · 2026-09-16T07:25:56Z

+        // Pin the planner's default on every split; Lance otherwise inherits an index metric.
+        vectorParams.setMetric(params.containsKey(METRIC) ? parseMetric(params.get(METRIC)) : TVectorMetric.L2);
+        LanceVectorQuery.validateMultiVectorBudget(queryVector, common.topK(), common.offset(),
+                params.containsKey(REFINE_FACTOR) ? parsePositiveInt(params.get(REFINE_FACTOR), REFINE_FACTOR) : 1);


REFINE_FACTOR must greater than 0 ?

Yes. An explicitly supplied refine_factor must be positive: the pinned Lance indexed-search planner rejects Some(0) with 'Refine factor cannot be zero'. Doris already enforced this through parsePositiveInt before this PR. A factor of 1 still re-ranks candidates using original vectors; it does not disable refinement. Multi-vector queries default to 1 to refine scores and validate the stored values.

zhangstar333 · 2026-09-16T07:29:28Z

 public final class LanceVectorQuery {
+    // Keep aligned with the BE and Lance-C limits: each subvector expands into an ANN branch.
+    public static final int MAX_QUERY_VECTORS = 128;
+    public static final long MAX_QUERY_VECTOR_CANDIDATES = 100_000;


To limit this in Doris? maybe no need?

The limit is a resource guard, not an algorithmic requirement. Lance creates an ANN plan branch for each query subvector and overfetches candidates in each branch, so a small input matrix can still expand into substantial planning and execution work. FE validation rejects oversized requests before dispatch; BE and the merged lance-c #83 API also enforce the same bounds. Removing only the Doris check would therefore defer the rejection to the backend. I agree that 128 subvectors and the 100,000 candidate budget are policy choices and that using the same budget for exact and indexed searches is conservative. We can revisit the thresholds and separate the execution paths with workload evidence, but I would retain the aligned checks in this PR rather than remove the protection.

Gabriel39 · 2026-09-16T11:18:53Z

run buildall

Gabriel39 · 2026-09-16T11:18:54Z

/review

…sertions

Gabriel39 · 2026-09-17T13:22:39Z

run buildall

Gabriel39 · 2026-09-17T13:22:40Z

/review

hello-stephen · 2026-09-17T13:45:55Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.16% (32878/53753)
Region Coverage	48.73% (31132/63884)
Branch Coverage	51.64% (9656/18700)

github-actions

Static-only review of head 1883c5bd124f81f327fd1e0aa74ca6be979f37fc against base 4714241b9936c434d22ceb5cfa84288f28b05124.

Requested changes:

Preserve nullable StructArray ancestor validity while resolving nested multi-vector leaves; otherwise a logically NULL row can rank in exact search and indexed refinement.
Fence multi-vector requests from old smooth-upgrade BEs; schema version 1 preserves decoding but an old BE still interprets the matrix bytes as one vector and rejects the request according to split placement.

The existing finite-distance-overflow issue in discussion r4022237009 remains open and was not duplicated.

Coverage included the complete 181-path authoritative diff, FE/Thrift/BE/C-API contracts, exact/indexed/hybrid scoring and final-window behavior, patch-chain and all seven duplicated fixture trees, docs, regression coverage, and the current inline-thread fence. Two normal full-review rounds plus separate cross-layer risk review converged with no further findings. No additional user focus was supplied. Builds and tests were intentionally not run under the static-review contract.

github-actions · 2026-09-17T13:56:38Z

+        array = array
+            .as_any()
+            .downcast_ref::<arrow_array::StructArray>()
+            .and_then(|parent| parent.column_by_name(part))


[P2] Preserve nullable struct ancestors while resolving this leaf. Arrow keeps a StructArray's validity separately from its children, and column_by_name returns the raw child, so a row with payload = NULL can still expose populated payload.vectors storage here. exact_batch then checks only the leaf ListArray and can rank that logically NULL row; indexed refinement reuses the same scorer. Please AND every traversed ancestor mask into the leaf validity and cover exact/refined searches with a null parent whose child buffer contains a near vector.

github-actions · 2026-09-17T13:56:38Z

    2: optional i32 dimension
    3: optional binary values
+    // Present only for one multi-vector query; values contains a row-major matrix.
+    4: optional i32 num_vectors


[P2] Fence this request from old smooth-upgrade BEs. Keeping schema version 1 preserves decoding, but it does not make the matrix payload executable: an old BE ignores this optional field, validates values as one dimension-sized vector, and rejects every query with two or more subvectors. LanceScanNode currently checks old BEs only for additional projected types, so success depends on split placement during an upgrade. Please add a multi-vector capability/old-BE guard there and a mixed-version test.

hello-stephen · 2026-09-17T16:02:21Z

FE UT Coverage Report

Increment line coverage 76.00% (38/50) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-17T22:39:18Z

BE UT Coverage Report

Increment line coverage 90.16% (55/61) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.89% (26015/43440)
Line Coverage	44.65% (269024/602470)
Region Coverage	40.49% (213441/527143)
Branch Coverage	41.97% (98667/235079)

…rage

Gabriel39 · 2026-09-18T01:12:39Z

run buildall

Gabriel39 · 2026-09-18T01:12:40Z

/review

hello-stephen · 2026-09-18T01:38:00Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.27% (1826/2492)
Line Coverage	61.13% (32888/53800)
Region Coverage	48.71% (31147/63943)
Branch Coverage	51.61% (9662/18720)

github-actions

Static review of exact head 214f51c7f10ecfee921541ebc2cb5f223a4638c0 requests changes. No new non-duplicate inline comment survived review; the blockers remain covered by these current-head threads:

[P2] Finite L2/dot operands can overflow the f32 distance/result representation and abort the whole batch instead of excluding only the unrepresentable row. This thread is marked resolved in the UI, but it has no reply and the exact-head code still contains the reported path: #68028 (comment)
[P2] Nested field traversal drops nullable StructArray ancestor validity, so a logically null parent can expose and rank populated child vectors in exact search and indexed refinement: #68028 (comment)
[P2] The schema-v1 request can carry num_vectors to a smooth-upgrade source BE that ignores the optional field and rejects the matrix bytes using the old scalar-size contract: #68028 (comment)

Critical-checkpoint summary: correctness/numerics remains blocked by finite-distance overflow and dropped ancestor nullability; rolling-upgrade compatibility remains blocked by the schema-v1 matrix request. FE/BE/C validation bounds, plan/window/batch behavior, patch ordering/ABI, fixture parity, and docs/tests produced no additional distinct defect after static tracing. No additional user focus was supplied.

Validation was static-only as required: no build or test command was run. Author and CI test claims were treated as claims, not independent execution evidence.

1

hello-stephen · 2026-09-18T03:46:02Z

FE UT Coverage Report

Increment line coverage 76.00% (38/50) 🎉
Increment coverage report
Complete coverage report

Document how to query a Lance multi-vector column with `vector_search()`, including data preparation, expected results, scoring, index usage, and parameter limits in the English and Chinese 4.x Lance Catalog guides. Depends on apache/doris#68028; merge the documentation with or after that implementation. Related feature request: apache/doris#66494. The guide explains supported Arrow types and nullability, row-level L2/cosine/dot scoring, matrix dimensions, the 128-subvector limit, both 100,000-candidate budgets, mandatory multi-vector refinement, and cosine index usage. It also corrects the metric/refinement defaults and reader versions to match the implementation. ## Versions - [ ] dev - [x] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) The Dev Lance Catalog pages were deliberately removed in #4086. This change updates the existing 4.x pages without restoring those pages or advertising support in older releases. This explains the existing missing-current-counterpart warning from the locale/version check. ## Languages - [x] Chinese - [x] English ## Validation - Reviewed the complete diff against the FE parameter validation and Lance C multi-vector scoring implementation. - Compiled both changed MDX files with GFM/front matter support and normalized Docusaurus explicit heading IDs. - Changed-file link and feature checks: no errors. The external Lance object-store documentation was checked separately. - Verified that both locales contain identical executable examples. Executed the data builder using pylance 11.0.0 with temporary local storage, then checked the documented basic, offset, and prefilter results through the patched Lance C scanner, including empty/null outer rows. - `git diff --check` passed. - No full-site build or new live Doris SQL run. SEO checks reported only existing metadata warnings; no metadata is changed. ## Docs Checklist - [x] Checked by AI - [x] Test Cases Built - [x] Updated required version and language counterparts, or explained why not - [x] If only one language changed, confirmed whether source/translation counterparts need sync

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

@zhangstar333

### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: zy-kkk <[email protected]>

Gabriel39 requested a review from yiguolei as a code owner September 15, 2026 14:55

github-actions Bot requested changes Sep 15, 2026

View reviewed changes

Gabriel39 mentioned this pull request Sep 16, 2026

feat: support multi-vector queries through C and C++ APIs lance-format/lance-c#83

Merged

github-actions Bot requested changes Sep 16, 2026

View reviewed changes

Comment thread thirdparty/patches/lance-c-0.1.9-pr-83.patch Resolved

Comment thread thirdparty/patches/lance-c-0.1.9-multivector.patch Resolved

github-actions Bot requested changes Sep 16, 2026

View reviewed changes

Comment thread thirdparty/patches/lance-c-0.1.9-pr-83.patch Resolved

Gabriel39 mentioned this pull request Sep 16, 2026

[improvement](build) Add merged lance-c multi-vector API patch #68056

Merged

4 of 8 tasks

zhangstar333 reviewed Sep 16, 2026

View reviewed changes

Gabriel39 added 2 commits September 17, 2026 21:18

test(lance): expand multi-vector coverage and adopt community patches

9829663

[fix](lance) retain search protocol version and correct regression as…

1883c5b

…sertions

Gabriel39 force-pushed the dev/lance-multivector-branch-4.1 branch from 80a6aab to 1883c5b Compare September 17, 2026 13:22

github-actions Bot requested changes Sep 17, 2026

View reviewed changes

[fix](test) use Lance lazy materialization switch in multivector cove…

214f51c

…rage

github-actions Bot previously requested changes Sep 18, 2026

View reviewed changes

yiguolei approved these changes Sep 18, 2026

View reviewed changes

Gabriel39 merged commit 8d2018b into apache:branch-4.1 Sep 18, 2026
34 of 35 checks passed

Gabriel39 mentioned this pull request Sep 18, 2026

[Feature](lance) Support multi-vector columns and queries #66494

Closed

9 tasks done

Gabriel39 mentioned this pull request Sep 30, 2026

[Tracking] Lance feature matrix for Doris 4.2 #66340

Open

68 of 87 tasks

morningman mentioned this pull request Oct 25, 2026

[feature](be) Read Lance datasets with FileScannerV2 #69581

Merged

3 of 12 tasks

Conversation

Gabriel39 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Validation

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 15, 2026

Uh oh!

Gabriel39 commented Sep 15, 2026

Uh oh!

Gabriel39 commented Sep 15, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Sep 15, 2026

FE UT Coverage Report

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Sep 16, 2026

Cloud UT Coverage Report

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Cloud UT Coverage Report

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

Gabriel39 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Cloud UT Coverage Report

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

zhangstar333 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Gabriel39 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

zhangstar333 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Gabriel39 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

zhangstar333 Sep 16, 2026

Choose a reason for hiding this comment

Gabriel39 commented Sep 15, 2026 •

edited

Loading