You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6d43ec5
Browse filesBrowse the repository at this point in the historyBrowse files
committed
[feature](be) Read Lance datasets with FileScannerV2
### What problem does this PR solve?
Issue Number: None
Related PR: #65730#66608#66805#66841#66999#67005#67011#67110#67268#67289#67325#67650#67921#68028#68335#68375#68615#68690
Problem Summary:
Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.
- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
(plain scans, vector and full-text search, Substrait and runtime
filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
profiles), copied from branch-4.2 except one local renamed for
-Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
do: there is no V1 Lance reader. The second phase of TopN lazy
materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
64-bit Lance row id (the version 0 layout is unchanged),
PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
the materialization operator and RowIdStorageReader decode and
validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
docker fixture all_types.lance that one of them reads. Under master's
-ffile-prefix-map the session test locates its fixture from the
repository root, and the reader tests shrink the data cache to 64 MiB
under the UT temp dir: the default 100 GiB disk tier is 51200 block
files that stay open for the rest of the test process.
Not included: the MaterializationOperator::push timer fix (#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (#68707, #68813, #68815).
### Release note
Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.
### Check List (For Author)
- Test: Unit Test
- BE UT: the Lance reader, nested Null, runtime filter and session
suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
second phase of TopN lazy materialization rejects row locations of an
unknown version or size.
- Does this need documentation: No (documented with the FE connector)
Co-authored-by: zhangstar333 <[email protected]>
Co-authored-by: Gabriel <[email protected]>
Co-authored-by: zhaobo wang <[email protected]>
Co-authored-by: jay <[email protected]>
Co-authored-by: FANNG <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
0 commit comments