Skip to content

Commit 6d43ec5

Browse files
morningmanzhangstar333Gabriel39wangzhaobo957-cloudJay-ju
committed
[feature](be) Read Lance datasets with FileScannerV2
### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (#68707, #68813, #68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: zhaobo wang <[email protected]> Co-authored-by: jay <[email protected]> Co-authored-by: FANNG <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 7c049b0 commit 6d43ec5

150 files changed

Lines changed: 8918 additions & 64 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/CMakeLists.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -894,6 +894,8 @@ else()
894894
"-framework Foundation"
895895
"-framework SystemConfiguration"
896896
"-framework Security"
897+
# liblance_c.a pulls in the Rust `sysinfo` crate, whose macOS backend calls IOKit.
898+
"-framework IOKit"
897899
)
898900
if (USE_JEMALLOC OR (NOT CMAKE_BUILD_TYPE STREQUAL "DEBUG" AND NOT CMAKE_BUILD_TYPE STREQUAL "RELEASE"))
899901
set(DORIS_LINK_LIBS

‎be/src/common/config.cpp‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1232,6 +1232,26 @@ DEFINE_Validator(variant_max_json_key_length,
12321232
DEFINE_Validator(variant_storage_parse_mode,
12331233
[](const int config) -> bool { return config >= 0 && config <= 2; });
12341234

1235+
// Lance uses one BE-wide session so metadata/index caches and the optional Foyer data-file cache
1236+
// can be shared by all Lance dataset readers.
1237+
DEFINE_Int64(lance_index_cache_size_bytes, "10737418240"); // 10 GiB
1238+
DEFINE_Int64(lance_metadata_cache_size_bytes, "1073741824"); // 1GB
1239+
DEFINE_Bool(enable_lance_data_cache, "true");
1240+
DEFINE_String(lance_data_cache_path, "${DORIS_HOME}/lance_data_cache");
1241+
DEFINE_Int64(lance_data_cache_disk_capacity_bytes, "107374182400"); // 100GB
1242+
DEFINE_Int64(lance_data_cache_read_block_size_bytes, "1048576"); // 1MB
1243+
1244+
// I/O buffering budget per Lance scanner, not a cap on its total memory usage.
1245+
// Runtime changes apply to newly created scanners.
1246+
DEFINE_mInt64(lance_io_buffer_size_bytes, "2147483648"); // 2 GiB
1247+
DEFINE_Validator(lance_io_buffer_size_bytes, [](int64_t value) { return value > 0; });
1248+
1249+
// Read-ahead limits per Lance scanner. Runtime changes apply to newly created scanners.
1250+
DEFINE_mInt32(lance_batch_readahead, "5");
1251+
DEFINE_Validator(lance_batch_readahead, [](int32_t value) { return value > 0; });
1252+
DEFINE_mInt32(lance_fragment_readahead, "5");
1253+
DEFINE_Validator(lance_fragment_readahead, [](int32_t value) { return value > 0; });
1254+
12351255
// block file cache
12361256
DEFINE_Bool(enable_file_cache, "true");
12371257
// ATTENTION: For test only. Keep this enabled in production.

‎be/src/common/config.h‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1286,6 +1286,21 @@ DECLARE_Bool(enable_debug_points);
12861286
DECLARE_Int32(pipeline_executor_size);
12871287
DECLARE_Int32(blocking_pipeline_executor_size);
12881288

1289+
// Lance shared session and optional Foyer data-file cache.
1290+
DECLARE_Int64(lance_index_cache_size_bytes);
1291+
DECLARE_Int64(lance_metadata_cache_size_bytes);
1292+
DECLARE_Bool(enable_lance_data_cache);
1293+
DECLARE_String(lance_data_cache_path);
1294+
DECLARE_Int64(lance_data_cache_disk_capacity_bytes);
1295+
DECLARE_Int64(lance_data_cache_read_block_size_bytes);
1296+
1297+
// I/O buffering budget per Lance scanner, applied when a new scanner is created.
1298+
DECLARE_mInt64(lance_io_buffer_size_bytes);
1299+
1300+
// Read-ahead limits per Lance scanner, applied when a new scanner is created.
1301+
DECLARE_mInt32(lance_batch_readahead);
1302+
DECLARE_mInt32(lance_fragment_readahead);
1303+
12891304
// block file cache
12901305
DECLARE_Bool(enable_file_cache);
12911306
DECLARE_mBool(enable_file_cache_write_from_s3_file_writer);

‎be/src/core/data_type/data_type.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,9 @@ void IDataType::to_protobuf(PTypeDesc* ptype) const {
8484
auto node = ptype->add_types();
8585
node->set_type(TTypeNodeType::SCALAR);
8686
auto scalar_type = node->mutable_scalar_type();
87-
scalar_type->set_type(doris::to_thrift(get_primitive_type()));
87+
// NULL uses UInt8 internally; preserve its logical type even inside complex schemas.
88+
scalar_type->set_type(is_null_literal() ? TPrimitiveType::NULL_TYPE
89+
: doris::to_thrift(get_primitive_type()));
8890
to_protobuf(ptype, node, scalar_type);
8991
}
9092

‎be/src/core/data_type_serde/data_type_nullable_serde.cpp‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -410,6 +410,18 @@ Status DataTypeNullableSerDe::read_column_from_arrow(IColumn& column,
410410
const arrow::Array* arrow_array, int64_t start,
411411
int64_t end,
412412
const cctz::time_zone& ctz) const {
413+
if (arrow_array->type_id() == arrow::Type::NA) {
414+
// Arrow Null has neither a validity bitmap nor values, even when sliced. Avoid the
415+
// physical SerDe for Doris's UInt8 placeholder and keep nested values and nulls aligned.
416+
if (arrow_array->offset() < 0 || start < 0 || end < start || end > arrow_array->length()) {
417+
return Status::InvalidArgument(
418+
"Invalid Arrow Null read range: start={}, end={}, "
419+
"length={}, offset={}",
420+
start, end, arrow_array->length(), arrow_array->offset());
421+
}
422+
column.insert_many_defaults(end - start);
423+
return Status::OK();
424+
}
413425
if (config::enable_arrow_input_validation) {
414426
check_arrow_array_range(*arrow_array, start, end);
415427
check_arrow_validity_bitmap(*arrow_array);

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -167,9 +167,12 @@ bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions& query_o
167167
// fluss scan carries that same format -- log ranges and wrapped lake splits alike -- and is
168168
// dispatched inside FlussHybridReader, which only FileScannerV2 builds. Loads are left alone for
169169
// the same reason as adbc: there is no fluss load path.
170-
if (!is_load && scan_params.__isset.table_format_params &&
171-
(scan_params.table_format_params.table_format_type == "adbc" ||
172-
scan_params.table_format_params.table_format_type == "fluss")) {
170+
// Lance has no v1 reader either. Its scans carry FORMAT_LANCE at scan level, whether they come
171+
// from a Lance catalog or from a file TVF, and there is no Lance load path.
172+
if (!is_load && (scan_params.format_type == TFileFormatType::FORMAT_LANCE ||
173+
(scan_params.__isset.table_format_params &&
174+
(scan_params.table_format_params.table_format_type == "adbc" ||
175+
scan_params.table_format_params.table_format_type == "fluss")))) {
173176
return true;
174177
}
175178
const bool is_transactional_hive =
@@ -194,8 +197,7 @@ bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions& query_o
194197
// after scanner construction. Keep versioned Iceberg scans on V2 without eagerly listing
195198
// every file.
196199
return scanner_v2_requested && !is_load &&
197-
scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP &&
198-
scan_params.format_type != TFileFormatType::FORMAT_LANCE && !is_transactional_hive;
200+
scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP && !is_transactional_hive;
199201
}
200202

201203
Status FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {

‎be/src/exec/operator/materialization_opertor.cpp‎

Lines changed: 82 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
#include <fmt/format.h>
2222
#include <gen_cpp/internal_service.pb.h>
2323

24+
#include <cstring>
2425
#include <set>
2526
#include <sstream>
2627
#include <utility>
@@ -72,6 +73,57 @@ constexpr const char* TOPN_LAZY_MAT_PHASE2_PER_BACKEND_REMOTE_IO_TIME =
7273
constexpr const char* TOPN_LAZY_MAT_PHASE2_PER_BACKEND_WRITE_CACHE_IO_TIME =
7374
"TopNLazyMaterializationSecondPhasePerBackendWriteCacheIOTime";
7475

76+
struct MaterializationRowLocation {
77+
ROW_VERSION version = ROW_VERSION::FILE_LOCAL_ROW_ID;
78+
int64_t backend_id = 0;
79+
uint32_t file_id = 0;
80+
uint64_t row_id = 0;
81+
};
82+
83+
Status decode_global_row_location_v2(const StringRef& encoded, ROW_VERSION version,
84+
MaterializationRowLocation* decoded) {
85+
if (encoded.size != sizeof(GlobalRowLoacationV2)) {
86+
return Status::InternalError(
87+
"invalid global row location size for version {}: actual={}, expected={}",
88+
static_cast<int>(version), encoded.size, sizeof(GlobalRowLoacationV2));
89+
}
90+
GlobalRowLoacationV2 location(GlobalRowLoacationV2::VERSION, 0, 0, 0);
91+
std::memcpy(&location, encoded.data, sizeof(location));
92+
decoded->version = version;
93+
decoded->backend_id = location.backend_id;
94+
switch (version) {
95+
case ROW_VERSION::FILE_LOCAL_ROW_ID:
96+
decoded->file_id = location.file_local.file_id;
97+
decoded->row_id = location.file_local.row_id;
98+
return Status::OK();
99+
case ROW_VERSION::LANCE_DATASET_ROW_ID:
100+
decoded->file_id = location.lance_file_id;
101+
decoded->row_id = location.lance_row_id;
102+
return Status::OK();
103+
}
104+
return Status::NotSupported("unsupported V2 global row location version: {}",
105+
static_cast<int>(version));
106+
}
107+
108+
Status decode_materialization_row_location(const StringRef& encoded,
109+
MaterializationRowLocation* decoded) {
110+
if (encoded.size < sizeof(uint8_t)) {
111+
return Status::InternalError("global row location is empty");
112+
}
113+
114+
const auto version_value = static_cast<uint8_t>(encoded.data[0]);
115+
const auto version = static_cast<ROW_VERSION>(version_value);
116+
// Keep size validation inside each version-family decoder. A future version can therefore
117+
// use a different encoded size without being rejected by the current 24-byte V2 contract.
118+
switch (version) {
119+
case ROW_VERSION::FILE_LOCAL_ROW_ID:
120+
case ROW_VERSION::LANCE_DATASET_ROW_ID:
121+
return decode_global_row_location_v2(encoded, version, decoded);
122+
}
123+
return Status::NotSupported("unsupported global row location version: {}, encoded_size={}",
124+
static_cast<int>(version_value), encoded.size);
125+
}
126+
75127
void update_counter(RuntimeProfile* profile, const std::string& name, TUnit::type unit,
76128
int64_t value) {
77129
COUNTER_UPDATE(ADD_COUNTER_WITH_LEVEL(profile, name, unit, 2), value);
@@ -403,16 +455,20 @@ Status MaterializationSharedState::validate_rpc_results(int node_id) {
403455

404456
void MaterializationSharedState::_update_profile_info(int64_t backend_id,
405457
RuntimeProfile* response_profile) {
458+
DORIS_CHECK(response_profile != nullptr);
406459
if (!backend_profile_info_string.contains(backend_id)) {
407460
backend_profile_info_string.emplace(backend_id,
408461
std::map<std::string, fmt::memory_buffer> {});
409462
}
410463
auto& info_map = backend_profile_info_string[backend_id];
411464

412-
auto update_profile_info_key = [&](const std::string& info_key) {
465+
auto update_profile_info_key = [&](const std::string& info_key, bool warn_if_missing = true) {
413466
const auto* info_value = response_profile->get_info_string(info_key);
414467
if (info_value == nullptr) [[unlikely]] {
415-
LOG(WARNING) << "Get row id fetch rpc profile success, but no info key :" << info_key;
468+
if (warn_if_missing) {
469+
LOG(WARNING) << "Get row id fetch rpc profile success, but no info key :"
470+
<< info_key;
471+
}
416472
return;
417473
}
418474
if (!info_map.contains(info_key)) {
@@ -427,6 +483,13 @@ void MaterializationSharedState::_update_profile_info(int64_t backend_id,
427483
update_profile_info_key(RowIdStorageReader::FileReadLinesProfile);
428484
update_profile_info_key(FileScanner::FileReadBytesProfile);
429485
update_profile_info_key(FileScanner::FileReadTimeProfile);
486+
update_profile_info_key(RowIdStorageReader::LanceDatasetOpenTimeProfile, false);
487+
update_profile_info_key(RowIdStorageReader::LanceRowIdTakeReadTimeProfile, false);
488+
update_profile_info_key(RowIdStorageReader::LanceArrowToDorisBlockTimeProfile, false);
489+
update_profile_info_key(RowIdStorageReader::LanceRowIdFetchTotalTimeProfile, false);
490+
for (const auto& [name, unit] : RowIdStorageReader::LanceFetchCountersProfile) {
491+
update_profile_info_key(name, false);
492+
}
430493
}
431494

432495
Status MaterializationSharedState::create_muiltget_result(const Columns& columns, bool child_eos) {
@@ -451,19 +514,28 @@ Status MaterializationSharedState::create_muiltget_result(const Columns& columns
451514

452515
for (int j = 0; j < rows; ++j) {
453516
if (!null_map || !null_map[j]) {
454-
DCHECK(column_rowid->get_data_at(j).size == sizeof(GlobalRowLoacationV2));
455-
GlobalRowLoacationV2 row_location =
456-
*((GlobalRowLoacationV2*)column_rowid->get_data_at(j).data);
517+
MaterializationRowLocation row_location;
518+
RETURN_IF_ERROR(decode_materialization_row_location(column_rowid->get_data_at(j),
519+
&row_location));
457520
auto rpc_struct = rpc_struct_map.find(row_location.backend_id);
458521
if (UNLIKELY(rpc_struct == rpc_struct_map.end())) {
459522
return Status::InternalError(
460523
"MaterializationSinkOperatorX failed to find rpc_struct, backend_id={}",
461524
row_location.backend_id);
462525
}
463-
rpc_struct->second.request.mutable_request_block_descs(i)->add_row_id(
464-
row_location.row_id);
465-
rpc_struct->second.request.mutable_request_block_descs(i)->add_file_id(
466-
row_location.file_id);
526+
auto* request_block_desc =
527+
rpc_struct->second.request.mutable_request_block_descs(i);
528+
const auto row_location_version = static_cast<uint32_t>(row_location.version);
529+
if (request_block_desc->row_id_size() == 0) {
530+
request_block_desc->set_row_location_version(row_location_version);
531+
} else if (request_block_desc->row_location_version() != row_location_version) {
532+
return Status::InternalError(
533+
"mixed row location versions in one materialization request: "
534+
"actual={}, expected={}",
535+
row_location_version, request_block_desc->row_location_version());
536+
}
537+
request_block_desc->add_row_id(row_location.row_id);
538+
request_block_desc->add_file_id(row_location.file_id);
467539
block_order[j] = row_location.backend_id;
468540

469541
// Count rows per backend
@@ -593,8 +665,7 @@ Status MaterializationOperator::pull(RuntimeState* state, Block* output_block, b
593665
local_state._materialization_state.backend_profile_info_string) {
594666
auto* child_profile = local_state.operator_profile()->create_child(
595667
"RowIDFetcher: BackendId:" + std::to_string(backend_id));
596-
for (const auto& [info_key, info_value] :
597-
local_state._materialization_state.backend_profile_info_string[backend_id]) {
668+
for (const auto& [info_key, info_value] : child_info) {
598669
child_profile->add_info_string(info_key, "{" + fmt::to_string(info_value) + "}");
599670
}
600671
local_state.operator_profile()->add_child(child_profile, true);

0 commit comments

Comments
 (0)