Skip to content

Commit b613c37

Browse files
tomz-alttom zhangclaudemorningman
authored
[feature](lance) Add Rust-based Lance format reader for AI-native dat… (#62182)
Add native Lance format support to Doris via Rust FFI integration, enabling SQL queries over AI-native Lance datasets from local disk and S3. [Lance](https://lancedb.github.io/lance/) is a columnar format designed for vector search, multimodal data (images, embeddings), and fast random access -- widely used in AI/ML pipelines. ## Quick Examples **Read from S3:** ```sql SELECT * FROM s3( "uri" = "s3://bucket/embeddings.lance/data/fragment.lance", "format" = "lance", "s3.access_key" = "...", "s3.secret_key" = "...", "s3.region" = "us-east-1", "s3.endpoint" = "https://s3.us-east-1.amazonaws.com" ) ORDER BY id LIMIT 10; ``` **Read from local disk (for testing):** ```sql -- Get backend_id from: SHOW BACKENDS; SELECT * FROM local( "file_path" = "data/my_dataset.lance/data/fragment.lance", "backend_id" = "<backend_id from SHOW BACKENDS>", "format" = "lance" ) ORDER BY id LIMIT 10; ``` **Aggregation across multi-fragment dataset:** ```sql SELECT count(*), min(id), max(id) FROM s3( "uri" = "s3://bucket/large.lance/data/fragment.lance", "format" = "lance", "s3.access_key" = "...", "s3.secret_key" = "...", "s3.region" = "us-east-1", "s3.endpoint" = "https://s3.us-east-1.amazonaws.com" ); ``` ## Architecture - **Data exchange**: Arrow C Data Interface (zero-copy between Rust and C++) - **Async containment**: block_on() with single-threaded tokio runtime (zero extra OS threads) - **Build gating**: BUILD_RUST_READERS=OFF by default, zero impact on existing builds ## What Works (Verified on Live Cluster) | Feature | Status | |---------|--------| | SELECT * / column projection | Tested | | WHERE filter / LIMIT / COUNT(*) | Tested | | SUM() / AVG() aggregation | Tested | | Multi-fragment datasets (3 fragments, 15 rows) | Tested | | S3 access with AWS credentials | Tested | | Schema inference (fetch_table_schema) | Tested | | Time travel version (config wired) | Config ready | | Vector ANN search / FTS / filter pushdown | Config ready | ## Known Limitations - **TVF path only**: No CREATE CATALOG support yet. Must use local() or s3() TVF - **Directory-based format workaround**: Lance datasets are directories. The TVF file_path must point to a single `.lance` data file inside the dataset; the reader auto-strips the path back to the dataset root and reads all fragments. If the TVF glob matches multiple `.lance` files (multi-fragment dataset), each scan range reopens the full dataset causing **duplicate rows**. Workaround: ensure the file_path glob matches exactly one data file per dataset - **No Doris data cache integration**: Lance reads bypass BlockFileCache. S3 reads are not cached on local SSD - **No filter/vector pushdown from FE**: The Rust config supports filter, vector_search, full_text_search but the FE planner does not populate them yet - **BUILD_RUST_READERS=OFF default**: Requires explicit opt-in and Rust toolchain - **Binary size**: Rust static lib ~430MB (.a), adds ~50-80MB to final doris_be after LTO ## How to Build ```bash # Rust tests only: cd be/src/rust/doris-native && cargo test # Full BE with Lance: BUILD_RUST_READERS=ON ./build.sh --be # Regression test: ./run-regression-test.sh --run -s test_lance_tvf ``` ## Changes **Thrift** (2 files): FORMAT_LANCE = 19, TLanceFileDesc, enable_rust_lance_reader **FE** (4 files): LanceFileFormatProperties, FileFormatProperties factory, FileFormatConstants, SessionVariable **BE - Rust** (be/src/rust/doris-native/, 6 files): error.rs, lance_reader.rs (LanceReaderConfig with S3/version/vector/FTS support), ffi.rs (extern C functions), lib.rs **BE - C++** (5 files): lance_ffi.h, lance_rust_reader.h/cpp (GenericReader with Arrow import), file_scanner.cpp (FORMAT_LANCE dispatch), internal_service.cpp (fetch_table_schema) **Build** (3 files): rust.cmake (Corrosion v0.5), CMakeLists.txt (BUILD_RUST_READERS option), format/CMakeLists.txt **Tests** (4 files): 24 Rust tests, C++ GTest (7), C++ standalone (8), Groovy regression (9) ## Future Work - **Lance Catalog**: CREATE CATALOG for dataset discovery (eliminates TVF path limitations and backend_id requirement) - **Fragment-level scan ranges**: FE lists fragments and creates one scan range per fragment with fragment ID, avoiding duplicate reads - **FE filter/vector pushdown**: Pass WHERE predicates and ANN queries to lance-rs scanner - **Doris BlockFileCache integration**: Route lance I/O through Doris cache for S3 data caching - **Lance Session cache**: Shared IndexCache for vector/FTS index reuse across queries - **Time travel SQL**: FOR VERSION AS OF N via LanceMvccSnapshot --------- Co-authored-by: tom zhang <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
1 parent 7554ffb commit b613c37

42 files changed

Lines changed: 8979 additions & 1 deletion

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/CMakeLists.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,27 @@ if (DEFINED ENV{PAIMON_HOME} AND NOT PAIMON_HOME)
165165
set(PAIMON_HOME "$ENV{PAIMON_HOME}" CACHE PATH "" FORCE)
166166
endif()
167167

168+
option(BUILD_RUST_READERS "Build Rust-based format readers (Lance, etc.)" OFF)
169+
if (DEFINED ENV{BUILD_RUST_READERS})
170+
set(BUILD_RUST_READERS "$ENV{BUILD_RUST_READERS}" CACHE BOOL "" FORCE)
171+
endif()
172+
# Auto-enable if pre-built Rust library exists (from zigbuild or manylinux2014)
173+
if (NOT BUILD_RUST_READERS)
174+
if (EXISTS "${SRC_DIR}/rust/doris-native/target/x86_64-unknown-linux-gnu/release/libdoris_ffi.a"
175+
OR EXISTS "${SRC_DIR}/rust/doris-native/target/release/libdoris_ffi.a")
176+
set(BUILD_RUST_READERS ON)
177+
message(STATUS "Auto-enabling BUILD_RUST_READERS: pre-built library found")
178+
endif()
179+
endif()
180+
if (BUILD_RUST_READERS)
181+
# rust.cmake detects pre-built .a or builds via Corrosion.
182+
# If neither exists, it sets BUILD_RUST_READERS=OFF and returns.
183+
include(cmake/rust.cmake)
184+
endif()
185+
if (BUILD_RUST_READERS)
186+
add_definitions(-DBUILD_RUST_READERS)
187+
endif()
188+
168189
set(CMAKE_SKIP_RPATH TRUE)
169190
set(Boost_USE_STATIC_LIBS ON)
170191
set(Boost_USE_STATIC_RUNTIME ON)
@@ -686,6 +707,10 @@ if (ENABLE_PAIMON_CPP)
686707
set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES} tbb_paimon)
687708
endif()
688709

710+
if (BUILD_RUST_READERS)
711+
set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES} doris_ffi_lib)
712+
endif()
713+
689714
set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES} ${WL_END_GROUP})
690715

691716
# Add all external dependencies. They should come after the palo libs.

‎be/cmake/rust.cmake‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Rust integration for doris-ffi static library.
19+
#
20+
# Two modes:
21+
# 1. Pre-built: If libdoris_ffi.a already exists (e.g., built by cargo-zigbuild
22+
# on the CI host), use it directly. No Rust toolchain needed inside the
23+
# build container.
24+
# 2. Corrosion: If no pre-built .a found and cargo is available, build via
25+
# Corrosion (FetchContent). Used on developer machines.
26+
27+
# Check for pre-built .a (from zigbuild, manylinux2014, or manual build)
28+
set(PREBUILT_RUST_PATHS
29+
"${SRC_DIR}/rust/doris-native/target/x86_64-unknown-linux-gnu/release/libdoris_ffi.a"
30+
"${SRC_DIR}/rust/doris-native/target/release/libdoris_ffi.a"
31+
"${CMAKE_BINARY_DIR}/libdoris_ffi.a"
32+
)
33+
34+
set(RUST_LIB_PATH "")
35+
foreach(p ${PREBUILT_RUST_PATHS})
36+
if (EXISTS "${p}")
37+
set(RUST_LIB_PATH "${p}")
38+
message(STATUS "Rust readers: using pre-built library at ${p}")
39+
break()
40+
endif()
41+
endforeach()
42+
43+
if (RUST_LIB_PATH)
44+
# Mode 1: Pre-built .a found — no Corrosion needed
45+
add_library(doris_ffi_lib STATIC IMPORTED GLOBAL)
46+
set_target_properties(doris_ffi_lib PROPERTIES
47+
IMPORTED_LOCATION "${RUST_LIB_PATH}"
48+
IMPORTED_LINK_INTERFACE_LIBRARIES "m;dl;pthread"
49+
)
50+
message(STATUS "Rust readers enabled (pre-built)")
51+
else()
52+
# Mode 2: Build via Corrosion (developer machines with cargo)
53+
find_program(CARGO_EXECUTABLE cargo)
54+
if (NOT CARGO_EXECUTABLE)
55+
message(WARNING "BUILD_RUST_READERS=ON but no pre-built libdoris_ffi.a and no cargo in PATH. Disabling.")
56+
set(BUILD_RUST_READERS OFF PARENT_SCOPE)
57+
return()
58+
endif()
59+
60+
include(FetchContent)
61+
FetchContent_Declare(
62+
Corrosion
63+
GIT_REPOSITORY https://github.com/corrosion-rs/corrosion.git
64+
GIT_TAG v0.5.1
65+
)
66+
FetchContent_MakeAvailable(Corrosion)
67+
68+
set(RUST_MANIFEST_PATH "${SRC_DIR}/rust/doris-native/Cargo.toml")
69+
corrosion_import_crate(
70+
MANIFEST_PATH ${RUST_MANIFEST_PATH}
71+
CRATES doris-ffi
72+
)
73+
74+
add_library(doris_ffi_lib STATIC IMPORTED GLOBAL)
75+
set_target_properties(doris_ffi_lib PROPERTIES
76+
IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/libdoris_ffi.a"
77+
IMPORTED_LINK_INTERFACE_LIBRARIES "m;dl;pthread"
78+
)
79+
add_dependencies(doris_ffi_lib cargo-build_doris_ffi)
80+
message(STATUS "Rust readers enabled (Corrosion)")
81+
endif()

‎be/src/exec/scan/file_scanner.cpp‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,9 @@
7979
#include "format/table/transactional_hive_reader.h"
8080
#include "format/table/trino_connector_jni_reader.h"
8181
#include "format/text/text_reader.h"
82+
#ifdef BUILD_RUST_READERS
83+
#include "format/lance/lance_rust_reader.h"
84+
#endif
8285
#include "io/cache/block_file_cache_profile.h"
8386
#include "load/group_commit/wal/wal_reader.h"
8487
#include "runtime/descriptors.h"
@@ -1140,6 +1143,16 @@ Status FileScanner::_get_next_reader() {
11401143
}
11411144
break;
11421145
}
1146+
#ifdef BUILD_RUST_READERS
1147+
case TFileFormatType::FORMAT_LANCE: {
1148+
auto lance_reader = LanceRustReader::create_unique(_file_slot_descs, _state, _profile,
1149+
range, _params);
1150+
init_status = lance_reader->init_reader();
1151+
_cur_reader = std::move(lance_reader);
1152+
need_to_get_parsed_schema = true;
1153+
break;
1154+
}
1155+
#endif
11431156
default:
11441157
return Status::NotSupported("Not supported create reader for file format: {}.",
11451158
to_string(_params->format_type));

‎be/src/format/CMakeLists.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,13 @@ set(LIBRARY_OUTPUT_PATH "${BUILD_DIR}/src/format")
2222
set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/src/format")
2323

2424
file(GLOB_RECURSE SRC_FILES CONFIGURE_DEPENDS *.cpp)
25+
26+
# Lance reader requires Rust static library (BUILD_RUST_READERS=ON)
27+
if (NOT BUILD_RUST_READERS)
28+
file(GLOB_RECURSE LANCE_FILES CONFIGURE_DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/lance/*.cpp)
29+
list(REMOVE_ITEM SRC_FILES ${LANCE_FILES})
30+
endif()
31+
2532
add_library(Format STATIC ${SRC_FILES})
2633

2734
pch_reuse(Format)

‎be/src/format/lance/lance_ffi.h‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#ifdef BUILD_RUST_READERS
21+
22+
#include <arrow/c/abi.h>
23+
24+
#include <cstddef>
25+
#include <cstdint>
26+
27+
namespace doris::lance_ffi {
28+
29+
// FFI status codes (must match Rust error.rs)
30+
constexpr int32_t LANCE_FFI_OK = 0;
31+
constexpr int32_t LANCE_FFI_EOF = 1;
32+
constexpr int32_t LANCE_FFI_ERR_LANCE = -1;
33+
constexpr int32_t LANCE_FFI_ERR_ARROW = -2;
34+
constexpr int32_t LANCE_FFI_ERR_IO = -3;
35+
constexpr int32_t LANCE_FFI_ERR_PANIC = -4;
36+
constexpr int32_t LANCE_FFI_ERR_INVALID_ARG = -5;
37+
38+
} // namespace doris::lance_ffi
39+
40+
// Opaque handle to a Rust LanceReader.
41+
using LanceReaderHandle = void*;
42+
43+
extern "C" {
44+
45+
/// Open a Lance dataset and create a reader.
46+
int32_t lance_reader_open(const uint8_t* uri_ptr, size_t uri_len,
47+
const uint8_t* const* column_names_ptr,
48+
const size_t* column_names_len_ptr, size_t num_columns, size_t batch_size,
49+
LanceReaderHandle* handle_out);
50+
51+
/// Read the next batch via Arrow C Data Interface.
52+
/// Returns LANCE_FFI_OK with data, LANCE_FFI_EOF on end, negative on error.
53+
int32_t lance_reader_next_batch(LanceReaderHandle handle, ArrowSchema* schema_out,
54+
ArrowArray* array_out, bool* eof_out, int64_t* bytes_out);
55+
56+
/// Get the schema of the scan output.
57+
int32_t lance_reader_get_schema(LanceReaderHandle handle, ArrowSchema* schema_out);
58+
59+
/// Close the reader and free resources. Safe to call with null handle.
60+
void lance_reader_close(LanceReaderHandle handle);
61+
62+
/// Retrieve the last error message. Returns bytes written (excluding null terminator).
63+
size_t lance_reader_last_error(uint8_t* buf, size_t buf_len);
64+
65+
/// Open a Lance dataset from a JSON config string.
66+
/// Config JSON: {"uri":"...", "columns":[], "batch_size":N, "version":N, "storage_options":{}}
67+
/// storage_options carries S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.)
68+
int32_t lance_reader_open_json(const uint8_t* config_json_ptr, size_t config_json_len,
69+
LanceReaderHandle* handle_out);
70+
71+
/// Phase 0 echo function for build verification.
72+
int32_t rust_echo(int32_t x);
73+
74+
/// Create a test Lance dataset at the given path. For GTests only.
75+
/// Dataset has 5 rows: id(INT32), name(UTF8), score(FLOAT64).
76+
int32_t lance_test_create_dataset(const uint8_t* path_ptr, size_t path_len);
77+
78+
/// Create a multi-fragment test dataset. 3 fragments, 5 rows each = 15 total.
79+
/// Columns: id(INT32), name(UTF8), value(FLOAT64).
80+
int32_t lance_test_create_multi_fragment_dataset(const uint8_t* path_ptr, size_t path_len);
81+
82+
} // extern "C"
83+
84+
#endif // BUILD_RUST_READERS

0 commit comments

Comments
 (0)