Repository navigation
Commit b613c37
[feature](lance) Add Rust-based Lance format reader for AI-native dat… (#62182)
Add native Lance format support to Doris via Rust FFI integration,
enabling SQL queries over AI-native Lance datasets from local disk and
S3.
[Lance](https://lancedb.github.io/lance/) is a columnar format designed
for vector search, multimodal data (images, embeddings), and fast random
access -- widely used in AI/ML pipelines.
## Quick Examples
**Read from S3:**
```sql
SELECT * FROM s3(
"uri" = "s3://bucket/embeddings.lance/data/fragment.lance",
"format" = "lance",
"s3.access_key" = "...",
"s3.secret_key" = "...",
"s3.region" = "us-east-1",
"s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
) ORDER BY id LIMIT 10;
```
**Read from local disk (for testing):**
```sql
-- Get backend_id from: SHOW BACKENDS;
SELECT * FROM local(
"file_path" = "data/my_dataset.lance/data/fragment.lance",
"backend_id" = "<backend_id from SHOW BACKENDS>",
"format" = "lance"
) ORDER BY id LIMIT 10;
```
**Aggregation across multi-fragment dataset:**
```sql
SELECT count(*), min(id), max(id) FROM s3(
"uri" = "s3://bucket/large.lance/data/fragment.lance",
"format" = "lance",
"s3.access_key" = "...", "s3.secret_key" = "...",
"s3.region" = "us-east-1",
"s3.endpoint" = "https://s3.us-east-1.amazonaws.com"
);
```
## Architecture
- **Data exchange**: Arrow C Data Interface (zero-copy between Rust and
C++)
- **Async containment**: block_on() with single-threaded tokio runtime
(zero extra OS threads)
- **Build gating**: BUILD_RUST_READERS=OFF by default, zero impact on
existing builds
## What Works (Verified on Live Cluster)
| Feature | Status |
|---------|--------|
| SELECT * / column projection | Tested |
| WHERE filter / LIMIT / COUNT(*) | Tested |
| SUM() / AVG() aggregation | Tested |
| Multi-fragment datasets (3 fragments, 15 rows) | Tested |
| S3 access with AWS credentials | Tested |
| Schema inference (fetch_table_schema) | Tested |
| Time travel version (config wired) | Config ready |
| Vector ANN search / FTS / filter pushdown | Config ready |
## Known Limitations
- **TVF path only**: No CREATE CATALOG support yet. Must use local() or
s3() TVF
- **Directory-based format workaround**: Lance datasets are directories.
The TVF file_path must point to a single `.lance` data file inside the
dataset; the reader auto-strips the path back to the dataset root and
reads all fragments. If the TVF glob matches multiple `.lance` files
(multi-fragment dataset), each scan range reopens the full dataset
causing **duplicate rows**. Workaround: ensure the file_path glob
matches exactly one data file per dataset
- **No Doris data cache integration**: Lance reads bypass
BlockFileCache. S3 reads are not cached on local SSD
- **No filter/vector pushdown from FE**: The Rust config supports
filter, vector_search, full_text_search but the FE planner does not
populate them yet
- **BUILD_RUST_READERS=OFF default**: Requires explicit opt-in and Rust
toolchain
- **Binary size**: Rust static lib ~430MB (.a), adds ~50-80MB to final
doris_be after LTO
## How to Build
```bash
# Rust tests only:
cd be/src/rust/doris-native && cargo test
# Full BE with Lance:
BUILD_RUST_READERS=ON ./build.sh --be
# Regression test:
./run-regression-test.sh --run -s test_lance_tvf
```
## Changes
**Thrift** (2 files): FORMAT_LANCE = 19, TLanceFileDesc,
enable_rust_lance_reader
**FE** (4 files): LanceFileFormatProperties, FileFormatProperties
factory, FileFormatConstants, SessionVariable
**BE - Rust** (be/src/rust/doris-native/, 6 files): error.rs,
lance_reader.rs (LanceReaderConfig with S3/version/vector/FTS support),
ffi.rs (extern C functions), lib.rs
**BE - C++** (5 files): lance_ffi.h, lance_rust_reader.h/cpp
(GenericReader with Arrow import), file_scanner.cpp (FORMAT_LANCE
dispatch), internal_service.cpp (fetch_table_schema)
**Build** (3 files): rust.cmake (Corrosion v0.5), CMakeLists.txt
(BUILD_RUST_READERS option), format/CMakeLists.txt
**Tests** (4 files): 24 Rust tests, C++ GTest (7), C++ standalone (8),
Groovy regression (9)
## Future Work
- **Lance Catalog**: CREATE CATALOG for dataset discovery (eliminates
TVF path limitations and backend_id requirement)
- **Fragment-level scan ranges**: FE lists fragments and creates one
scan range per fragment with fragment ID, avoiding duplicate reads
- **FE filter/vector pushdown**: Pass WHERE predicates and ANN queries
to lance-rs scanner
- **Doris BlockFileCache integration**: Route lance I/O through Doris
cache for S3 data caching
- **Lance Session cache**: Shared IndexCache for vector/FTS index reuse
across queries
- **Time travel SQL**: FOR VERSION AS OF N via LanceMvccSnapshot
---------
Co-authored-by: tom zhang <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: Mingyu Chen (Rayner) <[email protected]>1 parent 7554ffb commit b613c37
42 files changed
Lines changed: 8979 additions & 1 deletion
File tree
- be
- cmake
- src
- exec/scan
- format
- lance
- rust/doris-native
- crates/doris-ffi
- src
- service
- test
- format/lance
- fe/fe-core/src/main/java/org/apache/doris
- common/util
- datasource/property/fileformat
- qe
- gensrc/thrift
- regression-test
- data/external_table_p0/tvf/lance
- multi.lance
- _transactions
- _versions
- data
- single.lance
- _transactions
- _versions
- data
- suites/external_table_p0/tvf/lance
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
168 | 189 | | |
169 | 190 | | |
170 | 191 | | |
| |||
686 | 707 | | |
687 | 708 | | |
688 | 709 | | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
689 | 714 | | |
690 | 715 | | |
691 | 716 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
82 | 85 | | |
83 | 86 | | |
84 | 87 | | |
| |||
1140 | 1143 | | |
1141 | 1144 | | |
1142 | 1145 | | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
1143 | 1156 | | |
1144 | 1157 | | |
1145 | 1158 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
25 | 32 | | |
26 | 33 | | |
27 | 34 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
0 commit comments