Repository navigation
Commit e3289c1
authored
[feature](lance) support lance catalog in doris (#65730)
### What problem does this PR solve?
Issue #66340
Problem Summary:
doc: apache/doris-website#4024
This PR introduces read support for the Lance format in Doris,
including:
1. Querying Lance datasets through a Lance Catalog.
2. Pushing supported predicates down to Lance to reduce the amount of
data scanned.
3. Reading Lance datasets directly through S3 and Local TVFs.
4. Performing vector similarity search through the vector_search() TVF,
with the distance exposed through the _distance column.
5. Lance is currently read-only and uses File Scanner V2. Writing to
Lance is not supported yet.
```
-- Create a Lance Catalog
CREATE CATALOG lance_catalog PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "filesystem",
"warehouse" = "s3://warehouse/lance",
"s3.endpoint" = "http://127.0.0.1:9000",
"s3.access_key" = "admin",
"s3.secret_key" = "password",
"s3.region" = "us-east-1",
"use_path_style" = "true"
);
-- Regular query; supported WHERE predicates are pushed down to Lance
SELECT user_id, user_name
FROM lance_catalog.doris.user_profiles
WHERE engagement_score >= 90;
-- Read a Lance dataset directly through the S3 TVF
SELECT user_id, user_name
FROM s3(
"uri" = "s3://warehouse/lance/doris/user_profiles.lance",
"format" = "lance",
"s3.endpoint" = "http://127.0.0.1:9000",
"s3.access_key" = "admin",
"s3.secret_key" = "password",
"s3.region" = "us-east-1",
"use_path_style" = "true"
);
-- Perform a vector similarity search
SELECT row_id, label, _distance
FROM vector_search(
"table" = "lance_catalog.doris.vector_search",
"column" = "embedding",
"query_vector" = "[0,0,0,0]",
"top_k" = "5",
"metric" = "l2"
)
ORDER BY _distance, row_id;
```
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->1 parent cc2d6a8 commit e3289c1
130 files changed
Lines changed: 12324 additions & 122 deletions
File tree
- be
- src
- core/data_type_serde
- exec
- operator
- scan
- format_v2
- table
- format
- service
- test
- core/data_type_serde
- exec/scan
- format_v2/table
- lance/data/all_types.lance
- _transactions
- _versions
- data
- docker/thirdparties
- docker-compose/iceberg
- scripts
- create_preinstalled_scripts/lance
- preinstalled_data/lance
- __manifest
- _indices
- 859ed2a5-7560-4701-a5be-e93d512df107
- a72dde56-017f-4679-a8c8-f2898fa14c02
- cd4c2b88-a91b-4ad4-9957-c98473f1e258
- _transactions
- _versions
- data
- all_types.lance
- _transactions
- _versions
- data
- fe
- fe-core
- src
- main/java/org/apache/doris
- catalog
- common/util
- datasource
- lance
- source
- property
- fileformat
- metastore
- tvf/source
- nereids
- glue/translator
- rules/analysis
- trees/expressions/functions/table
- persist/gson
- statistics
- tablefunction
- test/java/org/apache/doris
- common/util
- datasource
- lance
- tvf/source
- tablefunction
- gensrc/thrift
- regression-test
- conf
- data
- connector_p0/spark_connector
- external_table_p0/lance
- pipeline
- external/conf
- nonConcurrent/conf
- p0/conf
- suites/external_table_p0
- lance
- trino_connector
- thirdparty
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
96 | 105 | | |
97 | 106 | | |
98 | 107 | | |
| |||
210 | 219 | | |
211 | 220 | | |
212 | 221 | | |
213 | | - | |
| 222 | + | |
214 | 223 | | |
215 | 224 | | |
216 | | - | |
| 225 | + | |
| 226 | + | |
217 | 227 | | |
218 | | - | |
| 228 | + | |
219 | 229 | | |
220 | 230 | | |
221 | | - | |
| 231 | + | |
| 232 | + | |
222 | 233 | | |
223 | 234 | | |
224 | | - | |
| 235 | + | |
225 | 236 | | |
226 | 237 | | |
227 | 238 | | |
228 | 239 | | |
229 | 240 | | |
230 | 241 | | |
231 | | - | |
| 242 | + | |
232 | 243 | | |
233 | 244 | | |
234 | 245 | | |
235 | 246 | | |
236 | 247 | | |
237 | | - | |
| 248 | + | |
238 | 249 | | |
239 | 250 | | |
240 | 251 | | |
| |||
253 | 264 | | |
254 | 265 | | |
255 | 266 | | |
256 | | - | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
257 | 283 | | |
258 | 284 | | |
259 | 285 | | |
| |||
267 | 293 | | |
268 | 294 | | |
269 | 295 | | |
270 | | - | |
| 296 | + | |
271 | 297 | | |
272 | 298 | | |
273 | 299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
326 | 326 | | |
327 | 327 | | |
328 | 328 | | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
355 | 391 | | |
356 | 392 | | |
357 | 393 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
619 | 619 | | |
620 | 620 | | |
621 | 621 | | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
622 | 625 | | |
623 | 626 | | |
624 | 627 | | |
625 | 628 | | |
626 | 629 | | |
627 | 630 | | |
628 | | - | |
629 | | - | |
630 | 631 | | |
631 | | - | |
632 | | - | |
633 | | - | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
634 | 643 | | |
635 | 644 | | |
636 | | - | |
637 | | - | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
638 | 648 | | |
639 | 649 | | |
640 | 650 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
204 | 204 | | |
205 | 205 | | |
206 | 206 | | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
218 | 260 | | |
219 | 261 | | |
220 | 262 | | |
| |||
0 commit comments