Repository navigation
Commit 5dcf106
authored
[opt](lance) Support distributed execution for vector_search (#66841)
### What problem does this PR solve?
support two phase in distributed execution for lance vector_search.
1. lance_scan_node will query nearest with query_vector and prefilter in
lance rust,
will return _row_id, _distance and some postfilter column from lance
scan result, and execute the where postfilter.
2. then local topn and global merge topn will get top_k rows.
3. MaterializeNode will use _row_id to fetch the payload project column
from lance rust.
```
mysql> desc SELECT row_id, category, label, _distance FROM vector_search( "table"="test_lance_vector_search_two_phase.doris.vs_ivf_pq_f32", "column"="embedding", "query_vector"="[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]", "top_k"="3", "metric"="l2", "nprobes"="4",
"refine_factor"="10", "use_index"="true") WHERE category = 'odd' ORDER BY _distance;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| row_id[#17] |
| category[#18] |
| label[#19] |
| _distance[#20] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCOL |
| |
| 4:VSORT(193) |
| | order by: _distance[#20] ASC |
| | algorithm: full sort |
| | offset: 0 |
| | distribute expr lists: |
| | |
| 3:VMaterializeNode |
| | materialize tuple id:3 |
| | output tuple id:4 |
| | projectList:[row_id, category, label, _distance] |
| | column_descs_lists[[`row_id` bigint NOT NULL, `label` text NOT NULL]] |
| | locations: [[2, 3]] |
| | column_idxs_lists: [[0, 2]] |
| | row_ids: [__DORIS_GLOBAL_ROWID_COL__vector_search] |
| | isTopMaterializeNode: true |
| | final projections: row_id[#11], category[#9], label[#12], _distance[#10] |
| | final project output tuple id: 4 |
| | |
| 2:VMERGING-EXCHANGE |
| offset: 0 |
| limit: 3 |
| distribute expr lists: |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: RANDOM |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 02 |
| UNPARTITIONED |
| IS_MERGE: true |
| |
| 1:VTOP-N(182) |
| | order by: _distance[#7] ASC |
| | algorithm: heap sort |
| | local merge sort |
| | merge by exchange |
| | offset: 0 |
| | limit: 3 |
| | distribute expr lists: |
| | |
| 0:VLANCE_SCAN_NODE(174) |
| table: null.null.VectorSearchTableValuedFunction<test_lance_vector_search_two_phase.doris.vs_ivf_pq_f32> |
| predicates: (_tvf_vector_search.category[#0] = 'odd') |
| inputSplitNum=2, totalFileSize=0, scanRanges=2 |
| partition=0/0 |
| numNodes=1 |
| pushdown agg=NONE |
| externalSearchType=VECTOR |
| lanceVectorColumn=embedding |
| lanceTopK=3 |
| lanceOffset=0 |
| lanceMetric=l2 |
| lanceVersion=3 |
| lanceSearchFragments=2 |
| final projections: _tvf_vector_search.category[#0], _tvf_vector_search._distance[#1], _tvf_vector_search.__DORIS_GLOBAL_ROWID_COL__vector_search[#2] |
| final project output tuple id: 1 |
| |
| |
| |
| ========== STATISTICS ========== |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
76 rows in set (0.04 sec)
mysql>
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->1 parent f77661a commit 5dcf106
45 files changed
Lines changed: 3126 additions & 404 deletions
File tree
- be
- src
- exec
- operator
- format_v2/table
- storage
- test
- exec/operator
- format_v2
- orc
- parquet
- table
- format
- orc
- parquet
- fe/fe-core/src
- main/java/org/apache/doris
- datasource
- lance
- source
- tvf/source
- nereids
- processor/post/materialize
- rules
- analysis
- rewrite
- tablefunction
- test/java/org/apache/doris
- datasource
- lance
- source
- tvf/source
- nereids/processor/post/materialize
- tablefunction
- gensrc
- proto
- thrift
- regression-test
- data/external_table_p0/lance
- suites/external_table_p0/lance
- thirdparty
- patches
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
69 | 121 | | |
70 | 122 | | |
71 | 123 | | |
| |||
446 | 498 | | |
447 | 499 | | |
448 | 500 | | |
449 | | - | |
450 | | - | |
451 | | - | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
452 | 504 | | |
453 | 505 | | |
454 | 506 | | |
455 | 507 | | |
456 | 508 | | |
457 | 509 | | |
458 | | - | |
459 | | - | |
460 | | - | |
461 | | - | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
462 | 523 | | |
463 | 524 | | |
464 | 525 | | |
| |||
0 commit comments