Repository navigation
Commit d306323
[feature](lance) Support distance bounds in vector_search (#68815)
### What problem does this PR solve?
Issue Number: Closes #66493
`vector_search()` cannot currently restrict results by distance. This
adds optional `distance_lower_bound` (inclusive) and
`distance_upper_bound` (exclusive) parameters for single-vector
Radius/Range Search on Lance tables.
The FE validates finite FLOAT bounds and their order, includes the range
in EXPLAIN, and carries it through the external-search request. The BE
applies the bounds through the existing pinned lance-c API after
configuring the query vector, covering indexed and Flat Search splits,
including appended data outside index coverage. The existing
external-search schema version remains unchanged at 1.
Results retain the existing table columns and FLOAT `_distance`,
prefilter semantics, offset, and Top-K cap. Bounds can be bound through
prepared-statement parameters and are revalidated on each execution.
Multi-vector distance bounds are rejected.
Bounded index search is limited to explicitly identified IVF_FLAT
segments with both bounds present. Quantized or unknown index types and
single-sided ranges fall back to exact fragment scans, reported as
`DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement
quantization errors and the pinned index implementation’s finite
sentinels for absent bounds, at the cost of additional scanning.
Unbounded queries retain their index selection; indexed IVF_FLAT recall
still depends on probed partitions.
Standalone BM25 is already delivered; the distributed BM25 statistics
follow-up remains tracked separately in #67435.
### Release note
Lance `vector_search()` supports optional lower and upper distance
bounds for single-vector radius and range queries.
### Validation
- BE: rebuilt the current branch and passed all 68 `LanceTableReader*`
tests under ASAN, including exact SQ threshold crossing, the finite
FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage.
- FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`,
`LancePreparedSearchTest`, `LancePreparedStatementTest`, and
`LanceScanNodeTest`, including repeated bound binding, invalid
parameters, all index subtypes, and indexed/flat split selection.
- Native reproduction: the pinned indexed paths drop both the SQ
threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat
Search returns both. Added self-contained fixtures and reader coverage
for these cases.
- Native execution: 10 cases passed against the pinned lance-c revision,
covering indexed/Flat Search, appended unindexed data,
inclusive/exclusive bounds, empty results, prefiltering, and offset.
- FE Checkstyle and clang-format 16 checks passed.
- Added SQL regression coverage for range results, projected stored
columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy
syntax compilation passed; the SQL suite has not yet run against a live
Doris cluster and requires External Regression CI.
### Check List (For Author)
- Test
- [x] Regression test added (execution pending CI)
- [x] Unit Test
- [x] Manual native execution test
- Behavior changed:
- [x] Yes: optional distance bounds on single-vector searches; existing
unbounded queries retain their behavior.
- Does this need documentation?
- [x] Yes: bilingual 4.x Lance Catalog documentation:
apache/doris-website#4200.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 698a018 commit d306323
40 files changed
Lines changed: 638 additions & 4 deletions
File tree
- be
- src/format_v2/table
- test/format_v2/table
- lance
- data
- distance_range.lance
- _indices/9634c881-df64-46a5-95dc-2468928008a5
- _transactions
- _versions
- data
- distance_range_dot.lance
- _indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e
- _transactions
- _versions
- data
- distance_range_sq.lance
- _indices/16361222-c72d-4428-9826-beeb3bfd3667
- _transactions
- _versions
- data
- fe/fe-core/src
- main/java/org/apache/doris
- datasource/lance/source
- nereids/parser
- tablefunction
- test/java/org/apache/doris
- datasource/lance/source
- tablefunction
- gensrc/thrift
- regression-test/suites/external_table_p0/lance
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
448 | 448 | | |
449 | 449 | | |
450 | 450 | | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
451 | 473 | | |
452 | 474 | | |
453 | 475 | | |
| |||
1177 | 1199 | | |
1178 | 1200 | | |
1179 | 1201 | | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
1180 | 1213 | | |
1181 | 1214 | | |
1182 | 1215 | | |
| |||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
0 commit comments