Skip to content

Commit d306323

Browse files
Gabriel39yiguolei
authored andcommitted
[feature](lance) Support distance bounds in vector_search (#68815)
### What problem does this PR solve? Issue Number: Closes #66493 `vector_search()` cannot currently restrict results by distance. This adds optional `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) parameters for single-vector Radius/Range Search on Lance tables. The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1. Results retain the existing table columns and FLOAT `_distance`, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected. Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as `DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions. Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435. ### Release note Lance `vector_search()` supports optional lower and upper distance bounds for single-vector radius and range queries. ### Validation - BE: rebuilt the current branch and passed all 68 `LanceTableReader*` tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage. - FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`, `LancePreparedSearchTest`, `LancePreparedStatementTest`, and `LanceScanNodeTest`, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection. - Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases. - Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset. - FE Checkstyle and clang-format 16 checks passed. - Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI. ### Check List (For Author) - Test - [x] Regression test added (execution pending CI) - [x] Unit Test - [x] Manual native execution test - Behavior changed: - [x] Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior. - Does this need documentation? - [x] Yes: bilingual 4.x Lance Catalog documentation: apache/doris-website#4200. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 698a018 commit d306323

40 files changed

Lines changed: 638 additions & 4 deletions

File tree

‎be/src/format_v2/table/lance_reader.cpp‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -448,6 +448,28 @@ Status LanceTableReader::_validate_external_search_request() const {
448448
static_cast<int>(query_vector.element_type));
449449
}
450450
const bool multi_vector = query_vector.__isset.num_vectors;
451+
const bool has_distance_range =
452+
vector.__isset.distance_lower_bound || vector.__isset.distance_upper_bound;
453+
if (has_distance_range && multi_vector) {
454+
return Status::NotSupported(
455+
"Distance bounds are not supported for Lance multi-vector search");
456+
}
457+
// Validate doubles before narrowing: non-finite or out-of-range values cannot reach the C API.
458+
const auto valid_bound = [](double bound) {
459+
return std::isfinite(bound) && std::abs(bound) <= std::numeric_limits<float>::max();
460+
};
461+
if ((vector.__isset.distance_lower_bound && !valid_bound(vector.distance_lower_bound)) ||
462+
(vector.__isset.distance_upper_bound && !valid_bound(vector.distance_upper_bound))) {
463+
return Status::InvalidArgument("Lance distance bounds must be finite FLOAT values");
464+
}
465+
if (vector.__isset.distance_lower_bound && vector.__isset.distance_upper_bound &&
466+
static_cast<float>(vector.distance_lower_bound) >=
467+
static_cast<float>(vector.distance_upper_bound)) {
468+
return Status::InvalidArgument(
469+
"distance_lower_bound must be less than distance_upper_bound at FLOAT "
470+
"precision");
471+
}
472+
451473
// The optional count distinguishes a query matrix, including a one-row matrix.
452474
if (multi_vector && query_vector.num_vectors <= 0) {
453475
return Status::InvalidArgument(
@@ -1177,6 +1199,17 @@ Status LanceTableReader::_configure_vector_search(LanceScanner* scanner,
11771199
}
11781200
}
11791201

1202+
if (vector.__isset.distance_lower_bound || vector.__isset.distance_upper_bound) {
1203+
const float lower = static_cast<float>(vector.distance_lower_bound);
1204+
const float upper = static_cast<float>(vector.distance_upper_bound);
1205+
// nearest() clears the range; install bounds only after setting the query vector.
1206+
if (lance_scanner_set_distance_range(
1207+
scanner, vector.__isset.distance_lower_bound ? &lower : nullptr,
1208+
vector.__isset.distance_upper_bound ? &upper : nullptr) != 0) {
1209+
return lance_error("set Lance vector distance range");
1210+
}
1211+
}
1212+
11801213
// Exact refinement validates stored elements and gives indexed and unindexed candidates
11811214
// the same row-level score before either path truncates its results.
11821215
if (query.__isset.num_vectors && lance_scanner_set_refine_factor(scanner, 1) != 0) {
Binary file not shown.
Binary file not shown.
Binary file not shown.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"version":3}

0 commit comments

Comments
 (0)