Skip to content

Commit 60aebf8

Browse files
morningmanzhangstar333claude
committed
[feature](be) Honor the prefilter flag and group the Lance reader profile as in branch-4.x
### What problem does this PR solve? Issue Number: None Related PR: #69581, #68813 Problem Summary: #69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After that snapshot, #68813 changed two things in the 4.x BE, both ported here. 1. Search filter timing. #68813 added `TExternalSearchRequest.prefilter` (field 5): unset or false means the TVF search filter refines the top-k candidates (postfilter), true means it constrains them before top-k (prefilter). The 4.x FE sends it for `vector_search()` and `full_text_search()`. Without the field, a master BE always prefiltered vector search and never prefiltered full-text search, so a 4.x FE paired with it got different rows: on the all_types.lance fixture, top_k 1 with search filter `row_id >= 3` returns row 4 with prefilter and no row with postfilter. - `gensrc/thrift/PlanNodes.thrift`: `5: optional bool prefilter` with the 4.x comment; the stale `TSearchFilter` comment is replaced. - `_configure_vector_search()` and `_configure_full_text_search()` pass `request.__isset.prefilter && request.prefilter` to `lance_scanner_set_prefilter()`, replacing the comment that fragment-scoped nearest queries need prefiltering (master's lance-c pin 4fafcd64525 no longer does). - Compatibility (gensrc/AGENTS.md): field 5 is `optional bool prefilter` on the branch-4.2 (a7ae6f4) and branch-4.1 (3f911d0) heads; branch-4.0 and branch-3.1 have no `TExternalSearchRequest`. No id is reused. 2. Profile layout. The snapshot hangs about 45 counters directly under `LanceReader`, looks Lance's scan metrics up in two maps, and creates two scalar-index timers from Lance's statistics callback the first time they are reported, that is from Lance's execution threads while the scan runs. As in #68813: - The counters are grouped under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache` (label counters, `TUnit::NONE`, like `RuntimeFilterInfo`). - One immutable table, `LANCE_SCAN_METRICS` in `lance_reader_helper.h`, drives both registration and the callback lookup. Every metric is registered in `_init_scanner_profile()` before the callback is installed, so the callback only updates existing counters and every split has the same counter tree. - The callback is `noexcept` and swallows exceptions, so nothing unwinds through the lance-c ABI. - `LanceIndexObjectLoads` (`indices_loaded`), `LanceIndexCacheHits` and `LanceIndexCacheMisses` are new; the info strings `LanceTopKPlusOffset` and `LanceSubstraitFilterBytes` are dropped. - Renames: `LanceScannerReadTime` -> `LanceScannerNextTime`, `LanceRuntimeFilterSqlTime` -> `LanceRuntimeFilterSqlTranslationTime`, `LanceExecutionIOOps` / `IORequests` / `IOBytesRead` -> `LanceIOReadOps` / `LanceIOReadRequests` / `LanceIOReadBytes`, `LanceIndexPartitionCacheMissLoads` -> `LanceIndexComponentsLoaded`, `LancePrefilter*` -> `RowIdPrefilter*` (`LancePrefilterRowIds` -> `RowIdPrefilterIds`), `LanceIVFPartitionsRanked` -> `LanceIVFPartitionsConsidered`, `LanceIVFPartitionsSearched` -> `LanceIndexPartitionsSearched`, `LanceVectorIndexSegmentsSearched` -> `LanceVectorIndexDeltasSearched`, `LanceTaskWaitTime` -> `LanceScanTaskWaitTime`, `LanceIVFPartitionRankingTime` -> `LanceIVFPartitionSelectionTime`, `LanceScalarIndexSegmentPrepareTime` -> `LanceScalarIndexCandidateEvaluationTime`. - Master differs from 4.x in one line: the callback asserts with `DCHECK_LT` that a found metric has a registered counter, where 4.x checks it in the `if`; both come from the same array. - Not ported: #68813's FE change that makes `AggCounter` print nothing for `TUnit.NONE`. It belongs to the FE port; until then master's merged profile prints the group labels like the existing `RuntimeFilterInfo`. ### Release note None ### Check List (For Author) - Test: Unit Test - New: `LanceTableReaderVectorSearchTest.SearchFilterDefaultsToPostfilter` (from #68813) and `LanceTableReaderFullTextSearchTest.SearchFilterHonorsPrefilterFlag` (top_k 1 with filter `row_id <= 2`: no row with postfilter, row 2 with prefilter; 4.x has no BE test for the full-text path). - Changed as in #68813: `AppliesSearchFilterBeforeTopK` becomes `AppliesSearchFilterBeforeTopKWhenPrefilterEnabled`; `MultiVectorScoresFiltersOffsetsAndIndexedCoverage` and `MultiVectorSegmentTopKMatchesIndependentGlobalOracle` set `prefilter` for their filtered cases, and the former checks the `RowIdPrefilter*` counters and that a flat split exposes `LanceIndexPartitionLoadTime`. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With vector search always prefiltering and full-text search never prefiltering, the two new cases fail. - Behavior changed: Yes, on the Lance path added by #69581. A search filter without `prefilter` now postfilters vector search, and full-text search honors `prefilter = true`, as on branch-4.1/4.2. Lance profile counters are grouped and renamed. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zhangstar333 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent bf5dcbd commit 60aebf8

6 files changed

Lines changed: 401 additions & 221 deletions

File tree

‎be/src/format_v2/lance/lance_reader_helper.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,14 @@
4444
#include "exec/common/endian.h"
4545

4646
namespace doris::format::lance {
47+
48+
RuntimeProfile::Counter* add_lance_counter(RuntimeProfile* profile, const char* name,
49+
TUnit::type unit, const char* group) {
50+
// These counters label sections; they do not represent time or a sum of their children.
51+
ADD_CHILD_COUNTER_WITH_LEVEL(profile, group, TUnit::NONE, LANCE_READER_PROFILE, 1);
52+
return ADD_CHILD_COUNTER_WITH_LEVEL(profile, name, unit, group, 1);
53+
}
54+
4755
namespace {
4856

4957
constexpr std::string_view ARROW_EXTENSION_NAME = "ARROW:extension:name";

‎be/src/format_v2/lance/lance_reader_helper.h‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
#include "common/status.h"
2828
#include "core/data_type/data_type.h"
2929
#include "gen_cpp/PlanNodes_types.h"
30+
#include "runtime/runtime_profile.h"
3031

3132
struct LanceBatch;
3233
struct LanceDataset;
@@ -46,6 +47,203 @@ inline constexpr std::string_view LANCE_DISTANCE_COLUMN = "_distance";
4647
inline constexpr std::string_view LANCE_SCORE_COLUMN = "_score";
4748
inline constexpr std::string_view LANCE_ROW_ID_COLUMN = "_rowid";
4849
inline constexpr const char* LANCE_READER_PROFILE = "LanceReader";
50+
inline constexpr const char* LANCE_TIMING_PROFILE = "LanceTiming";
51+
inline constexpr const char* LANCE_IO_PROFILE = "LanceIO";
52+
inline constexpr const char* LANCE_INDEX_PROFILE = "LanceIndex";
53+
inline constexpr const char* LANCE_VECTOR_INDEX_PROFILE = "LanceVectorIndex";
54+
inline constexpr const char* LANCE_SCALAR_INDEX_PROFILE = "LanceScalarIndex";
55+
inline constexpr const char* LANCE_EXECUTION_PLAN_PROFILE = "LanceExecutionPlan";
56+
inline constexpr const char* LANCE_SCAN_PROFILE = "LanceScan";
57+
inline constexpr const char* LANCE_SEARCH_PLAN_PROFILE = "LanceSearchPlan";
58+
inline constexpr const char* LANCE_DATA_CACHE_PROFILE = "LanceDataCache";
59+
60+
struct LanceProfileMetric {
61+
std::string_view native_name;
62+
const char* profile_name;
63+
const char* group;
64+
TUnit::type unit;
65+
};
66+
67+
// Lance invokes the statistics callback from its execution streams. Keep the metric definitions
68+
// immutable and pre-register their counters before installing the callback. The callback then
69+
// updates only metrics reported by Lance without changing the profile counter tree.
70+
inline constexpr LanceProfileMetric LANCE_SCAN_METRICS[] = {
71+
{.native_name = "index_cache_hits",
72+
.profile_name = "LanceIndexCacheHits",
73+
.group = LANCE_INDEX_PROFILE,
74+
.unit = TUnit::UNIT},
75+
{.native_name = "index_cache_misses",
76+
.profile_name = "LanceIndexCacheMisses",
77+
.group = LANCE_INDEX_PROFILE,
78+
.unit = TUnit::UNIT},
79+
// Row-ID prefilter loader metrics are separate from ANN/FTS result counts.
80+
{.native_name = "prefilter_loads",
81+
.profile_name = "RowIdPrefilterLoads",
82+
.group = LANCE_SCAN_PROFILE,
83+
.unit = TUnit::UNIT},
84+
{.native_name = "prefilter_input_rows",
85+
.profile_name = "RowIdPrefilterInputRows",
86+
.group = LANCE_SCAN_PROFILE,
87+
.unit = TUnit::UNIT},
88+
{.native_name = "prefilter_input_batches",
89+
.profile_name = "RowIdPrefilterInputBatches",
90+
.group = LANCE_SCAN_PROFILE,
91+
.unit = TUnit::UNIT},
92+
{.native_name = "prefilter_row_ids",
93+
.profile_name = "RowIdPrefilterIds",
94+
.group = LANCE_SCAN_PROFILE,
95+
.unit = TUnit::UNIT},
96+
// LoadTime includes input polling and set construction, so it overlaps component timers.
97+
{.native_name = "prefilter_load_time",
98+
.profile_name = "RowIdPrefilterLoadTime",
99+
.group = LANCE_SCAN_PROFILE,
100+
.unit = TUnit::TIME_NS},
101+
{.native_name = "prefilter_input_time",
102+
.profile_name = "RowIdPrefilterInputTime",
103+
.group = LANCE_SCAN_PROFILE,
104+
.unit = TUnit::TIME_NS},
105+
{.native_name = "prefilter_build_time",
106+
.profile_name = "RowIdPrefilterBuildTime",
107+
.group = LANCE_SCAN_PROFILE,
108+
.unit = TUnit::TIME_NS},
109+
{.native_name = "fragments_scanned",
110+
.profile_name = "LanceFragmentsScanned",
111+
.group = LANCE_SCAN_PROFILE,
112+
.unit = TUnit::UNIT},
113+
{.native_name = "ranges_scanned",
114+
.profile_name = "LanceRowOffsetRangesScanned",
115+
.group = LANCE_SCAN_PROFILE,
116+
.unit = TUnit::UNIT},
117+
{.native_name = "rows_scanned",
118+
.profile_name = "LanceRowsScanned",
119+
.group = LANCE_SCAN_PROFILE,
120+
.unit = TUnit::UNIT},
121+
// This is Lance scan execution wait time, not Doris scanner scheduling wait time.
122+
{.native_name = "task_wait_time",
123+
.profile_name = "LanceScanTaskWaitTime",
124+
.group = LANCE_SCAN_PROFILE,
125+
.unit = TUnit::TIME_NS},
126+
{.native_name = "partitions_ranked",
127+
.profile_name = "LanceIVFPartitionsConsidered",
128+
.group = LANCE_VECTOR_INDEX_PROFILE,
129+
.unit = TUnit::UNIT},
130+
// Lance also reports partitions_searched for FTS, so keep it in the generic index group.
131+
{.native_name = "partitions_searched",
132+
.profile_name = "LanceIndexPartitionsSearched",
133+
.group = LANCE_INDEX_PROFILE,
134+
.unit = TUnit::UNIT},
135+
{.native_name = "deltas_searched",
136+
.profile_name = "LanceVectorIndexDeltasSearched",
137+
.group = LANCE_VECTOR_INDEX_PROFILE,
138+
.unit = TUnit::UNIT},
139+
// These partition stages accumulate across concurrent work and can overlap each other.
140+
{.native_name = "find_partitions_elapsed",
141+
.profile_name = "LanceIVFPartitionSelectionTime",
142+
.group = LANCE_VECTOR_INDEX_PROFILE,
143+
.unit = TUnit::TIME_NS},
144+
{.native_name = "scalar_segments_requested",
145+
.profile_name = "LanceScalarIndexSegmentsRequested",
146+
.group = LANCE_SCALAR_INDEX_PROFILE,
147+
.unit = TUnit::UNIT},
148+
{.native_name = "scalar_segments_searched",
149+
.profile_name = "LanceScalarIndexSegmentsSearched",
150+
.group = LANCE_SCALAR_INDEX_PROFILE,
151+
.unit = TUnit::UNIT},
152+
{.native_name = "scalar_segment_fallbacks",
153+
.profile_name = "LanceScalarIndexSegmentFallbacks",
154+
.group = LANCE_SCALAR_INDEX_PROFILE,
155+
.unit = TUnit::UNIT},
156+
{.native_name = "scalar_segment_candidate_rows",
157+
.profile_name = "LanceScalarIndexCandidateRows",
158+
.group = LANCE_SCALAR_INDEX_PROFILE,
159+
.unit = TUnit::UNIT},
160+
{.native_name = "scalar_segment_prepare_time",
161+
.profile_name = "LanceScalarIndexCandidateEvaluationTime",
162+
.group = LANCE_SCALAR_INDEX_PROFILE,
163+
.unit = TUnit::TIME_NS},
164+
{.native_name = "scalar_segment_search_time",
165+
.profile_name = "LanceScalarIndexSegmentSearchTime",
166+
.group = LANCE_SCALAR_INDEX_PROFILE,
167+
.unit = TUnit::TIME_NS},
168+
{.native_name = "search_time",
169+
.profile_name = "LanceScalarIndexQueryTime",
170+
.group = LANCE_SCALAR_INDEX_PROFILE,
171+
.unit = TUnit::TIME_NS},
172+
{.native_name = "serialization_time",
173+
.profile_name = "LanceScalarIndexResultSerializationTime",
174+
.group = LANCE_SCALAR_INDEX_PROFILE,
175+
.unit = TUnit::TIME_NS},
176+
// Index partition times overlap; DistanceTopK includes candidate filtering, scoring,
177+
// and heap updates.
178+
{.native_name = "index_open_time",
179+
.profile_name = "LanceIndexOpenTime",
180+
.group = LANCE_INDEX_PROFILE,
181+
.unit = TUnit::TIME_NS},
182+
{.native_name = "index_partition_load_time",
183+
.profile_name = "LanceIndexPartitionLoadTime",
184+
.group = LANCE_VECTOR_INDEX_PROFILE,
185+
.unit = TUnit::TIME_NS},
186+
{.native_name = "index_partition_prepare_time",
187+
.profile_name = "LanceIndexPartitionPrepareTime",
188+
.group = LANCE_VECTOR_INDEX_PROFILE,
189+
.unit = TUnit::TIME_NS},
190+
{.native_name = "index_prefilter_wait_time",
191+
.profile_name = "LanceIndexPrefilterWaitTime",
192+
.group = LANCE_VECTOR_INDEX_PROFILE,
193+
.unit = TUnit::TIME_NS},
194+
{.native_name = "index_cpu_queue_wait_time",
195+
.profile_name = "LanceIndexCpuQueueWaitTime",
196+
.group = LANCE_VECTOR_INDEX_PROFILE,
197+
.unit = TUnit::TIME_NS},
198+
{.native_name = "index_search_time",
199+
.profile_name = "LanceIndexSearchTime",
200+
.group = LANCE_VECTOR_INDEX_PROFILE,
201+
.unit = TUnit::TIME_NS},
202+
{.native_name = "index_query_prepare_time",
203+
.profile_name = "LanceIndexQueryPrepareTime",
204+
.group = LANCE_VECTOR_INDEX_PROFILE,
205+
.unit = TUnit::TIME_NS},
206+
{.native_name = "index_distance_topk_time",
207+
.profile_name = "LanceIndexDistanceTopKTime",
208+
.group = LANCE_VECTOR_INDEX_PROFILE,
209+
.unit = TUnit::TIME_NS},
210+
{.native_name = "index_result_materialize_time",
211+
.profile_name = "LanceIndexResultMaterializeTime",
212+
.group = LANCE_VECTOR_INDEX_PROFILE,
213+
.unit = TUnit::TIME_NS},
214+
// These execution-plan stage timers are also cumulative across concurrent tasks.
215+
{.native_name = "ANNIVFPartitionExec_elapsed_compute",
216+
.profile_name = "LanceANNPartitionExecTime",
217+
.group = LANCE_VECTOR_INDEX_PROFILE,
218+
.unit = TUnit::TIME_NS},
219+
{.native_name = "ANNSubIndexExec_elapsed_compute",
220+
.profile_name = "LanceANNSubIndexExecTime",
221+
.group = LANCE_VECTOR_INDEX_PROFILE,
222+
.unit = TUnit::TIME_NS},
223+
{.native_name = "ANNIvfBatchExec_elapsed_compute",
224+
.profile_name = "LanceANNBatchExecTime",
225+
.group = LANCE_VECTOR_INDEX_PROFILE,
226+
.unit = TUnit::TIME_NS},
227+
{.native_name = "SortExec_elapsed_compute",
228+
.profile_name = "LanceSortComputeTime",
229+
.group = LANCE_EXECUTION_PLAN_PROFILE,
230+
.unit = TUnit::TIME_NS},
231+
{.native_name = "SortPreservingMergeExec_elapsed_compute",
232+
.profile_name = "LanceSortMergeComputeTime",
233+
.group = LANCE_EXECUTION_PLAN_PROFILE,
234+
.unit = TUnit::TIME_NS},
235+
{.native_name = "TakeExec_elapsed_compute",
236+
.profile_name = "LanceTakeExecTime",
237+
.group = LANCE_EXECUTION_PLAN_PROFILE,
238+
.unit = TUnit::TIME_NS},
239+
{.native_name = "KNNVectorDistanceExec_elapsed_compute",
240+
.profile_name = "LanceVectorDistanceComputeTime",
241+
.group = LANCE_VECTOR_INDEX_PROFILE,
242+
.unit = TUnit::TIME_NS},
243+
};
244+
245+
RuntimeProfile::Counter* add_lance_counter(RuntimeProfile* profile, const char* name,
246+
TUnit::type unit, const char* group);
49247

50248
struct LanceDatasetDeleter {
51249
void operator()(LanceDataset* dataset) const;

0 commit comments

Comments
 (0)