Repository navigation
Commit 60aebf8
[feature](be) Honor the prefilter flag and group the Lance reader profile as in branch-4.x
### What problem does this PR solve?
Issue Number: None
Related PR: #69581, #68813
Problem Summary:
#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, #68813 changed two things in the 4.x BE, both ported here.
1. Search filter timing. #68813 added `TExternalSearchRequest.prefilter`
(field 5): unset or false means the TVF search filter refines the top-k
candidates (postfilter), true means it constrains them before top-k
(prefilter). The 4.x FE sends it for `vector_search()` and
`full_text_search()`. Without the field, a master BE always prefiltered
vector search and never prefiltered full-text search, so a 4.x FE paired
with it got different rows: on the all_types.lance fixture, top_k 1 with
search filter `row_id >= 3` returns row 4 with prefilter and no row with
postfilter.
- `gensrc/thrift/PlanNodes.thrift`: `5: optional bool prefilter` with the
4.x comment; the stale `TSearchFilter` comment is replaced.
- `_configure_vector_search()` and `_configure_full_text_search()` pass
`request.__isset.prefilter && request.prefilter` to
`lance_scanner_set_prefilter()`, replacing the comment that
fragment-scoped nearest queries need prefiltering (master's lance-c pin
4fafcd64525 no longer does).
- Compatibility (gensrc/AGENTS.md): field 5 is `optional bool prefilter` on
the branch-4.2 (a7ae6f4) and branch-4.1 (3f911d0) heads;
branch-4.0 and branch-3.1 have no `TExternalSearchRequest`. No id is
reused.
2. Profile layout. The snapshot hangs about 45 counters directly under
`LanceReader`, looks Lance's scan metrics up in two maps, and creates two
scalar-index timers from Lance's statistics callback the first time they
are reported, that is from Lance's execution threads while the scan runs.
As in #68813:
- The counters are grouped under `LanceTiming`, `LanceIO`, `LanceIndex`,
`LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`,
`LanceScan`, `LanceSearchPlan` and `LanceDataCache` (label counters,
`TUnit::NONE`, like `RuntimeFilterInfo`).
- One immutable table, `LANCE_SCAN_METRICS` in `lance_reader_helper.h`,
drives both registration and the callback lookup. Every metric is
registered in `_init_scanner_profile()` before the callback is installed,
so the callback only updates existing counters and every split has the
same counter tree.
- The callback is `noexcept` and swallows exceptions, so nothing unwinds
through the lance-c ABI.
- `LanceIndexObjectLoads` (`indices_loaded`), `LanceIndexCacheHits` and
`LanceIndexCacheMisses` are new; the info strings `LanceTopKPlusOffset`
and `LanceSubstraitFilterBytes` are dropped.
- Renames: `LanceScannerReadTime` -> `LanceScannerNextTime`,
`LanceRuntimeFilterSqlTime` -> `LanceRuntimeFilterSqlTranslationTime`,
`LanceExecutionIOOps` / `IORequests` / `IOBytesRead` -> `LanceIOReadOps` /
`LanceIOReadRequests` / `LanceIOReadBytes`,
`LanceIndexPartitionCacheMissLoads` -> `LanceIndexComponentsLoaded`,
`LancePrefilter*` -> `RowIdPrefilter*` (`LancePrefilterRowIds` ->
`RowIdPrefilterIds`), `LanceIVFPartitionsRanked` ->
`LanceIVFPartitionsConsidered`, `LanceIVFPartitionsSearched` ->
`LanceIndexPartitionsSearched`, `LanceVectorIndexSegmentsSearched` ->
`LanceVectorIndexDeltasSearched`, `LanceTaskWaitTime` ->
`LanceScanTaskWaitTime`, `LanceIVFPartitionRankingTime` ->
`LanceIVFPartitionSelectionTime`, `LanceScalarIndexSegmentPrepareTime` ->
`LanceScalarIndexCandidateEvaluationTime`.
- Master differs from 4.x in one line: the callback asserts with
`DCHECK_LT` that a found metric has a registered counter, where 4.x
checks it in the `if`; both come from the same array.
- Not ported: #68813's FE change that makes `AggCounter` print nothing for
`TUnit.NONE`. It belongs to the FE port; until then master's merged
profile prints the group labels like the existing `RuntimeFilterInfo`.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- New: `LanceTableReaderVectorSearchTest.SearchFilterDefaultsToPostfilter`
(from #68813) and `LanceTableReaderFullTextSearchTest.SearchFilterHonorsPrefilterFlag`
(top_k 1 with filter `row_id <= 2`: no row with postfilter, row 2 with
prefilter; 4.x has no BE test for the full-text path).
- Changed as in #68813: `AppliesSearchFilterBeforeTopK` becomes
`AppliesSearchFilterBeforeTopKWhenPrefilterEnabled`;
`MultiVectorScoresFiltersOffsetsAndIndexedCoverage` and
`MultiVectorSegmentTopKMatchesIndependentGlobalOracle` set `prefilter`
for their filtered cases, and the former checks the `RowIdPrefilter*`
counters and that a flat split exposes `LanceIndexPartitionLoadTime`.
- All pass on macOS arm64 (ASAN) against master's pinned lance-c,
with the Lance suites. With vector search always prefiltering and
full-text search never prefiltering, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by #69581. A search filter
without `prefilter` now postfilters vector search, and full-text search
honors `prefilter = true`, as on branch-4.1/4.2. Lance profile counters are
grouped and renamed. Master has no FE that plans Lance scans yet.
- Does this need documentation: No
Co-authored-by: zhangstar333 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent bf5dcbd commit 60aebf8
6 files changed
Lines changed: 401 additions & 221 deletions
File tree
- be
- src/format_v2
- lance
- table
- test/format_v2/table
- gensrc/thrift
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
47 | 55 | | |
48 | 56 | | |
49 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
49 | 247 | | |
50 | 248 | | |
51 | 249 | | |
| |||
0 commit comments