Repository navigation
Commit 79bc425
[fix](be) Skip BKD indexes when collecting BM25 statistics
### What problem does this PR solve?
Issue Number: None
Related PR: #69790
Problem Summary: keyword MATCH scoring made statistics collection reach
indexes that are BKD, and on branch-4.1 that aborts the BE, because
IndexReader::open throws on a scanner thread with nothing to catch it:
-- n INT with an inverted index
SELECT score() FROM t WHERE body MATCH 'apple'
AND CAST(n AS STRING) MATCH '1' ORDER BY score() DESC LIMIT 10;
-- terminate called after throwing an instance of 'CLuceneError'
-- what(): No segments* file found in DorisCompoundReader@
A VARIANT path reaches it too: it is typed per segment, so a numeric path
is BKD where it holds numbers while the schema calls it text. And when a
numeric predicate has cached the BKD searcher, the cache hit throws
std::bad_variant_access from std::get<FulltextIndexSearcherPtr> instead:
SELECT COUNT(*) FROM t WHERE CAST(v['n'] AS BIGINT) = 1;
SELECT score() FROM t WHERE CAST(v['n'] AS STRING) MATCH '1'
ORDER BY score() DESC LIMIT 10;
Fix: a BKD index has no terms, so it adds no statistics, as a segment
without the path already does; the cache miss detects it by the BKD data
file in the compound directory and the cache hit by the searcher type.
Its rows match without the index and score 0, and text paths of the same
column keep their BM25 score.
The other follow-ups of #69790 need nothing here: branch-4.1 has no SNII,
so a VARIANT path absent from a segment was already skipped, and its
keyword reader takes the whole MATCH_PHRASE value as the term, so the
collector already agrees with it on a trailing " ~N".
### Release note
None
### Check List (For Author)
- Test: Regression test / Manual test
- Regression test: test_match_keyword_score adds the numeric cast, a
VARIANT numeric, text and sparse path, and the score query after a
numeric predicate, on V2/V3. All 32 blocks equal the master PR's;
with the old std::get the suite fails with the same abort;
15 score-related suites in inverted_index_p0 and search pass
- Unit Test: 65 tests in IndexReaderHelperTest, CollectionStatistics*
- Manual test: keyword and analyzed VARIANT indexes on a branch-4.1
ASAN build; the numeric cast aborted the BE before this change
- Behavior changed: Yes. These queries return rows instead of aborting
the BE.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent 15b2d09 commit 79bc425
3 files changed
Lines changed: 119 additions & 7 deletions
File tree
- be/src/storage/compaction
- regression-test
- data/inverted_index_p0
- suites/inverted_index_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
54 | 63 | | |
55 | 64 | | |
56 | 65 | | |
| |||
225 | 234 | | |
226 | 235 | | |
227 | 236 | | |
228 | | - | |
229 | | - | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
230 | 244 | | |
231 | 245 | | |
232 | 246 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
55 | 76 | | |
56 | 77 | | |
57 | 78 | | |
| |||
105 | 126 | | |
106 | 127 | | |
107 | 128 | | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
38 | 40 | | |
39 | | - | |
| 41 | + | |
| 42 | + | |
40 | 43 | | |
41 | 44 | | |
42 | 45 | | |
| |||
46 | 49 | | |
47 | 50 | | |
48 | 51 | | |
49 | | - | |
| 52 | + | |
50 | 53 | | |
51 | 54 | | |
52 | 55 | | |
| |||
56 | 59 | | |
57 | 60 | | |
58 | 61 | | |
59 | | - | |
60 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
61 | 65 | | |
62 | 66 | | |
63 | 67 | | |
| |||
75 | 79 | | |
76 | 80 | | |
77 | 81 | | |
78 | | - | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
79 | 86 | | |
80 | 87 | | |
81 | 88 | | |
82 | 89 | | |
83 | 90 | | |
84 | 91 | | |
85 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
86 | 142 | | |
87 | 143 | | |
0 commit comments