Repository navigation
Commit 27cdbb5
[fix](be) Skip BKD segments when collecting BM25 statistics
### What problem does this PR solve?
Issue Number: None
Related PR: #69790
Problem Summary: a VARIANT path is typed per segment, so a numeric path
is indexed as BKD in the segments that hold numbers while the tablet
schema only knows it as text. score() over a MATCH on that path collects
statistics for it and opens the BKD index as a text index:
CREATE TABLE t (id INT, v VARIANT, INDEX idx_v(v) USING INVERTED) ...;
INSERT INTO t VALUES (1, parse_to_variant('{"n": 1, "s": "abc"}')), ...;
SELECT score() FROM t WHERE CAST(v['n'] AS STRING) MATCH '1'
ORDER BY score() DESC LIMIT 10;
-- V2/V3: CLuceneError occurred opening the index for BM25 statistics:
-- No segments* file found
-- SNII: segment: logical index is not a text inverted index
With an analyzed VARIANT index (parser=english) this failed before as
well; keyword MATCH scoring made the plain VARIANT index, the common
case, reach it too. Without score() the query works.
Fix: a segment that indexed the field with BKD holds no terms, so it adds
no statistics, as a segment without the path already does. V2/V3 detect
it by the BKD data file in the compound directory before opening a
CLucene reader; SNII by the logical index kind in its directory. The rows
of such a segment still match without the index and score 0; the text
path of the same column keeps its BM25 score.
### Release note
None
### Check List (For Author)
- Test: Regression test / Manual test
- Regression test: test_match_keyword_score adds a VARIANT case with
a numeric path (BKD in one segment, text in another) and a text
path, on V2/V3/SNII; before this change it failed on all three.
26 score-related suites in inverted_index_p0, search and variant_p0
pass.
- Manual test: keyword and analyzed VARIANT indexes on V2/V3/SNII.
- Unit Test: CollectionStatistics*, IndexReaderHelperTest. No new
unit test: the skip needs a BKD index on disk, which the regression
case builds on every format.
- Behavior changed: Yes. The query above returns its rows (score 0)
instead of failing.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent 7881be5 commit 27cdbb5
3 files changed
Lines changed: 92 additions & 0 deletions
File tree
- be/src/storage/index/inverted/reader
- regression-test
- data/inverted_index_p0
- suites/inverted_index_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
76 | 85 | | |
77 | 86 | | |
78 | 87 | | |
| |||
89 | 98 | | |
90 | 99 | | |
91 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
92 | 112 | | |
93 | 113 | | |
94 | 114 | | |
| |||
547 | 567 | | |
548 | 568 | | |
549 | 569 | | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
550 | 575 | | |
551 | 576 | | |
552 | 577 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
69 | 78 | | |
70 | 79 | | |
71 | 80 | | |
| |||
133 | 142 | | |
134 | 143 | | |
135 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
136 | 154 | | |
137 | 155 | | |
138 | 156 | | |
| |||
200 | 218 | | |
201 | 219 | | |
202 | 220 | | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
96 | 136 | | |
97 | 137 | | |
0 commit comments