Skip to content

Commit 79bc425

Browse files
airborne12claude
andcommitted
[fix](be) Skip BKD indexes when collecting BM25 statistics
### What problem does this PR solve? Issue Number: None Related PR: #69790 Problem Summary: keyword MATCH scoring made statistics collection reach indexes that are BKD, and on branch-4.1 that aborts the BE, because IndexReader::open throws on a scanner thread with nothing to catch it: -- n INT with an inverted index SELECT score() FROM t WHERE body MATCH 'apple' AND CAST(n AS STRING) MATCH '1' ORDER BY score() DESC LIMIT 10; -- terminate called after throwing an instance of 'CLuceneError' -- what(): No segments* file found in DorisCompoundReader@ A VARIANT path reaches it too: it is typed per segment, so a numeric path is BKD where it holds numbers while the schema calls it text. And when a numeric predicate has cached the BKD searcher, the cache hit throws std::bad_variant_access from std::get<FulltextIndexSearcherPtr> instead: SELECT COUNT(*) FROM t WHERE CAST(v['n'] AS BIGINT) = 1; SELECT score() FROM t WHERE CAST(v['n'] AS STRING) MATCH '1' ORDER BY score() DESC LIMIT 10; Fix: a BKD index has no terms, so it adds no statistics, as a segment without the path already does; the cache miss detects it by the BKD data file in the compound directory and the cache hit by the searcher type. Its rows match without the index and score 0, and text paths of the same column keep their BM25 score. The other follow-ups of #69790 need nothing here: branch-4.1 has no SNII, so a VARIANT path absent from a segment was already skipped, and its keyword reader takes the whole MATCH_PHRASE value as the term, so the collector already agrees with it on a trailing " ~N". ### Release note None ### Check List (For Author) - Test: Regression test / Manual test - Regression test: test_match_keyword_score adds the numeric cast, a VARIANT numeric, text and sparse path, and the score query after a numeric predicate, on V2/V3. All 32 blocks equal the master PR's; with the old std::get the suite fails with the same abort; 15 score-related suites in inverted_index_p0 and search pass - Unit Test: 65 tests in IndexReaderHelperTest, CollectionStatistics* - Manual test: keyword and analyzed VARIANT indexes on a branch-4.1 ASAN build; the numeric cast aborted the BE before this change - Behavior changed: Yes. These queries return rows instead of aborting the BE. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 15b2d09 commit 79bc425

3 files changed

Lines changed: 119 additions & 7 deletions

File tree

‎be/src/storage/compaction/collection_statistics.cpp‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,15 @@ Result<std::shared_ptr<lucene::search::IndexSearcher>> open_optional_clucene_sta
5151
}
5252
return ResultError(status);
5353
}
54+
// Only text indexes have a segments file: opening a BKD index throws, and nothing above this
55+
// catches it on a scanner thread. A MATCH can reach one: CAST(n AS STRING) MATCH '1' on an INT
56+
// column, or on a VARIANT path v.n, which is typed per segment and indexed as BKD where it
57+
// holds numbers. Such an index has no terms and adds no statistics, as a segment without the
58+
// path does; its rows match without the index and score nothing.
59+
if (compound.value()->fileExists(
60+
InvertedIndexDescriptor::get_temporary_bkd_index_data_file_name())) {
61+
return std::shared_ptr<lucene::search::IndexSearcher> {};
62+
}
5463
auto* reader = lucene::index::IndexReader::open(compound.value().get());
5564
return std::make_shared<lucene::search::IndexSearcher>(reader, true);
5665
}
@@ -225,8 +234,13 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset, int3
225234
}
226235

227236
auto searcher_variant = inverted_index_cache_handle.get_index_searcher();
228-
auto index_searcher_ptr = std::get<FulltextIndexSearcherPtr>(searcher_variant);
229-
index_searcher = index_searcher_ptr.get();
237+
// CAST(v['n'] AS BIGINT) = 1 caches the BKD searcher of v.n under this key, so a BKD index
238+
// can also arrive from the cache: no terms, as in open_optional_clucene_stats_searcher.
239+
const auto* fulltext_searcher = std::get_if<FulltextIndexSearcherPtr>(&searcher_variant);
240+
if (fulltext_searcher == nullptr) {
241+
continue;
242+
}
243+
index_searcher = fulltext_searcher->get();
230244
index_reader = index_searcher->getReader();
231245
#endif
232246
total_seg_num_docs = std::max(total_seg_num_docs, index_reader->maxDoc());

‎regression-test/data/inverted_index_p0/test_match_keyword_score.out‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,27 @@
5252
2 0.20021
5353
5 0.268636
5454

55+
-- !V2_analyzed_and_numeric_cast --
56+
1 0.290716
57+
4 0.268636
58+
59+
-- !V2_variant_numeric_path --
60+
1 0
61+
3 0
62+
4 0.287682
63+
64+
-- !V2_variant_text_path --
65+
1 0.693147
66+
4 0.693147
67+
68+
-- !V2_variant_sparse_path --
69+
1 0.980829
70+
71+
-- !V2_variant_numeric_path_cached_bkd --
72+
1 0
73+
3 0
74+
4 0.287682
75+
5576
-- !V3_single_row_match --
5677
1 0.287682
5778

@@ -105,3 +126,24 @@
105126
2 0.20021
106127
5 0.268636
107128

129+
-- !V3_analyzed_and_numeric_cast --
130+
1 0.290716
131+
4 0.268636
132+
133+
-- !V3_variant_numeric_path --
134+
1 0
135+
3 0
136+
4 0.287682
137+
138+
-- !V3_variant_text_path --
139+
1 0.693147
140+
4 0.693147
141+
142+
-- !V3_variant_sparse_path --
143+
1 0.980829
144+
145+
-- !V3_variant_numeric_path_cached_bkd --
146+
1 0
147+
3 0
148+
4 0.287682
149+

‎regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy‎

Lines changed: 61 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ suite("test_match_keyword_score", "p0") {
2323
// MATCH and SEARCH need index execution even when CI fuzzes these session defaults.
2424
sql "set enable_common_expr_pushdown = true"
2525
sql "set enable_inverted_index_query = true"
26+
sql "set enable_match_without_inverted_index = true"
2627

2728
for (def storageFormat : ["V2", "V3"]) {
2829
sql "DROP TABLE IF EXISTS test_match_keyword_score"
@@ -32,11 +33,13 @@ suite("test_match_keyword_score", "p0") {
3233
body STRING NULL,
3334
tag STRING NULL,
3435
tag_none STRING NULL,
36+
n INT NULL,
3537
INDEX idx_body(body) USING INVERTED
3638
PROPERTIES("parser" = "english", "support_phrase" = "true"),
3739
INDEX idx_tag(tag) USING INVERTED PROPERTIES("ignore_above" = "1024"),
3840
INDEX idx_tag_none(tag_none) USING INVERTED
39-
PROPERTIES("parser" = "none", "support_phrase" = "true")
41+
PROPERTIES("parser" = "none", "support_phrase" = "true"),
42+
INDEX idx_n(n) USING INVERTED
4043
) ENGINE=OLAP
4144
DUPLICATE KEY(id)
4245
DISTRIBUTED BY HASH(id) BUCKETS 1
@@ -46,7 +49,7 @@ suite("test_match_keyword_score", "p0") {
4649
)
4750
"""
4851
// The first row alone is the reported case: one document scores idf = ln(4/3) = 0.287682.
49-
sql """INSERT INTO test_match_keyword_score VALUES (1, 'apple apple', 'BP', 'Hello World')"""
52+
sql """INSERT INTO test_match_keyword_score VALUES (1, 'apple apple', 'BP', 'Hello World', 1)"""
5053
sql "sync"
5154
quickTest("${storageFormat}_single_row_match", """
5255
SELECT id, ROUND(s, 6) FROM (
@@ -56,8 +59,9 @@ suite("test_match_keyword_score", "p0") {
5659
""", true)
5760

5861
sql """INSERT INTO test_match_keyword_score VALUES
59-
(2, 'apple pear', 'BP', 'hello world'), (3, 'pear', 'other', 'Hello World'),
60-
(4, 'apple', 'x', 'Hello'), (5, 'apple', 'BP', NULL), (6, 'apple', NULL, 'Hello World')
62+
(2, 'apple pear', 'BP', 'hello world', 2), (3, 'pear', 'other', 'Hello World', 1),
63+
(4, 'apple', 'x', 'Hello', 1), (5, 'apple', 'BP', NULL, NULL),
64+
(6, 'apple', NULL, 'Hello World', 2)
6165
"""
6266
sql "sync"
6367

@@ -75,13 +79,65 @@ suite("test_match_keyword_score", "p0") {
7579
["analyzed_and_keyword", "body MATCH 'apple' AND tag MATCH 'BP'"],
7680
["search_and", "SEARCH('body:apple AND tag:BP')"],
7781
// An equality is a filter: only the body leaf scores.
78-
["analyzed_and_equality", "body MATCH 'apple' AND tag = 'BP'"]]) {
82+
["analyzed_and_equality", "body MATCH 'apple' AND tag = 'BP'"],
83+
// A MATCH on a number cast to text is a filter over the BKD index of n: only the
84+
// body leaf scores. Opening that BKD index for statistics used to abort the BE.
85+
["analyzed_and_numeric_cast", "body MATCH 'apple' AND CAST(n AS STRING) MATCH '1'"]]) {
7986
quickTest("${storageFormat}_${query[0]}", """
8087
SELECT id, ROUND(s, 6) FROM (
8188
SELECT id, score() AS s FROM test_match_keyword_score
8289
WHERE ${query[1]} ORDER BY s DESC LIMIT 10
8390
) hits ORDER BY id
8491
""", true)
8592
}
93+
94+
// A VARIANT path is typed per segment: n holds numbers, indexed as BKD, in the first
95+
// insert and text in the second. The plain VARIANT index makes every path a keyword
96+
// index, so CAST(v['n'] AS STRING) MATCH collects statistics for n; the BKD segment adds
97+
// none instead of aborting the BE, and its rows match without the index with score 0. p
98+
// exists only in the first insert: the other segment has no index for it and adds nothing.
99+
sql "set default_variant_enable_typed_paths_to_sparse = false"
100+
sql "set default_variant_enable_doc_mode = false"
101+
sql "DROP TABLE IF EXISTS test_match_keyword_score_variant"
102+
sql """
103+
CREATE TABLE test_match_keyword_score_variant (
104+
id INT NOT NULL,
105+
v VARIANT NULL,
106+
INDEX idx_v(v) USING INVERTED
107+
) ENGINE=OLAP
108+
DUPLICATE KEY(id)
109+
DISTRIBUTED BY HASH(id) BUCKETS 1
110+
PROPERTIES (
111+
"replication_allocation" = "tag.location.default:1",
112+
"inverted_index_storage_format" = "${storageFormat}",
113+
"disable_auto_compaction" = "true"
114+
)
115+
"""
116+
sql """INSERT INTO test_match_keyword_score_variant VALUES
117+
(1, '{"n": 1, "s": "abc", "p": "abc"}'), (2, '{"n": 2, "s": "abd", "p": "abd"}'),
118+
(3, '{"n": 1, "s": "xyz", "p": "xyz"}')
119+
"""
120+
sql """INSERT INTO test_match_keyword_score_variant VALUES (4, '{"n": "1", "s": "abc"}')"""
121+
sql "sync"
122+
for (def query : [
123+
["variant_numeric_path", "CAST(v['n'] AS STRING) MATCH '1'"],
124+
["variant_text_path", "CAST(v['s'] AS STRING) MATCH 'abc'"],
125+
["variant_sparse_path", "CAST(v['p'] AS STRING) MATCH 'abc'"]]) {
126+
quickTest("${storageFormat}_${query[0]}", """
127+
SELECT id, ROUND(s, 6) FROM (
128+
SELECT id, score() AS s FROM test_match_keyword_score_variant
129+
WHERE ${query[1]} ORDER BY s DESC LIMIT 10
130+
) hits ORDER BY id
131+
""", true)
132+
}
133+
// A numeric predicate leaves the BKD searcher of n in the searcher cache, where collecting
134+
// statistics finds it next: it used to take it for a text searcher and abort the BE.
135+
sql "SELECT COUNT(*) FROM test_match_keyword_score_variant WHERE CAST(v['n'] AS BIGINT) = 1"
136+
quickTest("${storageFormat}_variant_numeric_path_cached_bkd", """
137+
SELECT id, ROUND(s, 6) FROM (
138+
SELECT id, score() AS s FROM test_match_keyword_score_variant
139+
WHERE CAST(v['n'] AS STRING) MATCH '1' ORDER BY s DESC LIMIT 10
140+
) hits ORDER BY id
141+
""", true)
86142
}
87143
}

0 commit comments

Comments
 (0)