Skip to content

Commit 15b2d09

Browse files
airborne12claude
andcommitted
[fix](be) Score MATCH on keyword inverted indexes like SEARCH
### What problem does this PR solve? Issue Number: None Related PR: (master PR of the same title, pending) Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25: INDEX idx_redirect (redirect) USING INVERTED -- one row ('BP') SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0 SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821 Elasticsearch scores a match, term or match_phrase query on a keyword field with BM25 in query context: no norms and no frequencies, so freq = dl = avgdl = 1 and the score is the idf of the whole value (0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0. SEARCH already gives exactly these values; MATCH on keyword behaved like a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag clause were not there. Root cause on branch-4.1, three places keep keyword MATCH out of scoring: StringTypeInvertedIndexReader::query never sets is_similarity_score (and answers score() queries from the query cache), MatchPredicateCollector skips indexes without an analyzer, and is_need_similarity_score only accepts phrase-capable or lowercase-normalizer indexes. Fix: a keyword (non-analyzed) index is eligible for MATCH scoring regardless of support_phrase; the keyword reader requests scoring for the MATCH query types and bypasses the result cache like the full-text reader does; MatchPredicateCollector collects the whole value as one term (case and spaces kept), the term the reader looks up. The legacy scorer already uses norm 1 when a field has none. Equality runs as MATCH_ANY inside the reader but keeps EQUAL_QUERY for the decision, so it stays a pure filter. MATCH_REGEXP and MATCH_PHRASE_PREFIX keep their current scores. ### Release note score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized) inverted index now returns the BM25 score of the value, the same as SEARCH and Elasticsearch, instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new test_match_keyword_score (V2/V3, implicit and explicit parser=none); its output equals the master PR's V2/V3 blocks - Unit Test: IndexReaderHelperTest, CollectionStatisticsTest - Manual test: the reported table on a branch-4.1 ASAN build; MATCH equals SEARCH and Elasticsearch 9.5.5 to 7 digits, also for ARRAY<STRING>, a column with both a keyword and an analyzed index, and OR/AND mixes - Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score() instead of 0. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 5fc51d2 commit 15b2d09

7 files changed

Lines changed: 271 additions & 35 deletions

File tree

‎be/src/storage/index/index_reader_helper.h‎

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include <string>
2323

2424
#include "storage/index/index_iterator.h"
25+
#include "storage/index/inverted/analyzer/analyzer.h"
2526
#include "storage/index/inverted/inverted_index_reader.h"
2627

2728
namespace doris::segment_v2 {
@@ -106,33 +107,34 @@ class IndexReaderHelper {
106107

107108
static bool is_need_similarity_score(InvertedIndexQueryType query_type,
108109
const TabletIndex* index_meta) {
109-
if (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
110-
query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
111-
query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY ||
112-
query_type == InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY) {
113-
const auto& properties = index_meta->properties();
114-
if (get_parser_phrase_support_string_from_properties(properties) ==
115-
INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
116-
is_normalizer_only(properties)) {
117-
return true;
118-
}
119-
}
120-
return false;
110+
return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
111+
query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
112+
query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY ||
113+
query_type == InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY) &&
114+
can_score_match(index_meta);
121115
}
122116

123117
static bool is_need_similarity_score(TExprOpcode::type query_type,
124118
const TabletIndex* index_meta) {
125-
if (query_type == TExprOpcode::MATCH_ANY || query_type == TExprOpcode::MATCH_ALL ||
126-
query_type == TExprOpcode::MATCH_PHRASE ||
127-
query_type == TExprOpcode::MATCH_PHRASE_PREFIX) {
128-
const auto& properties = index_meta->properties();
129-
if (get_parser_phrase_support_string_from_properties(properties) ==
130-
INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
131-
is_normalizer_only(properties)) {
132-
return true;
133-
}
119+
return (query_type == TExprOpcode::MATCH_ANY || query_type == TExprOpcode::MATCH_ALL ||
120+
query_type == TExprOpcode::MATCH_PHRASE ||
121+
query_type == TExprOpcode::MATCH_PHRASE_PREFIX) &&
122+
can_score_match(index_meta);
123+
}
124+
125+
private:
126+
static bool can_score_match(const TabletIndex* index_meta) {
127+
const auto& properties = index_meta->properties();
128+
// A keyword index keeps each value as one term, without positions or norms, and its
129+
// postings score with unit frequency and length whatever support_phrase says, as an
130+
// Elasticsearch keyword field does: over tags (x, x, y, z), tag MATCH 'x' scores
131+
// idf(x) = ln(2) for both matching rows, the same as SEARCH('tag:x').
132+
if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)) {
133+
return true;
134134
}
135-
return false;
135+
return get_parser_phrase_support_string_from_properties(properties) ==
136+
INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
137+
is_normalizer_only(properties);
136138
}
137139
};
138140

‎be/src/storage/index/inverted/inverted_index_reader.cpp‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -530,15 +530,23 @@ Status StringTypeInvertedIndexReader::query(const IndexQueryContextPtr& context,
530530
search_str};
531531
auto* cache = InvertedIndexQueryCache::instance();
532532
InvertedIndexQueryCacheHandle cache_handler;
533-
if (handle_query_cache(context, cache, cache_key, &cache_handler, bit_map)) {
534-
return Status::OK();
533+
// A MATCH on a keyword index scores its whole value like SEARCH does, so with score() it
534+
// must run and publish its scores instead of answering from the cached bitmap. Equality
535+
// runs as MATCH_ANY below but keeps EQUAL_QUERY here, so it stays an unscored filter.
536+
const bool is_similarity_score =
537+
IndexReaderHelper::is_need_similarity_score(query_type, &_index_meta);
538+
if (!(context->collection_similarity && is_similarity_score)) {
539+
if (handle_query_cache(context, cache, cache_key, &cache_handler, bit_map)) {
540+
return Status::OK();
541+
}
535542
}
536543

537544
std::wstring column_name_ws = StringUtil::string_to_wstring(column_name);
538545

539546
InvertedIndexQueryInfo query_info;
540547
query_info.field_name = column_name_ws;
541548
query_info.term_infos.emplace_back(search_str, 0);
549+
query_info.is_similarity_score = is_similarity_score;
542550

543551
auto result = std::make_shared<roaring::Roaring>();
544552
FulltextIndexSearcherPtr* searcher_ptr = nullptr;

‎be/src/storage/predicate_collector.cpp‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -288,10 +288,6 @@ Status MatchPredicateCollector::collect(RuntimeState* state, const TabletSchemaS
288288
index_metas = {selected};
289289
}
290290
for (const auto* index_meta : index_metas) {
291-
if (!InvertedIndexAnalyzer::should_analyzer(index_meta->properties())) {
292-
continue;
293-
}
294-
295291
if (!IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
296292
continue;
297293
}
@@ -300,7 +296,12 @@ Status MatchPredicateCollector::collect(RuntimeState* state, const TabletSchemaS
300296
options.timezone = &state->timezone_obj();
301297
const std::string value = right_literal->value(options);
302298
std::vector<TermInfo> term_infos;
303-
if (match_analyzer_ctx != nullptr) {
299+
if (!InvertedIndexAnalyzer::should_analyzer(index_meta->properties())) {
300+
// A keyword index stores the whole value as one term, case and spaces included, and
301+
// its reader looks it up unanalyzed: tag MATCH 'Hello World' scores the single term
302+
// "Hello World", as SEARCH('tag:"Hello World"') does.
303+
term_infos.emplace_back(value);
304+
} else if (match_analyzer_ctx != nullptr) {
304305
term_infos = DORIS_TRY(analyze_plain_query(value, *match_analyzer_ctx));
305306
} else {
306307
term_infos = DORIS_TRY(analyze_plain_query(value, index_meta->properties()));

‎be/test/storage/compaction/collection_statistics_test.cpp‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1333,10 +1333,9 @@ TEST_F(CollectionStatisticsTest, CollectNotExtractedColumnSkipsFallback) {
13331333
EXPECT_TRUE(collect_infos.empty());
13341334
}
13351335

1336-
// L1: Index whose properties do not request an analyzer
1337-
// (should_analyzer returns false). The matching index_meta is iterated
1338-
// but skipped before insertion.
1339-
TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
1336+
// L1: Index whose properties do not request an analyzer (should_analyzer returns false). A keyword
1337+
// MATCH scores like SEARCH on the same field, so its whole value is collected as one term.
1338+
TEST_F(CollectionStatisticsTest, CollectsKeywordIndexValueAsOneTerm) {
13401339
auto tablet_schema = std::make_shared<TabletSchema>();
13411340

13421341
constexpr int32_t kColUid = 1300;
@@ -1361,7 +1360,7 @@ TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
13611360

13621361
auto match_expr = std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
13631362
auto slot_ref = std::make_shared<collection_statistics::MockVSlotRef>("kw", SlotId(kSlotId));
1364-
auto literal = std::make_shared<collection_statistics::MockVLiteral>("v");
1363+
auto literal = std::make_shared<collection_statistics::MockVLiteral>("Hello World");
13651364
match_expr->_children.push_back(slot_ref);
13661365
match_expr->_children.push_back(literal);
13671366

@@ -1370,7 +1369,12 @@ TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
13701369
auto status =
13711370
collector.collect(runtime_state_.get(), tablet_schema, match_expr, &collect_infos);
13721371
ASSERT_TRUE(status.ok()) << status.msg();
1373-
EXPECT_TRUE(collect_infos.empty());
1372+
ASSERT_EQ(collect_infos.size(), 1U);
1373+
const auto& collect_info = collect_infos.begin()->second;
1374+
ASSERT_NE(collect_info.index_meta, nullptr);
1375+
EXPECT_EQ(collect_info.index_meta->index_id(), 2300);
1376+
ASSERT_EQ(collect_info.term_infos.size(), 1U);
1377+
EXPECT_EQ(collect_info.term_infos.begin()->get_single_term(), "Hello World");
13741378
}
13751379

13761380
// L2: Index whose analyzer is set (should_analyzer returns true) but does

‎be/test/storage/segment/index_reader_helper_test.cpp‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -439,6 +439,33 @@ TEST_F(IndexReaderHelperTest, NormalizerOnlyPhraseAndScoringNeedNoPositions) {
439439
InvertedIndexQueryType::MATCH_ANY_QUERY, &custom_meta));
440440
}
441441

442+
TEST_F(IndexReaderHelperTest, MatchScoringAcceptsKeywordIndexes) {
443+
// Keyword indexes score with unit frequency and length, so phrase support does not matter.
444+
for (const auto& properties : std::vector<std::map<std::string, std::string>> {
445+
{}, {{"parser", "none"}}, {{"parser", "none"}, {"support_phrase", "true"}}}) {
446+
SCOPED_TRACE(::testing::PrintToString(properties));
447+
TabletIndexPB pb;
448+
pb.set_index_id(1);
449+
pb.set_index_name("keyword_index");
450+
pb.set_index_type(IndexType::INVERTED);
451+
pb.add_col_unique_id(1);
452+
for (const auto& [key, value] : properties) {
453+
(*pb.mutable_properties())[key] = value;
454+
}
455+
TabletIndex index;
456+
index.init_from_pb(pb);
457+
for (auto op : {TExprOpcode::MATCH_ANY, TExprOpcode::MATCH_ALL, TExprOpcode::MATCH_PHRASE,
458+
TExprOpcode::MATCH_PHRASE_PREFIX}) {
459+
EXPECT_TRUE(IndexReaderHelper::is_need_similarity_score(op, &index));
460+
}
461+
EXPECT_TRUE(IndexReaderHelper::is_need_similarity_score(
462+
InvertedIndexQueryType::MATCH_ANY_QUERY, &index));
463+
EXPECT_FALSE(IndexReaderHelper::is_need_similarity_score(TExprOpcode::EQ, &index));
464+
EXPECT_FALSE(IndexReaderHelper::is_need_similarity_score(
465+
InvertedIndexQueryType::EQUAL_QUERY, &index));
466+
}
467+
}
468+
442469
TEST_F(IndexReaderHelperTest, IsNeedSimilarityScoreWithTExprOpcodeTest) {
443470
TabletIndex index_meta;
444471
auto index_meta_pb = std::make_unique<TabletIndexPB>();
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !V2_single_row_match --
3+
1 0.287682
4+
5+
-- !V2_match --
6+
1 0.693147
7+
2 0.693147
8+
5 0.693147
9+
10+
-- !V2_match_all --
11+
1 0.693147
12+
2 0.693147
13+
5 0.693147
14+
15+
-- !V2_match_phrase --
16+
1 0.693147
17+
2 0.693147
18+
5 0.693147
19+
20+
-- !V2_search --
21+
1 0.693147
22+
2 0.693147
23+
5 0.693147
24+
25+
-- !V2_none_match --
26+
1 0.693147
27+
3 0.693147
28+
6 0.693147
29+
30+
-- !V2_none_match_phrase --
31+
1 0.693147
32+
3 0.693147
33+
6 0.693147
34+
35+
-- !V2_none_search --
36+
1 0.693147
37+
3 0.693147
38+
6 0.693147
39+
40+
-- !V2_analyzed_and_keyword --
41+
1 0.983863
42+
2 0.893357
43+
5 0.961783
44+
45+
-- !V2_search_and --
46+
1 0.983863
47+
2 0.893357
48+
5 0.961783
49+
50+
-- !V2_analyzed_and_equality --
51+
1 0.290716
52+
2 0.20021
53+
5 0.268636
54+
55+
-- !V3_single_row_match --
56+
1 0.287682
57+
58+
-- !V3_match --
59+
1 0.693147
60+
2 0.693147
61+
5 0.693147
62+
63+
-- !V3_match_all --
64+
1 0.693147
65+
2 0.693147
66+
5 0.693147
67+
68+
-- !V3_match_phrase --
69+
1 0.693147
70+
2 0.693147
71+
5 0.693147
72+
73+
-- !V3_search --
74+
1 0.693147
75+
2 0.693147
76+
5 0.693147
77+
78+
-- !V3_none_match --
79+
1 0.693147
80+
3 0.693147
81+
6 0.693147
82+
83+
-- !V3_none_match_phrase --
84+
1 0.693147
85+
3 0.693147
86+
6 0.693147
87+
88+
-- !V3_none_search --
89+
1 0.693147
90+
3 0.693147
91+
6 0.693147
92+
93+
-- !V3_analyzed_and_keyword --
94+
1 0.983863
95+
2 0.893357
96+
5 0.961783
97+
98+
-- !V3_search_and --
99+
1 0.983863
100+
2 0.893357
101+
5 0.961783
102+
103+
-- !V3_analyzed_and_equality --
104+
1 0.290716
105+
2 0.20021
106+
5 0.268636
107+
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// score() over MATCH on a keyword (untokenized) inverted index. Like an Elasticsearch match or term
19+
// query on a keyword field, it scores BM25 with unit frequency and length, so every matching row
20+
// scores the idf of the whole value, the same as SEARCH on that field; it used to score 0. An
21+
// equality predicate stays a pure filter.
22+
suite("test_match_keyword_score", "p0") {
23+
// MATCH and SEARCH need index execution even when CI fuzzes these session defaults.
24+
sql "set enable_common_expr_pushdown = true"
25+
sql "set enable_inverted_index_query = true"
26+
27+
for (def storageFormat : ["V2", "V3"]) {
28+
sql "DROP TABLE IF EXISTS test_match_keyword_score"
29+
sql """
30+
CREATE TABLE test_match_keyword_score (
31+
id INT NOT NULL,
32+
body STRING NULL,
33+
tag STRING NULL,
34+
tag_none STRING NULL,
35+
INDEX idx_body(body) USING INVERTED
36+
PROPERTIES("parser" = "english", "support_phrase" = "true"),
37+
INDEX idx_tag(tag) USING INVERTED PROPERTIES("ignore_above" = "1024"),
38+
INDEX idx_tag_none(tag_none) USING INVERTED
39+
PROPERTIES("parser" = "none", "support_phrase" = "true")
40+
) ENGINE=OLAP
41+
DUPLICATE KEY(id)
42+
DISTRIBUTED BY HASH(id) BUCKETS 1
43+
PROPERTIES (
44+
"replication_allocation" = "tag.location.default:1",
45+
"inverted_index_storage_format" = "${storageFormat}"
46+
)
47+
"""
48+
// The first row alone is the reported case: one document scores idf = ln(4/3) = 0.287682.
49+
sql """INSERT INTO test_match_keyword_score VALUES (1, 'apple apple', 'BP', 'Hello World')"""
50+
sql "sync"
51+
quickTest("${storageFormat}_single_row_match", """
52+
SELECT id, ROUND(s, 6) FROM (
53+
SELECT id, score() AS s FROM test_match_keyword_score
54+
WHERE tag MATCH 'BP' ORDER BY s DESC LIMIT 10
55+
) hits ORDER BY id
56+
""", true)
57+
58+
sql """INSERT INTO test_match_keyword_score VALUES
59+
(2, 'apple pear', 'BP', 'hello world'), (3, 'pear', 'other', 'Hello World'),
60+
(4, 'apple', 'x', 'Hello'), (5, 'apple', 'BP', NULL), (6, 'apple', NULL, 'Hello World')
61+
"""
62+
sql "sync"
63+
64+
// The whole value is one term: MATCH, MATCH_ALL and MATCH_PHRASE score as SEARCH does.
65+
for (def query : [
66+
["match", "tag MATCH 'BP'"],
67+
["match_all", "tag MATCH_ALL 'BP'"],
68+
["match_phrase", "tag MATCH_PHRASE 'BP'"],
69+
["search", "SEARCH('tag:BP')"],
70+
// Case and spaces are kept, and phrase support inherited by parser=none is ignored.
71+
["none_match", "tag_none MATCH 'Hello World'"],
72+
["none_match_phrase", "tag_none MATCH_PHRASE 'Hello World'"],
73+
["none_search", "SEARCH('tag_none:\"Hello World\"')"],
74+
// Both leaves score: idf(apple) from body plus idf(BP) from tag.
75+
["analyzed_and_keyword", "body MATCH 'apple' AND tag MATCH 'BP'"],
76+
["search_and", "SEARCH('body:apple AND tag:BP')"],
77+
// An equality is a filter: only the body leaf scores.
78+
["analyzed_and_equality", "body MATCH 'apple' AND tag = 'BP'"]]) {
79+
quickTest("${storageFormat}_${query[0]}", """
80+
SELECT id, ROUND(s, 6) FROM (
81+
SELECT id, score() AS s FROM test_match_keyword_score
82+
WHERE ${query[1]} ORDER BY s DESC LIMIT 10
83+
) hits ORDER BY id
84+
""", true)
85+
}
86+
}
87+
}

0 commit comments

Comments
 (0)