Skip to content

Commit ae2b737

Browse files
airborne12claude
andcommitted
[fix](be) Score MATCH on keyword inverted indexes like SEARCH
### What problem does this PR solve? Issue Number: None Related PR: #69051 Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25: INDEX idx_redirect (redirect) USING INVERTED -- one row ('BP') SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0 SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821 Elasticsearch scores a match, term or match_phrase query on a keyword field with BM25 in query context: no norms and no frequencies, so freq = dl = avgdl = 1 and the score is the idf of the whole value (0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0. SEARCH already gives exactly these values. MATCH on keyword behaved like a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag clause were not there. Root cause: #69051 made the engine score keyword postings (unit frequency and norm) but kept MATCH out on purpose: can_score_match required an analyzed index, and MatchPredicateCollector returned early for indexes without an analyzer, so no statistics were collected for them. Fix: a keyword (non-analyzed) index is eligible for MATCH scoring regardless of support_phrase, and MatchPredicateCollector collects its whole value as one term (case and spaces kept), the same term the reader looks up and the same as SearchPredicateCollector collects. Analyzed indexes keep the existing support_phrase / lowercase-normalizer rule. Equality and IN stay pure filters. A keyword MATCH_PHRASE_PREFIX scores the constant 1 of a single-term prefix; MATCH_REGEXP is unchanged. ### Release note score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized) inverted index now returns the BM25 score of the value, the same as SEARCH and Elasticsearch, instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new test_match_keyword_score (V2/V3/SNII, implicit and explicit parser=none); test_search_score_keyword_filters .out regenerated: only the keyword MATCH blocks change, now adding idf(x) and equal to the SEARCH AND blocks; test_omit_norms .out regenerated: its queries are MATCH_PHRASE_PREFIX on keyword indexes, now 1.0 instead of 0.0. 25 score-related suites in inverted_index_p0, search and variant_p0 pass - Unit Test: IndexReaderHelperTest, CollectionStatisticsTest - Manual test: the reported table on an ASAN build; MATCH equals SEARCH and Elasticsearch 9.5.5 to 7 digits, also for ARRAY<STRING>, a column with both a keyword and an analyzed index, and OR/AND mixes - Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score() instead of 0. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 62e3d81 commit ae2b737

9 files changed

Lines changed: 373 additions & 91 deletions

File tree

‎be/src/storage/index/index_reader_helper.cpp‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,16 @@ namespace {
2424

2525
bool can_score_match(const TabletIndex* index_meta) {
2626
const auto& properties = index_meta->properties();
27-
// parser=none can inherit support_phrase=true from FE defaults. MATCH statistics skip
28-
// these keyword indexes, so their filters must not request a scoring context either.
29-
return inverted_index::InvertedIndexAnalyzer::should_analyzer(properties) &&
30-
(get_parser_phrase_support_string_from_properties(properties) ==
31-
INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
32-
IndexReaderHelper::is_normalizer_only(properties));
27+
// A keyword index keeps each value as one term, without positions or norms, and its postings
28+
// score with unit frequency and length whatever support_phrase says, as an Elasticsearch
29+
// keyword field does: over tags (x, x, y, z), tag MATCH 'x' scores idf(x) = ln(2) for both
30+
// matching rows, the same as SEARCH('tag:x'). MATCH statistics collect these terms too.
31+
if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)) {
32+
return true;
33+
}
34+
return get_parser_phrase_support_string_from_properties(properties) ==
35+
INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
36+
IndexReaderHelper::is_normalizer_only(properties);
3337
}
3438

3539
} // namespace

‎be/src/storage/index/inverted/reader/predicate_collector.cpp‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,24 @@ void preserve_selected_index_metadata(const ScoringIndexCandidates& candidates,
285285

286286
using namespace scoring_collection_detail;
287287

288+
namespace {
289+
290+
// The terms a MATCH scores, the ones its reader looks up. A keyword index stores the whole value
291+
// as one term, case and spaces included: tag MATCH 'Hello World' scores the single term
292+
// "Hello World", as SEARCH('tag:"Hello World"') does. Any other index cuts the value with the
293+
// query's analyzer.
294+
Result<std::vector<TermInfo>> match_scoring_terms(const std::string& value,
295+
const TabletIndex& index_meta,
296+
const InvertedIndexAnalyzerCtx& analyzer_ctx) {
297+
if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta.properties())) {
298+
return std::vector<TermInfo> {TermInfo(value)};
299+
}
300+
DORIS_CHECK(analyzer_ctx.analyzer_provider != nullptr);
301+
return analyze_plain_query(value, analyzer_ctx);
302+
}
303+
304+
} // namespace
305+
288306
VSlotRef* PredicateCollector::find_slot_ref(const VExprSPtr& expr) const {
289307
if (!expr) {
290308
return nullptr;
@@ -365,15 +383,14 @@ Status MatchPredicateCollector::collect(RuntimeState* state, const TabletSchemaS
365383
const auto* index_meta = DORIS_TRY(
366384
select_index_meta(candidates.index_metas, candidates.field_type, query_type,
367385
analyzer_ctx->analyzer_key, analyzer_ctx->legacy_analyzer_key));
368-
if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta->properties()) ||
369-
!IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
386+
if (!IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
370387
return Status::OK();
371388
}
372389

373-
DORIS_CHECK(analyzer_ctx->analyzer_provider != nullptr);
374390
auto options = DataTypeSerDe::get_default_format_options();
375391
options.timezone = &state->timezone_obj();
376-
auto term_infos = DORIS_TRY(analyze_plain_query(right_literal->value(options), *analyzer_ctx));
392+
auto term_infos = DORIS_TRY(
393+
match_scoring_terms(right_literal->value(options), *index_meta, *analyzer_ctx));
377394
// The prefix of a phrase prefix is scored by the terms it expands to, so the statistics are
378395
// collected for them per segment; a prefix alone scores a constant and needs none.
379396
std::optional<TermInfo> scoring_prefix;

‎be/test/storage/index/inverted/reader/collection_statistics_test.cpp‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2426,10 +2426,9 @@ TEST_F(CollectionStatisticsTest, CollectNotExtractedColumnSkipsFallback) {
24262426
EXPECT_TRUE(collect_infos.empty());
24272427
}
24282428

2429-
// L1: Index whose properties do not request an analyzer
2430-
// (should_analyzer returns false). The matching index_meta is iterated
2431-
// but skipped before insertion.
2432-
TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
2429+
// L1: Index whose properties do not request an analyzer (should_analyzer returns false). A keyword
2430+
// MATCH scores like SEARCH on the same field, so its whole value is collected as one term.
2431+
TEST_F(CollectionStatisticsTest, CollectsKeywordIndexValueAsOneTerm) {
24332432
auto tablet_schema = std::make_shared<TabletSchema>();
24342433

24352434
constexpr int32_t kColUid = 1300;
@@ -2456,20 +2455,30 @@ TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
24562455

24572456
auto match_expr = std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
24582457
auto slot_ref = std::make_shared<collection_statistics::MockVSlotRef>("kw", SlotId(kSlotId));
2459-
auto literal = std::make_shared<collection_statistics::MockVLiteral>("v");
2458+
auto literal = std::make_shared<collection_statistics::MockVLiteral>("Hello World");
24602459
match_expr->_children.push_back(slot_ref);
24612460
match_expr->_children.push_back(literal);
24622461
auto analyzer_ctx = std::make_shared<InvertedIndexAnalyzerCtx>();
24632462
analyzer_ctx->analyzer_key = "none";
24642463
analyzer_ctx->parser_type = InvertedIndexParserType::PARSER_NONE;
24652464
match_expr->set_analyzer_ctx(std::move(analyzer_ctx));
24662465

2467-
MatchPredicateCollector collector;
2468-
std::unordered_map<std::wstring, CollectInfo> collect_infos;
2469-
auto status =
2470-
collector.collect(runtime_state_.get(), tablet_schema, match_expr, &collect_infos);
2471-
ASSERT_TRUE(status.ok()) << status.msg();
2472-
EXPECT_TRUE(collect_infos.empty());
2466+
for (auto op : {TExprOpcode::MATCH_ANY, TExprOpcode::MATCH_ALL, TExprOpcode::MATCH_PHRASE}) {
2467+
SCOPED_TRACE(op);
2468+
match_expr->set_opcode(op);
2469+
MatchPredicateCollector collector;
2470+
std::unordered_map<std::wstring, CollectInfo> collect_infos;
2471+
auto status =
2472+
collector.collect(runtime_state_.get(), tablet_schema, match_expr, &collect_infos);
2473+
ASSERT_TRUE(status.ok()) << status.msg();
2474+
ASSERT_EQ(collect_infos.size(), 1U);
2475+
const auto& collect_info = collect_infos.begin()->second;
2476+
ASSERT_NE(collect_info.index_meta, nullptr);
2477+
EXPECT_EQ(collect_info.index_meta->index_id(), 2300);
2478+
EXPECT_EQ(collect_info.unique_terms, std::vector<std::string>({"Hello World"}));
2479+
ASSERT_EQ(collect_info.logical_scoring_leaves.size(), 1U);
2480+
EXPECT_EQ(collect_info.logical_scoring_leaves[0].clauses.size(), 1U);
2481+
}
24732482
}
24742483

24752484
TEST_F(CollectionStatisticsTest, ExplicitNoneDoesNotSelectNormalizerIndexForScoring) {

‎be/test/storage/segment/index_reader_helper_test.cpp‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -464,17 +464,20 @@ TEST_F(IndexReaderHelperTest, IsNeedSimilarityScoreWithTExprOpcodeTest) {
464464
EXPECT_FALSE(IndexReaderHelper::is_need_similarity_score(TExprOpcode::LT, &index_meta));
465465
}
466466

467-
TEST_F(IndexReaderHelperTest, MatchScoringRequiresAnalyzedPhraseIndex) {
467+
TEST_F(IndexReaderHelperTest, MatchScoringRequiresKeywordOrAnalyzedPhraseIndex) {
468468
struct Case {
469469
std::map<std::string, std::string> properties;
470470
bool expected;
471471
};
472+
// Keyword indexes score with unit frequency and length, so phrase support does not matter;
473+
// analyzed indexes still need it (or a lowercase normalizer) to score.
472474
for (const auto& test : std::vector<Case> {
473-
{.properties = {{"support_phrase", "true"}}, .expected = false},
474-
{.properties = {{"parser", "none"}, {"support_phrase", "true"}},
475-
.expected = false},
475+
{.properties = {}, .expected = true},
476+
{.properties = {{"support_phrase", "true"}}, .expected = true},
477+
{.properties = {{"parser", "none"}}, .expected = true},
478+
{.properties = {{"parser", "none"}, {"support_phrase", "true"}}, .expected = true},
476479
{.properties = {{"parser", "unknown"}, {"support_phrase", "true"}},
477-
.expected = false},
480+
.expected = true},
478481
{.properties = {{"normalizer", "lowercase"}, {"support_phrase", "true"}},
479482
.expected = true},
480483
{.properties = {{"parser", "english"}, {"support_phrase", "true"}},
Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !V2_single_row_match --
3+
1 0.287682
4+
5+
-- !V2_match --
6+
1 0.693147
7+
2 0.693147
8+
5 0.693147
9+
10+
-- !V2_match_all --
11+
1 0.693147
12+
2 0.693147
13+
5 0.693147
14+
15+
-- !V2_match_phrase --
16+
1 0.693147
17+
2 0.693147
18+
5 0.693147
19+
20+
-- !V2_search --
21+
1 0.693147
22+
2 0.693147
23+
5 0.693147
24+
25+
-- !V2_none_match --
26+
1 0.693147
27+
3 0.693147
28+
6 0.693147
29+
30+
-- !V2_none_match_phrase --
31+
1 0.693147
32+
3 0.693147
33+
6 0.693147
34+
35+
-- !V2_none_search --
36+
1 0.693147
37+
3 0.693147
38+
6 0.693147
39+
40+
-- !V2_analyzed_and_keyword --
41+
1 0.983863
42+
2 0.893357
43+
5 0.961783
44+
45+
-- !V2_search_and --
46+
1 0.983863
47+
2 0.893357
48+
5 0.961783
49+
50+
-- !V2_analyzed_and_equality --
51+
1 0.290716
52+
2 0.20021
53+
5 0.268636
54+
55+
-- !V3_single_row_match --
56+
1 0.287682
57+
58+
-- !V3_match --
59+
1 0.693147
60+
2 0.693147
61+
5 0.693147
62+
63+
-- !V3_match_all --
64+
1 0.693147
65+
2 0.693147
66+
5 0.693147
67+
68+
-- !V3_match_phrase --
69+
1 0.693147
70+
2 0.693147
71+
5 0.693147
72+
73+
-- !V3_search --
74+
1 0.693147
75+
2 0.693147
76+
5 0.693147
77+
78+
-- !V3_none_match --
79+
1 0.693147
80+
3 0.693147
81+
6 0.693147
82+
83+
-- !V3_none_match_phrase --
84+
1 0.693147
85+
3 0.693147
86+
6 0.693147
87+
88+
-- !V3_none_search --
89+
1 0.693147
90+
3 0.693147
91+
6 0.693147
92+
93+
-- !V3_analyzed_and_keyword --
94+
1 0.983863
95+
2 0.893357
96+
5 0.961783
97+
98+
-- !V3_search_and --
99+
1 0.983863
100+
2 0.893357
101+
5 0.961783
102+
103+
-- !V3_analyzed_and_equality --
104+
1 0.290716
105+
2 0.20021
106+
5 0.268636
107+
108+
-- !SNII_single_row_match --
109+
1 0.287682
110+
111+
-- !SNII_match --
112+
1 0.640724
113+
2 0.640724
114+
5 0.640724
115+
116+
-- !SNII_match_all --
117+
1 0.640724
118+
2 0.640724
119+
5 0.640724
120+
121+
-- !SNII_match_phrase --
122+
1 0.640724
123+
2 0.640724
124+
5 0.640724
125+
126+
-- !SNII_search --
127+
1 0.640724
128+
2 0.640724
129+
5 0.640724
130+
131+
-- !SNII_none_match --
132+
1 0.640724
133+
3 0.640724
134+
6 0.640724
135+
136+
-- !SNII_none_match_phrase --
137+
1 0.640724
138+
3 0.640724
139+
6 0.640724
140+
141+
-- !SNII_none_search --
142+
1 0.640724
143+
3 0.640724
144+
6 0.640724
145+
146+
-- !SNII_analyzed_and_keyword --
147+
1 0.93144
148+
2 0.840934
149+
5 0.90936
150+
151+
-- !SNII_search_and --
152+
1 0.93144
153+
2 0.840934
154+
5 0.90936
155+
156+
-- !SNII_analyzed_and_equality --
157+
1 0.290716
158+
2 0.20021
159+
5 0.268636
160+
Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,29 @@
11
-- This file is automatically generated. You should know what you did if you want to edit this
22
-- !sql_no_omit_any --
3-
GET /images/hm_bg.jpg HTTP/1.0 0.0
4-
GET /images/hm_bg.jpg HTTP/1.0 0.0
5-
GET /images/hm_bg.jpg HTTP/1.0 0.0
3+
GET /images/hm_bg.jpg HTTP/1.0 1.0
4+
GET /images/hm_bg.jpg HTTP/1.0 1.0
5+
GET /images/hm_bg.jpg HTTP/1.0 1.0
66

77
-- !sql_no_omit_phrase --
8-
GET /images/hello.jpg HTTP/1.0 0.0
9-
GET /images/hm_bg.jpg HTTP/1.0 0.0
10-
GET /images/hm_bg.jpg HTTP/1.0 0.0
11-
GET /images/hm_bg.jpg HTTP/1.0 0.0
8+
GET /images/hello.jpg HTTP/1.0 1.0
9+
GET /images/hm_bg.jpg HTTP/1.0 1.0
10+
GET /images/hm_bg.jpg HTTP/1.0 1.0
11+
GET /images/hm_bg.jpg HTTP/1.0 1.0
1212

1313
-- !sql_no_omit_all --
14-
POST /api/test HTTP/1.0 0.0
14+
POST /api/test HTTP/1.0 1.0
1515

1616
-- !sql_omit_any --
17-
GET /images/hm_bg.jpg HTTP/1.0 0.0
18-
GET /images/hm_bg.jpg HTTP/1.0 0.0
19-
GET /images/hm_bg.jpg HTTP/1.0 0.0
17+
GET /images/hm_bg.jpg HTTP/1.0 1.0
18+
GET /images/hm_bg.jpg HTTP/1.0 1.0
19+
GET /images/hm_bg.jpg HTTP/1.0 1.0
2020

2121
-- !sql_omit_phrase --
22-
GET /images/hello.jpg HTTP/1.0 0.0
23-
GET /images/hm_bg.jpg HTTP/1.0 0.0
24-
GET /images/hm_bg.jpg HTTP/1.0 0.0
25-
GET /images/hm_bg.jpg HTTP/1.0 0.0
22+
GET /images/hello.jpg HTTP/1.0 1.0
23+
GET /images/hm_bg.jpg HTTP/1.0 1.0
24+
GET /images/hm_bg.jpg HTTP/1.0 1.0
25+
GET /images/hm_bg.jpg HTTP/1.0 1.0
2626

2727
-- !sql_omit_all --
28-
POST /api/test HTTP/1.0 0.0
28+
POST /api/test HTTP/1.0 1.0
2929

0 commit comments

Comments
 (0)