Skip to content

Commit 68a2d36

Browse files
airborne12claude
andcommitted
[fix](inverted index) Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring"
### What problem does this PR solve? Issue Number: None Related PR: #60331 Problem Summary: SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score() DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring", while the same filter without score() works. The last term of a phrase prefix is a UnionTermIterator over the terms the prefix expands to. PhraseQuery sorts its iterators by document frequency and scores a row with the norm of the first one. When the terms the prefix expands to are rarer than another term of the phrase in a segment, the union comes first and UnionTermIterator::norm() threw. A union's document frequency is the sum of its terms', so whether a query fails depends on the data of each segment. On a table holding "apple red", "apple green" and "banana yellow", `apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table where `r*` matches more rows than apple the same query works. This regressed when the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading the norm of the first iterator. A norm belongs to the document, not to a term, so the sub-iterator positioned on the row, the top of the union's queue, reports it. The union now returns that norm. Result on a branch-4.1 cluster built with this change: the reported query returns 0.4700036 and `apple g` returns a score; on a table where branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row that the same filter without score() returns. This is not a cherry-pick: master replaced the phrase query engine, and the same query there scores the prefix on the tablet idf of the terms it expands to. ### Release note Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring" when the prefix matches fewer rows than the other terms. ### Check List (For Author) - Test: Regression test, Unit Test - Regression test test_match_phrase_prefix_score on a local branch-4.1 cluster. - BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass, including the new NormIsReadFromTheSubIteratorOnTheDocument. - Behavior changed: Yes (the query no longer fails) - Does this need documentation: No Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 657aba2 commit 68a2d36

4 files changed

Lines changed: 120 additions & 4 deletions

File tree

‎be/src/storage/index/inverted/util/union_term_iterator.h‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,9 @@ class UnionTermIterator {
142142

143143
int32_t doc_freq() const { return _cost; }
144144

145-
int32_t norm() const {
146-
throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
147-
"UnionTermIterator does not support scoring");
148-
}
145+
// The norm belongs to the document, so the sub-iterator positioned on it reports the same one
146+
// as any other.
147+
int32_t norm() const { return _docs_queue->top()->norm(); }
149148

150149
private:
151150
int32_t _cost = 0;

‎be/test/storage/index/inverted/util/union_term_iterator_test.cpp‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,4 +227,18 @@ TEST(UnionTermIteratorTest, PositionExhaustionAndError) {
227227
EXPECT_EQ(uti.freq(), 0);
228228
}
229229

230+
// A phrase prefix scores a document through the union of the terms its prefix expands to.
231+
TEST(UnionTermIteratorTest, NormIsReadFromTheSubIteratorOnTheDocument) {
232+
MockIterPtr it1 = std::make_shared<MockIterator>();
233+
MockIterPtr it2 = std::make_shared<MockIterator>();
234+
it1->set_postings({{10, {1}}, {30, {4}}});
235+
it2->set_postings({{20, {2}}, {30, {3}}});
236+
237+
UnionTermIterator<MockIterator> uti({it1, it2});
238+
for (int32_t doc : {10, 20, 30}) {
239+
ASSERT_EQ(uti.advance(doc), doc);
240+
EXPECT_EQ(uti.norm(), 1) << "doc=" << doc;
241+
}
242+
}
243+
230244
} // namespace doris::segment_v2
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !v2_apple_r --
3+
1 0.3544
4+
4 0.3008
5+
5 0.3008
6+
7+
-- !v3_apple_r --
8+
1 0.3544
9+
4 0.3008
10+
5 0.3008
11+
12+
-- !v2_apple_gr --
13+
2 0.3544
14+
7 0.3544
15+
16+
-- !v3_apple_gr --
17+
2 0.3544
18+
7 0.3544
19+
20+
-- !v2_apple_red_g --
21+
4 0.9415
22+
23+
-- !v3_apple_red_g --
24+
4 0.9415
25+
26+
-- !v3_rows --
27+
2
28+
7
29+
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// score() over MATCH_PHRASE_PREFIX must not fail. When the terms the prefix expands to were rarer
19+
// than the other terms of the phrase, the union of them led the phrase and the scorer read its
20+
// norm, which threw "UnionTermIterator does not support scoring".
21+
suite("test_match_phrase_prefix_score", "p0") {
22+
sql "DROP TABLE IF EXISTS test_match_phrase_prefix_score_v2"
23+
sql "DROP TABLE IF EXISTS test_match_phrase_prefix_score_v3"
24+
25+
sql """
26+
CREATE TABLE test_match_phrase_prefix_score_v2 (
27+
id INT NULL,
28+
c1 TEXT NULL,
29+
INDEX idx_c1 (c1) USING INVERTED PROPERTIES("lower_case" = "true", "parser" = "english", "support_phrase" = "true")
30+
) ENGINE=OLAP
31+
DUPLICATE KEY(id)
32+
DISTRIBUTED BY HASH(id) BUCKETS 1
33+
PROPERTIES ("replication_allocation" = "tag.location.default: 1", "inverted_index_storage_format" = "V2");
34+
"""
35+
sql """
36+
CREATE TABLE test_match_phrase_prefix_score_v3 (
37+
id INT NULL,
38+
c1 TEXT NULL,
39+
INDEX idx_c1 (c1) USING INVERTED PROPERTIES("lower_case" = "true", "parser" = "english", "support_phrase" = "true")
40+
) ENGINE=OLAP
41+
DUPLICATE KEY(id)
42+
DISTRIBUTED BY HASH(id) BUCKETS 1
43+
PROPERTIES ("replication_allocation" = "tag.location.default: 1", "inverted_index_storage_format" = "V3");
44+
"""
45+
46+
// One insert per table keeps every table to a single segment.
47+
def rows = """(1, 'apple red'), (2, 'apple green'), (3, 'banana yellow'), (4, 'apple red green'),
48+
(5, 'apple apple red'), (6, 'red red red'), (7, 'apple gray'), (8, 'apple garden path')"""
49+
sql "INSERT INTO test_match_phrase_prefix_score_v2 VALUES ${rows}"
50+
sql "INSERT INTO test_match_phrase_prefix_score_v3 VALUES ${rows}"
51+
sql "sync"
52+
53+
// 'apple r': the reported query, 'red' is rarer than 'apple'.
54+
order_qt_v2_apple_r """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v2
55+
WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score() DESC LIMIT 10 """
56+
order_qt_v3_apple_r """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v3
57+
WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score() DESC LIMIT 10 """
58+
59+
// 'apple gr' expands to green and gray, both rarer than apple.
60+
order_qt_v2_apple_gr """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v2
61+
WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' ORDER BY score() DESC LIMIT 10 """
62+
order_qt_v3_apple_gr """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v3
63+
WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' ORDER BY score() DESC LIMIT 10 """
64+
65+
// Two phrase terms before the prefix.
66+
order_qt_v2_apple_red_g """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v2
67+
WHERE c1 MATCH_PHRASE_PREFIX 'apple red g' ORDER BY score() DESC LIMIT 10 """
68+
order_qt_v3_apple_red_g """ SELECT id, round(score(), 4) s FROM test_match_phrase_prefix_score_v3
69+
WHERE c1 MATCH_PHRASE_PREFIX 'apple red g' ORDER BY score() DESC LIMIT 10 """
70+
71+
// The same rows without score().
72+
order_qt_v3_rows """ SELECT id FROM test_match_phrase_prefix_score_v3
73+
WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' """
74+
}

0 commit comments

Comments
 (0)