Repository navigation
[fix](inverted index) implement phrase-level BM25 scoring with phrase frequency - #60331
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31749 ms |
ClickBench: Total hot run time: 28.79 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
zzzxl1993
force-pushed
the
202501291132
branch
from
February 25, 2026 02:54
929c34a to
1e81f01
Compare
Contributor
Author
|
run buildall |
zzzxl1993
force-pushed
the
202501291132
branch
from
February 25, 2026 03:12
1e81f01 to
1b293f6
Compare
TPC-H: Total hot run time: 28707 ms |
TPC-DS: Total hot run time: 184213 ms |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 28765 ms |
TPC-DS: Total hot run time: 183744 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
PR approved by anyone and no changes requested. |
yiguolei
pushed a commit
that referenced
this pull request
Mar 12, 2026
… with phrase frequency #60331 (#61243) Cherry-picked from #60331 Co-authored-by: zzzxl <[email protected]>
Closed
This was referenced Oct 27, 2026
yiguolei
pushed a commit
that referenced
this pull request
Oct 28, 2026
…X failing with "UnionTermIterator does not support scoring" (#69699) ### What problem does this PR solve? Issue Number: None Related PR: #60331 Problem Summary: SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score() DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring", while the same filter without score() works. The last term of a phrase prefix is a UnionTermIterator over the terms the prefix expands to. PhraseQuery sorts its iterators by document frequency and scores a row with the norm of the first one. When the terms the prefix expands to are rarer than another term of the phrase in a segment, the union comes first and UnionTermIterator::norm() threw. A union's document frequency is the sum of its terms', so whether a query fails depends on the data of each segment. On a table holding "apple red", "apple green" and "banana yellow", `apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table where `r*` matches more rows than apple the same query works. This regressed when the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading the norm of the first iterator. A norm belongs to the document, not to a term, so the sub-iterator positioned on the row, the top of the union's queue, reports it. The union now returns that norm. Result on a branch-4.1 cluster built with this change: the reported query returns 0.4700036 and `apple g` returns a score; on a table where branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row that the same filter without score() returns. This is not a cherry-pick: master replaced the phrase query engine, and the same query there scores the prefix on the tablet idf of the terms it expands to (#69698). ### Release note Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring" when the prefix matches fewer rows than the other terms. ### Check List (For Author) - Test: Regression test, Unit Test - Regression test test_match_phrase_prefix_score on a local branch-4.1 cluster. - BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass, including the new NormIsReadFromTheSubIteratorOnTheDocument. - Behavior changed: Yes (the query no longer fails) - Does this need documentation: No ### Notes for reviewers The score this branch returns is still the phrase scored on the idf of its plain terms (`apple r` returns 0.4700036); it does not count the terms the prefix expands to. That is left as it was, because it needs the statistics and engine changes of master. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)