Repository navigation
branch-4.0: [fix](inverted index) implement phrase-level BM25 scoring with phrase frequency #60331 - #61243
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Mar 12, 2026
This was referenced Oct 27, 2026
yiguolei
pushed a commit
that referenced
this pull request
Oct 28, 2026
…X failing with "UnionTermIterator does not support scoring" (#69699) ### What problem does this PR solve? Issue Number: None Related PR: #60331 Problem Summary: SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score() DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring", while the same filter without score() works. The last term of a phrase prefix is a UnionTermIterator over the terms the prefix expands to. PhraseQuery sorts its iterators by document frequency and scores a row with the norm of the first one. When the terms the prefix expands to are rarer than another term of the phrase in a segment, the union comes first and UnionTermIterator::norm() threw. A union's document frequency is the sum of its terms', so whether a query fails depends on the data of each segment. On a table holding "apple red", "apple green" and "banana yellow", `apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table where `r*` matches more rows than apple the same query works. This regressed when the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading the norm of the first iterator. A norm belongs to the document, not to a term, so the sub-iterator positioned on the row, the top of the union's queue, reports it. The union now returns that norm. Result on a branch-4.1 cluster built with this change: the reported query returns 0.4700036 and `apple g` returns a score; on a table where branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row that the same filter without score() returns. This is not a cherry-pick: master replaced the phrase query engine, and the same query there scores the prefix on the tablet idf of the terms it expands to (#69698). ### Release note Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring" when the prefix matches fewer rows than the other terms. ### Check List (For Author) - Test: Regression test, Unit Test - Regression test test_match_phrase_prefix_score on a local branch-4.1 cluster. - BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass, including the new NormIsReadFromTheSubIteratorOnTheDocument. - Behavior changed: Yes (the query no longer fails) - Does this need documentation: No ### Notes for reviewers The score this branch returns is still the phrase scored on the idf of its plain terms (`apple r` returns 0.4700036); it does not count the terms the prefix expands to. That is left as it was, because it needs the statistics and engine changes of master. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #60331