Skip to content

branch-4.0: [fix](inverted index) implement phrase-level BM25 scoring with phrase frequency #60331 - #61243

Merged
yiguolei merged 1 commit into
branch-4.0from
auto-pick-60331-branch-4.0
Mar 12, 2026
Merged

yiguolei merged 1 commit into
branch-4.0from
auto-pick-60331-branch-4.0

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Cherry-picked from #60331

@github-actions
github-actions Bot requested a review from yiguolei as a code owner March 12, 2026 03:57
@Thearas

Thearas commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 12, 2026
@Thearas

Thearas commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 68.75% (44/64) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.99% (19162/36159)
Line Coverage 36.17% (178361/493098)
Region Coverage 32.78% (138286/421868)
Branch Coverage 33.77% (60036/177759)

@yiguolei
yiguolei merged commit 8d0ba18 into branch-4.0 Mar 12, 2026
26 of 29 checks passed
@github-actions
github-actions Bot deleted the auto-pick-60331-branch-4.0 branch March 12, 2026 07:19
yiguolei pushed a commit that referenced this pull request Oct 28, 2026
…X failing with "UnionTermIterator does not support scoring" (#69699)

### What problem does this PR solve?

Issue Number: None

Related PR: #60331

Problem Summary:
SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY
score()
DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support
scoring",
while the same filter without score() works.

The last term of a phrase prefix is a UnionTermIterator over the terms
the
prefix expands to. PhraseQuery sorts its iterators by document frequency
and
scores a row with the norm of the first one. When the terms the prefix
expands
to are rarer than another term of the phrase in a segment, the union
comes
first and UnionTermIterator::norm() threw. A union's document frequency
is the
sum of its terms', so whether a query fails depends on the data of each
segment. On a table holding "apple red", "apple green" and "banana
yellow",
`apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table
where
`r*` matches more rows than apple the same query works. This regressed
when
the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading
the
norm of the first iterator.

A norm belongs to the document, not to a term, so the sub-iterator
positioned
on the row, the top of the union's queue, reports it. The union now
returns
that norm.

Result on a branch-4.1 cluster built with this change: the reported
query
returns 0.4700036 and `apple g` returns a score; on a table where
branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row
that
the same filter without score() returns.

This is not a cherry-pick: master replaced the phrase query engine, and
the
same query there scores the prefix on the tablet idf of the terms it
expands to
(#69698).

### Release note

Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator
does not
support scoring" when the prefix matches fewer rows than the other
terms.

### Check List (For Author)

- Test: Regression test, Unit Test
- Regression test test_match_phrase_prefix_score on a local branch-4.1
      cluster.
    - BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass,
      including the new NormIsReadFromTheSubIteratorOnTheDocument.
- Behavior changed: Yes (the query no longer fails)
- Does this need documentation: No

### Notes for reviewers

The score this branch returns is still the phrase scored on the idf of
its
plain terms (`apple r` returns 0.4700036); it does not count the terms
the
prefix expands to. That is left as it was, because it needs the
statistics and
engine changes of master.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants