Skip to content

branch-4.2: [fix](inverted index) Score MATCH_PHRASE_PREFIX on the tablet idf of the terms the prefix expands to #69698 - #69801

Merged
yiguolei merged 1 commit into
apache:branch-4.2from
airborne12:backport/69698-branch-4.2
Oct 30, 2026
Merged

branch-4.2: [fix](inverted index) Score MATCH_PHRASE_PREFIX on the tablet idf of the terms the prefix expands to #69698#69801
yiguolei merged 1 commit into
apache:branch-4.2from
airborne12:backport/69698-branch-4.2

Conversation

@airborne12

@airborne12 airborne12 commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #69698, #69752

Problem Summary:
Backport of #69698 to branch-4.2. score() over MATCH_PHRASE_PREFIX ignored the prefix and
differed between inverted index storage formats:

  1. The idf of a phrase prefix now counts every term the prefix expands to, as Lucene's
    multi-phrase query and Elasticsearch do, and all idfs come from the tablet statistics. The
    statistics expand the prefix in every segment and keep the inverted_index_max_expansions
    smallest terms of the tablet that a returned row holds (a session limit of 0 keeps 50), so a
    term's document frequency is counted in every segment that holds it and merge-on-write hidden
    rows do not change the scores after a compaction. A term whose bytes are not valid UTF-8 is not
    scored.
  2. A phrase prefix of one token scores 1 on every storage format; on V2/V3 it read 0, so
    score() > 0 dropped every row it matched.

See #69698 for the full description and the review rounds.

This change builds on #69438 (BM25 statistics count only visible rows of merge-on-write tables),
whose branch-4.2 backport #69752 is merged. The PR is rebased onto the current branch-4.2 and
contains only the backport commit; its patch is identical to the one validated below.

Branch adaptations

Release note

score() over MATCH_PHRASE_PREFIX counts the terms the prefix expands to, and a phrase prefix of
one token scores 1 on every inverted index storage format.

Check List (For Author)

  • Test: Regression test, Unit Test
    • Validated locally on branch-4.2 at 711db43, the commit before [enhancement](lance) Add cache diagnostics and BE runtime index selection #69625, with the same two
      commits on top: the newest branch-4.2 calls a lance-c API that no thirdparty build on the
      machine used here provides. The PR branch itself is on the current branch-4.2 and is built
      by CI.
    • test_match_phrase_prefix_score passes with the .out of master unchanged (V2, V3, SNII and
      the capped two-segment case).
    • test_bm25_score, test_bm25_score_parallel_scan_statistics, test_bm25_score_range_filter,
      test_bm25_score_replica_statistics, test_bm25_score_variant, test_score_filter_matrix,
      test_score_filter_usage and test_search_score_topn_predicates pass.
    • 1644 related BE unit tests pass (PhrasePrefixQueryV2Test, CollectionStatistics*,
      PrefixExpansionSetTest, SNII, InvertedIndexReaderTest, MultiSegmentCollectorTest,
      BM25SimilarityTest).
  • Behavior changed: Yes (scores of MATCH_PHRASE_PREFIX change)
  • Does this need documentation: No

🤖 Generated with Claude Code

@airborne12
airborne12 requested a review from yiguolei as a code owner October 29, 2026 03:13
@hello-stephen

hello-stephen commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

run buildall

@airborne12 @claude
[fix](inverted index) Score MATCH_PHRASE_PREFIX on the tablet idf of …
370d68a 
…the terms the prefix expands to (apache#69698)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68758

Problem Summary:
score() over MATCH_PHRASE_PREFIX gave scores that ignored the prefix and
differed between inverted index storage formats.

1. The prefix of a phrase prefix never entered the idf. "apple r" scored
on
idf(apple) alone, and a rare expansion such as "rose" ranked like a
common
   one such as "red". Lucene's multi-phrase query, and so Elasticsearch,
sum the idf of the phrase terms and of every term the prefix expands to.
The statistics collector now registers the prefix; every segment expands
   it, counts the document frequency of each expanded term in the tablet
statistics, and records which terms the prefix expanded to. The scorer
   adds their idf to the phrase's, so all idfs come from the same tablet
   statistics and the score does not depend on how rows are split into
segments. With the reported table, `apple r` now scores 1.4508328, the
value Elasticsearch 9.5 gives; on a 10-row table, `apple r`, `apple ro`
and `apple g` match Elasticsearch to four decimals, over one segment and
   over three.
2. A prefix of one token scored 0 on V2/V3 (CLucene did not publish
scores
for expansion leaves) and 1 on SNII. It now scores 1 on every format, as
   the constant score of Lucene and Elasticsearch prefix queries. At 0,
   `score() > 0` dropped every row the prefix matched.
3. No test covered score() over MATCH_PHRASE_PREFIX across storage
formats.
Add unit tests for the collector, the segment expansion, the tablet idf
   and the constant score, and a regression suite over V2, V3 and SNII.

Two existing tests asserted the old behavior and are updated: the SNII
public phrase prefix scoring test (the idf now counts the expanded term)
and the CLucene one-token phrase prefix test (it asserted no score).

A prefix collected by a SEARCH predicate has no recorded expansion and
keeps
scoring the phrase terms alone.

Source-PR: apache#69698
Source-Head: 24b4c54

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12
airborne12 force-pushed the backport/69698-branch-4.2 branch from 98496f3 to 370d68a Compare October 29, 2026 06:57
@airborne12

airborne12 commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

hello-stephen commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 92.52% (136/147) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.57% (30143/46685)
Line Coverage 49.42% (318543/644569)
Region Coverage 45.19% (256576/567737)
Branch Coverage 46.85% (121106/258483)

@hello-stephen

hello-stephen commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.28% (143/147) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.35% (34616/45340)
Line Coverage 61.19% (391151/639190)
Region Coverage 57.97% (329613/568630)
Branch Coverage 58.98% (152270/258191)

@yiguolei
yiguolei merged commit da73961 into apache:branch-4.2 Oct 30, 2026
35 of 36 checks passed
@github-actions

github-actions Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@airborne12 @hello-stephen @yiguolei