Skip to content

branch-4.1: [fix](inverted index) Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring" - #69699

Merged
yiguolei merged 3 commits into
apache:branch-4.1from
airborne12:fix-match-phrase-prefix-score-branch-4.1
Oct 28, 2026
Merged

yiguolei merged 3 commits into
apache:branch-4.1from
airborne12:fix-match-phrase-prefix-score-branch-4.1

Conversation

@airborne12

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #60331

Problem Summary:
SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score()
DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring",
while the same filter without score() works.

The last term of a phrase prefix is a UnionTermIterator over the terms the
prefix expands to. PhraseQuery sorts its iterators by document frequency and
scores a row with the norm of the first one. When the terms the prefix expands
to are rarer than another term of the phrase in a segment, the union comes
first and UnionTermIterator::norm() threw. A union's document frequency is the
sum of its terms', so whether a query fails depends on the data of each
segment. On a table holding "apple red", "apple green" and "banana yellow",
apple r (red in 1 row, apple in 2) and apple g fail; on a table where
r* matches more rows than apple the same query works. This regressed when
the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading the
norm of the first iterator.

A norm belongs to the document, not to a term, so the sub-iterator positioned
on the row, the top of the union's queue, reports it. The union now returns
that norm.

Result on a branch-4.1 cluster built with this change: the reported query
returns 0.4700036 and apple g returns a score; on a table where
branch-4.1 (4.1.5) threw for apple gr, the query now returns the row that
the same filter without score() returns.

This is not a cherry-pick: master replaced the phrase query engine, and the
same query there scores the prefix on the tablet idf of the terms it expands to
(#69698).

Release note

Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not
support scoring" when the prefix matches fewer rows than the other terms.

Check List (For Author)

  • Test: Regression test, Unit Test
    • Regression test test_match_phrase_prefix_score on a local branch-4.1
      cluster.
    • BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass,
      including the new NormIsReadFromTheSubIteratorOnTheDocument.
  • Behavior changed: Yes (the query no longer fails)
  • Does this need documentation: No

Notes for reviewers

The score this branch returns is still the phrase scored on the idf of its
plain terms (apple r returns 0.4700036); it does not count the terms the
prefix expands to. That is left as it was, because it needs the statistics and
engine changes of master.

🤖 Generated with Claude Code

…th "UnionTermIterator does not support scoring"

### What problem does this PR solve?

Issue Number: None

Related PR: apache#60331

Problem Summary:
SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score()
DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring",
while the same filter without score() works.

The last term of a phrase prefix is a UnionTermIterator over the terms the
prefix expands to. PhraseQuery sorts its iterators by document frequency and
scores a row with the norm of the first one. When the terms the prefix expands
to are rarer than another term of the phrase in a segment, the union comes
first and UnionTermIterator::norm() threw. A union's document frequency is the
sum of its terms', so whether a query fails depends on the data of each
segment. On a table holding "apple red", "apple green" and "banana yellow",
`apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table where
`r*` matches more rows than apple the same query works. This regressed when
the phrase-level BM25 scoring of apache#60331 (apache#61243 in 4.0) started reading the
norm of the first iterator.

A norm belongs to the document, not to a term, so the sub-iterator positioned
on the row, the top of the union's queue, reports it. The union now returns
that norm.

Result on a branch-4.1 cluster built with this change: the reported query
returns 0.4700036 and `apple g` returns a score; on a table where
branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row that
the same filter without score() returns.

This is not a cherry-pick: master replaced the phrase query engine, and the
same query there scores the prefix on the tablet idf of the terms it expands to.

### Release note

Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not
support scoring" when the prefix matches fewer rows than the other terms.

### Check List (For Author)

- Test: Regression test, Unit Test
    - Regression test test_match_phrase_prefix_score on a local branch-4.1
      cluster.
    - BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass,
      including the new NormIsReadFromTheSubIteratorOnTheDocument.
- Behavior changed: Yes (the query no longer fails)
- Does this need documentation: No

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed files at 68a2d36. No actionable new findings and no existing inline blockers. Review converged in 1 round: production, tests, and the separate norm-invariant reviewer all returned NO_NEW_VALUABLE_FINDINGS; all candidates and risk focuses are dispositioned.

The fix is supported by the complete scoring path:
MATCH_PHRASE_PREFIX -> prefix expansion -> PhraseQuery -> document intersection -> phrase frequency -> UnionTermIterator::norm -> BM25 score.
The scored document is nonterminal and matches the heap root. All prefix alternatives share the same reader and field; the pinned CLucene implementation obtains norms from that field's document-indexed array. Consuming phrase positions leaves the document and norm unchanged. Empty expansions retain a fallback child, and exhausted postings do not reach scoring.

Required checkpoint conclusions:

  1. Goal: the delegation removes the reported data-dependent scoring exception. The new unit test and V2/V3 regressions target it; the r and gr cases make the union strictly lead by document frequency.
  2. Scope: one production accessor and focused tests; no unrelated production change.
  3. Concurrency and locking: query-local iterators execute synchronously under existing searcher ownership. No new shared mutation, thread, lock, heavy critical section, or lock-order issue.
  4. Lifecycle and static initialization: existing owners retain children and reader through scoring. No new allocation, ownership cycle, static dependency, or cleanup obligation.
  5. Configuration: no configuration item or dynamic-update behavior added.
  6. Compatibility: signatures and serialized/index formats are unchanged; both V2 and V3 use the reviewed matching path.
  7. Thrift/protobuf: no definitions changed; cross-release field-ID comparison is inapplicable.
  8. Parallel paths: standalone prefix traversal never calls this norm accessor; the separate query_v2 union implementations already obtain a current-document child's norm. No omitted counterpart fix found.
  9. Special conditions: existing nonempty, positioned-iterator prerequisites and the terminal-document scoring gate justify direct delegation. No new conditional is introduced.
  10. Test coverage: single/multiple expanded terms, varied document lengths, multiple preceding phrase terms, both formats, and an unscored row baseline are covered. The mock always returns norm 1, so document-dependent mock norms would be optional strengthening; real regression scores already distinguish document lengths. Missing/exhausted posting cases were checked statically.
  11. Expected results: every new row and rounded score follows from exact phrase positions and the branch's formulas: 8 documents, 20 tokens, average length 2.5, apple document frequency 6, red frequency 4, yielding 0.3544, 0.3008, and 0.9415. order_qt_ sorts comparison results, and LIMIT 10 includes all matches. Output generation provenance cannot be proved by static inspection.
  12. Observability and errors: existing query timers and CLucene/Doris exception-to-Status boundaries cover the accessor; no new logging or metric is required. No failure is silently discarded.
  13. Transactions and persistence: no EditLog, visibility-version, delete-bitmap, or failover behavior changes.
  14. Data writes and crash safety: no production write, transaction boundary, or resource acquisition is added.
  15. FE-BE variables: no new transmitted value or sending path.
  16. Performance and memory: constant-time heap-root delegation; no new scan, copy, allocation, or memory-tracking obligation.
  17. Other concerns: the branch's existing phrase-prefix IDF excludes expanded terms, as the PR notes; this change preserves that behavior. No new nullable-column, bitmap, or storage-visibility issue found.

User focus: no additional focus was supplied. Both initial risk focuses (norm validity and test/expected-score coverage) were resolved without an inline finding. No unresolved candidate or source-coverage gap remains.

Validation limits: static source review only. No builds, unit/regression tests, runtime queries, benchmarks, or sanitizers were run. Repository source was not modified and submodules were not initialized. The dependency norm implementation was inspected read-only at the checkout's pinned CLucene commit 08d45e541fef1e48dc4ab00d2d5fde9569b49ee3.

Review converged after 1 round(s).

…orm() to throw

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69699

Problem Summary:
PhraseFreqTest.UnionTermIterator_NormThrowsException asserted that
UnionTermIterator::norm() throws, which the previous commit removed: a union
now returns the norm of the sub-iterator on the current row. The test asserts
the norm instead. The earlier unit test run of the change used a filter that
did not include this test.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - 1364 inverted index related BE unit tests pass, including this one.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files at head 52df5f3. No verified new defect was found. There are no existing inline findings to retain. The review converged after one round: the production, test, and separate norm-risk reviewers all returned NO_NEW_VALUABLE_FINDINGS; all risk questions were independently checked and closed.

The complete path was traced: MATCH_PHRASE_PREFIX expansion -> same-field posting iterators -> union/phrase lead selection -> document alignment -> phrase-frequency counting -> norm -> BM25 collection. Position consumption preserves the current document, and the heap top is also the source of the union's document ID. The pinned CLucene dependency (08d45e541fef1e48dc4ab00d2d5fde9569b49ee3), inspected through read-only GitHub API requests, confirms that each child's norm comes from that document's field norm array.

  1. Goal and evidence: the change removes the throwing norm implementation on union-led scored phrase prefixes. Added V2/V3 cases force union leadership for both one and multiple expansions and check scores and matching rows. These tests were inspected, not executed.
  2. Scope and clarity: one constant-time delegation plus focused unit and regression changes; the comment explains document norm identity.
  3. Concurrency and locks: no new concurrency or shared mutable state. Each synchronous query owns its posting iterators and heap; cached searcher ownership remains retained during search. No lock order or work under locks changes.
  4. Lifecycle and initialization: shared child ownership keeps heap pointers valid. No owner cycle, cleanup obligation, global, or static initializer is introduced.
  5. Configuration: no configuration item or dynamic update behavior changes.
  6. Compatibility: no function signature, encoded norm representation, storage layout, or wire format changes; no rolling-upgrade adaptation is required.
  7. Thrift/protobuf: no definitions are touched, so release-branch field/enum comparison is inapplicable.
  8. Parallel paths: both V2/V3 MATCH_PHRASE_PREFIX cases reach this legacy implementation. Plain PrefixQuery does not read union norms; sloppy phrase paths reject multi-terms. The separate query_v2 UnionPostings already delegates to a child on the current document.
  9. Special conditions: no new condition is added. Expansion has a nonempty fallback, and scoring reads norm only after a finite aligned document with positive phrase frequency. Empty/default/exhausted unions are not newly reachable through this call.
  10. Test coverage: changed unit cases, registration, fixture setup, deterministic order_qt output, multiple expansions, three-term phrases, unequal document lengths, nonmatches, and the unscored control were checked. Existing unit cases also cover cross-document movement and exhaustion.
  11. Expected results: every new row and score is consistent with the branch formulas. Eight documents contain 20 tokens (avgdl 2.5); apple/red document frequencies are 6/4. The resulting rounded scores are 0.3544, 0.3008, and 0.9415, matching the oracle. This branch's existing plain-term-only phrase IDF behavior remains unchanged.
  12. Observability and errors: existing search timers and CLucene/Doris exception conversion cover this path; a scalar read adds no logging or metric requirement and discards no Status.
  13. Transactions and persistence: no EditLog, failover, committed-data visibility, visible-version, rowset, or delete-bitmap behavior changes.
  14. Writes and atomicity: no production data-write or transaction path changes; fixture inserts use existing mechanisms. The new read introduces no resource acquisition or crash cleanup.
  15. FE/BE propagation: no new transmitted variable; the existing similarity flag is preserved through phrase-prefix expansion and child creation.
  16. Performance and memory: the method is O(1) with no new allocation, copy, scan, or memory-accounting obligation.
  17. Other correctness: document/position separation, shared-document ties, missing expansions, omitted norms, field identity, and reader boundaries were rechecked. No nullable-column or three-valued bitmap semantics change, and no additional issue was substantiated.

No additional user focus was supplied. All changed files and assigned risk scopes are covered; no defect candidate remains unresolved. Validation is static only: no builds, tests, benchmarks, or runtime checks were performed, and submodules were left uninitialized. Test breadth limits remain: mock norms are constant, multi-expansion matches have equal lengths, and the new suite does not separately exercise null/empty input or multiple scanners/segments. Runtime results and oracle-generation provenance were not independently validated; these limits do not establish an introduced defect.

Review converged after 1 round(s).

@airborne12
airborne12 marked this pull request as ready for review October 28, 2026 02:43
@airborne12
airborne12 requested a review from yiguolei as a code owner October 28, 2026 02:43
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.22% (32278/42909)
Line Coverage 59.63% (361927/606919)
Region Coverage 56.28% (301458/535686)
Branch Coverage 57.28% (137309/239712)

@airborne12

Copy link
Copy Markdown
Member Author

run cloud_p0

…coring

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69699

Problem Summary: Cloud CI randomizes enable_common_expr_pushdown per
session. When it is false, MATCH_PHRASE_PREFIX still filters rows but
score() returns zero. The new regression therefore failed with [4, 0]
instead of [4, 0.9415]. Enable expression pushdown for the scoring test.

Compare the rounded scores as strings. The framework's numeric comparison
can round an expected score below 0.5 to the actual integer zero and accept
it, so the earlier assertions did not detect the missing scores.

### Release note

None

### Check List (For Author)

- Test: Regression test and comparison check
    - Replayed the failing three-term query with expression pushdown off
      and on: [4, 0] versus [4, 0.9415].
    - Focused V2/V3 regression replay through run-regression-test.sh on
      an existing branch-4.1 backend.
    - Loaded the current OutputUtils source to verify that string
      comparison rejects zero for every expected score; compiled the suite.
- Behavior changed: No (test setup and comparisons only)
- Does this need documentation: No
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.22% (32278/42909)
Line Coverage 59.63% (361927/606919)
Region Coverage 56.28% (301458/535686)
Branch Coverage 57.28% (137309/239712)

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: workspace routing discovery unauthorized (401)
Workflow run: https://github.com/apache/doris/actions/runs/40644995668

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new actionable defects found at head 365d0e9. Reviewed all five changed files and the complete caller-to-score path. Review converged in one round: both the complete-review and separate norm-risk checks returned NO_NEW_VALUABLE_FINDINGS; main independently verified their conclusions. There are no existing inline findings or outstanding candidates. No additional user focus was supplied.

The fix is consistent with the execution flow: MATCH_PHRASE_PREFIX expands its last token within one field, PhraseQuery intersects document iterators, exact matching consumes positions, and scoring reads the lead iterator's document norm. The heap top is a live child on that document. The pinned CLucene implementation refreshes its cached norm on next/skipTo and preserves it while reading positions, so same-document ties and exhausted siblings do not select a different document's norm.

Critical checkpoint conclusions:

  1. Goal: the forwarding accessor removes the reported exception when a rare prefix union becomes the phrase lead. Added unit and V2/V3 SQL tests exercise the affected path; inspected only.
  2. Scope and clarity: one production accessor change with focused unit/regression coverage.
  3. Concurrency and locks: posting cursors belong to a local query and are consumed synchronously; no new shared mutable state, threads, locks, heavy locked work or lock order.
  4. Lifecycle and static initialization: existing shared child ownership keeps heap pointers alive; no new allocation, ownership cycle, destruction dependency or static initialization.
  5. Configuration: none added or changed; the regression sets an existing expression-pushdown variable.
  6. Compatibility: no signature, on-disk index format, wire format or mixed-version protocol changes; V2 and V3 use the same norm behavior.
  7. Thrift/protobuf: no definitions changed; release-branch ID comparison is inapplicable.
  8. Parallel paths: checked non-leading unions, unscored phrases, single-token prefixes, ordinary/sloppy phrases and phrase-edge queries. SEARCH/query_v2 already reads a child norm for its current union document; no mirrored fix is needed. Existing single-token-prefix scoring behavior is outside this change.
  9. Special conditions: nonempty expansion fallback, finite document intersection and positive phrase frequency establish the norm accessor's preconditions. Position exhaustion does not advance documents or invalidate the cached norm.
  10. Test coverage: inspected child switching/ties in unit assertions and real-index V2/V3 cases with one/multiple expanded terms, multiple phrase terms, differing document lengths and an unscored control. Empty/exhausted and scoring-disabled paths were traced statically. Constant mock norms limit isolated unit sensitivity; the SQL regression checks real nonzero length-dependent scores.
  11. Expected results: every new result row and score agrees with token positions and the existing BM25 formula: eight documents, 20 tokens, avgdl=2.5, apple document frequency 6 and red frequency 4 yield 0.3544, 0.3008 and 0.9415 as recorded. String casts reject zero scores through exact comparison; order_qt_ provides deterministic output. Existing omission of expanded-prefix IDF remains unchanged, consistent with the PR's stated scope.
  12. Observability and errors: the existing query timers and exception-to-Status boundaries remain sufficient; no new distributed operation warrants additional logs or metrics.
  13. Transactions and persistence: no EditLog, commit, replay, failover, visibility-version or delete-bitmap changes.
  14. Data writes and atomicity: production code is query-only; no write/crash semantics changed.
  15. FE/BE variable propagation: no new transmitted state or send paths.
  16. Performance and memory: heap-top lookup and cached norm access are O(1), without new allocations, copies, scans or untracked ownership.
  17. Further checks: rechecked heap ties, exhausted siblings, missing-prefix fallback, norm-loading flags, phrase-position consumption, multiple CLucene segments and score-test sensitivity. No reportable issue or unresolved question remains.

Coverage limits: static review only. No builds, tests, benchmarks, source edits or submodule initialization were performed. Runtime behavior and generated-output provenance were not independently validated.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.83% (27298/44147)
Line Coverage 46.50% (284679/612269)
Region Coverage 42.07% (225138/535094)
Branch Coverage 43.71% (105052/240312)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.15% (32267/42934)
Line Coverage 59.60% (361979/607374)
Region Coverage 56.15% (301056/536145)
Branch Coverage 57.21% (137301/240009)

@yiguolei
yiguolei merged commit 5fc51d2 into apache:branch-4.1 Oct 28, 2026
30 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@airborne12
airborne12 deleted the fix-match-phrase-prefix-score-branch-4.1 branch October 29, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants