Skip to content

branch-4.1: [fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch - #69825

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
airborne12:fix-pattern-score-es-branch-4.1
Oct 30, 2026
Merged

yiguolei merged 1 commit into
apache:branch-4.1from
airborne12:fix-pattern-score-es-branch-4.1

Conversation

@airborne12

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #69815, #69269

Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself on branch-4.1:

-- 'Inverted indexes are suitable for full text search',
-- 'The parser splits text into multiple terms', ...
SELECT score() FROM document WHERE SEARCH('content:/.*ext/');  -- BM25 sum
SELECT score() FROM document WHERE SEARCH('content:*ext');     -- BM25 sum
SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0

Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.

#69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard
sum the BM25 of every expanded term, i.e. scoring_boolean always on,
with idf and avgdl taken from the current segment rather than the tablet
statistics MATCH uses, so identical rows in two segments scored
differently. MATCH_REGEXP never requested a scoring context, so it
scored 0 like a filter.

Fix, the same as #69815 on master:

  1. Revert branch-4.1: [fix](be) Score regexp and wildcard search terms with BM25 #69052 #69269: RegexpWeight scores every pattern with the constant
    scorer again; its suite that pinned the BM25 sums is removed.
  2. MATCH_REGEXP requests scoring on every index, and the legacy
    RegexpQuery records 1 for every matching row when score() is
    requested. The MATCH statistics collector skips it, since a constant
    reads no statistics and the pattern is not text to analyze.

A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the
keyword reader requests scoring (#69791); until then it keeps 0.

Release note

score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP on a tokenized index is 1 instead of 0.

Check List (For Author)

  • Test: Regression test / Unit Test / Manual test
    • Regression test: new search/test_search_score_pattern_constant
      (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3
      blocks; 16 existing score suites in inverted_index_p0 and search
      pass unchanged
    • Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*,
      RegexpQueryTest and the query_v2 regexp/wildcard tests restored by
      the revert)
    • Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment
      table now scores 1 for every pattern hit on a branch-4.1 ASAN build
  • Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP
    scores change as described in the release note.
  • Does this need documentation: No

🤖 Generated with Claude Code

…sticsearch

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69815, apache#69269

Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself on branch-4.1:

    -- 'Inverted indexes are suitable for full text search',
    -- 'The parser splits text into multiple terms', ...
    SELECT score() FROM document WHERE SEARCH('content:/.*ext/');  -- BM25 sum
    SELECT score() FROM document WHERE SEARCH('content:*ext');     -- BM25 sum
    SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0

Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.

apache#69269 (the branch-4.1 pick of apache#69052) made SEARCH regexp and wildcard
sum the BM25 of every expanded term, i.e. scoring_boolean always on,
with idf and avgdl taken from the current segment rather than the tablet
statistics MATCH uses, so identical rows in two segments scored
differently. MATCH_REGEXP never requested a scoring context, so it
scored 0 like a filter.

Fix, the same as apache#69815 on master:
1. Revert apache#69269: RegexpWeight scores every pattern with the constant
   scorer again; its suite that pinned the BM25 sums is removed.
2. MATCH_REGEXP requests scoring on every index, and the legacy
   RegexpQuery records 1 for every matching row when score() is
   requested. The MATCH statistics collector skips it, since a constant
   reads no statistics and the pattern is not text to analyze.

A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the
keyword reader requests scoring (apache#69791); until then it keeps 0.

### Release note

score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP on a tokenized index is 1 instead of 0.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - Regression test: new search/test_search_score_pattern_constant
      (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3
      blocks; 16 existing score suites in inverted_index_p0 and search
      pass unchanged
    - Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*,
      RegexpQueryTest and the query_v2 regexp/wildcard tests restored by
      the revert)
    - Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment
      table now scores 1 for every pattern hit on a branch-4.1 ASAN build
- Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP
  scores change as described in the release note.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12
airborne12 marked this pull request as ready for review October 29, 2026 12:06
@airborne12
airborne12 requested a review from yiguolei as a code owner October 29, 2026 12:06
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.82% (27307/44170)
Line Coverage 46.49% (284850/612669)
Region Coverage 42.09% (225358/535424)
Branch Coverage 43.71% (105134/240527)

@airborne12

Copy link
Copy Markdown
Member Author

run p0

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.26% (32331/42958)
Line Coverage 59.69% (362771/607781)
Region Coverage 56.34% (302262/536476)
Branch Coverage 57.35% (137774/240224)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (15/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.31% (32350/42958)
Line Coverage 59.73% (363032/607781)
Region Coverage 56.37% (302423/536476)
Branch Coverage 57.39% (137874/240224)

@yiguolei
yiguolei merged commit 4735ebc into apache:branch-4.1 Oct 30, 2026
34 of 36 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

airborne12 added a commit to airborne12/apache-doris that referenced this pull request Nov 1, 2026
…ch-score-branch-4.1

Resolve the conflicts with apache#69825 and the merge-on-write statistics change:

- index_reader_helper.h: keep the regexp shortcut in front of can_score_match.
- index_reader_helper_test.cpp: keep both tests.
- collection_statistics.cpp: keep the BKD check in front of the CLucene open,
  which is now in a try/catch, so a BKD index no longer needs to reach it.

With the keyword reader of this PR, a MATCH_REGEXP on a keyword index now scores
1, so test_search_score_pattern_constant gets the case it had to leave out.

Tested on the merged tree: test_match_keyword_score, the VARIANT and numeric
cases by hand, 17 score-related suites, 91 unit tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants