Repository navigation
branch-4.1: [fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch - #69825
Merged
yiguolei merged 1 commit intoOct 30, 2026
Conversation
…sticsearch ### What problem does this PR solve? Issue Number: None Related PR: apache#69815, apache#69269 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself on branch-4.1: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. apache#69269 (the branch-4.1 pick of apache#69052) made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, with idf and avgdl taken from the current segment rather than the tablet statistics MATCH uses, so identical rows in two segments scored differently. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix, the same as apache#69815 on master: 1. Revert apache#69269: RegexpWeight scores every pattern with the constant scorer again; its suite that pinned the BM25 sums is removed. 2. MATCH_REGEXP requests scoring on every index, and the legacy RegexpQuery records 1 for every matching row when score() is requested. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze. A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the keyword reader requests scoring (apache#69791); until then it keeps 0. ### Release note score() of SEARCH regexp and wildcard clauses is a constant 1 per matching row again, like prefix clauses and Elasticsearch's default. score() of MATCH_REGEXP on a tokenized index is 1 instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new search/test_search_score_pattern_constant (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3 blocks; 16 existing score suites in inverted_index_p0 and search pass unchanged - Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*, RegexpQueryTest and the query_v2 regexp/wildcard tests restored by the revert) - Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment table now scores 1 for every pattern hit on a branch-4.1 ASAN build - Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP scores change as described in the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
airborne12
marked this pull request as ready for review
October 29, 2026 12:06
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Member
Author
|
run p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Oct 30, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Nov 1, 2026
…ch-score-branch-4.1 Resolve the conflicts with apache#69825 and the merge-on-write statistics change: - index_reader_helper.h: keep the regexp shortcut in front of can_score_match. - index_reader_helper_test.cpp: keep both tests. - collection_statistics.cpp: keep the BKD check in front of the CLucene open, which is now in a try/catch, so a BKD index no longer needs to reach it. With the keyword reader of this PR, a MATCH_REGEXP on a keyword index now scores 1, so test_search_score_pattern_constant gets the case it had to leave out. Tested on the merged tree: test_match_keyword_score, the VARIANT and numeric cases by hand, 17 score-related suites, 91 unit tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #69815, #69269
Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself on branch-4.1:
Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.
#69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard
sum the BM25 of every expanded term, i.e. scoring_boolean always on,
with idf and avgdl taken from the current segment rather than the tablet
statistics MATCH uses, so identical rows in two segments scored
differently. MATCH_REGEXP never requested a scoring context, so it
scored 0 like a filter.
Fix, the same as #69815 on master:
scorer again; its suite that pinned the BM25 sums is removed.
RegexpQuery records 1 for every matching row when score() is
requested. The MATCH statistics collector skips it, since a constant
reads no statistics and the pattern is not text to analyze.
A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the
keyword reader requests scoring (#69791); until then it keeps 0.
Release note
score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP on a tokenized index is 1 instead of 0.
Check List (For Author)
(V2/V3, two segments); all 18 blocks equal the master PR's V2/V3
blocks; 16 existing score suites in inverted_index_p0 and search
pass unchanged
RegexpQueryTest and the query_v2 regexp/wildcard tests restored by
the revert)
table now scores 1 for every pattern hit on a branch-4.1 ASAN build
scores change as described in the release note.
🤖 Generated with Claude Code