Repository navigation
branch-4.1: [fix](be) Score regexp and wildcard search terms with BM25 #69052 - #69269
Merged
yiguolei merged 2 commits intoOct 20, 2026
Merged
Conversation
Backport apache#69052 through the existing query_v2 term scorers. Sum BM25 contributions for scored expansions and retain the unscored bitmap path. Add independent term-score assertions and V2/V3 SQL regression coverage. (cherry picked from commit 34a123b)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Oct 20, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
yiguolei
pushed a commit
that referenced
this pull request
Oct 30, 2026
…ant like Elasticsearch (#69825) ### What problem does this PR solve? Issue Number: None Related PR: #69815, #69269 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself on branch-4.1: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. #69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, with idf and avgdl taken from the current segment rather than the tablet statistics MATCH uses, so identical rows in two segments scored differently. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix, the same as #69815 on master: 1. Revert #69269: RegexpWeight scores every pattern with the constant scorer again; its suite that pinned the BM25 sums is removed. 2. MATCH_REGEXP requests scoring on every index, and the legacy RegexpQuery records 1 for every matching row when score() is requested. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze. A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the keyword reader requests scoring (#69791); until then it keeps 0. ### Release note score() of SEARCH regexp and wildcard clauses is a constant 1 per matching row again, like prefix clauses and Elasticsearch's default. score() of MATCH_REGEXP on a tokenized index is 1 instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new search/test_search_score_pattern_constant (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3 blocks; 16 existing score suites in inverted_index_p0 and search pass unchanged - Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*, RegexpQueryTest and the query_v2 regexp/wildcard tests restored by the revert) - Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment table now scores 1 for every pattern hit on a branch-4.1 ASAN build - Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP scores change as described in the release note. - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69052
Backport regexp and wildcard SEARCH scoring to
branch-4.1. Scored expansions currently return constant scores. The fix uses existingTermScorerinstances to compute BM25 for each expanded term and sums the matching contributions. Unscored queries retain the bitmap path.This uses the compatible original change from
34a123b85da6ffc4990157820211dd08462744ecwith the branch'squery_v2interfaces. Legacy indexes without total-term-frequency metadata retain the unit-average-length fallback. The backport also adds independent term-score assertions and SQL regression coverage.Release note
Scored SEARCH regexp and wildcard expansions use BM25 relevance scores.
Validation
Tested head:
07de3d3acc1adeddae8ba85c6da295aa047f0b6f, including currentbranch-4.1basec7f95adaaaec4bc0caeab9ee831a9009745d56d0.TermQuerycontributions.Additional static-check limit: clang-tidy 16 cannot parse the compiler-20 JNI assertions; clang-tidy 20 is blocked by an unmatched NOLINTEND in the unchanged baseline. An overall clang-tidy pass is not claimed.
Check List (For Author)
Check List (For Reviewer who merge this PR)