Skip to content

branch-4.1: [fix](be) Score regexp and wildcard search terms with BM25 #69052 - #69269

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:backport/69052-branch-4.1
Oct 20, 2026
Merged

yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:backport/69052-branch-4.1

Conversation

@airborne12

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #69052

Backport regexp and wildcard SEARCH scoring to branch-4.1. Scored expansions currently return constant scores. The fix uses existing TermScorer instances to compute BM25 for each expanded term and sums the matching contributions. Unscored queries retain the bitmap path.

This uses the compatible original change from 34a123b85da6ffc4990157820211dd08462744ec with the branch's query_v2 interfaces. Legacy indexes without total-term-frequency metadata retain the unit-average-length fallback. The backport also adds independent term-score assertions and SQL regression coverage.

Release note

Scored SEARCH regexp and wildcard expansions use BM25 relevance scores.

Validation

Tested head: 07de3d3acc1adeddae8ba85c6da295aa047f0b6f, including current branch-4.1 base c7f95adaaaec4bc0caeab9ee831a9009745d56d0.

  • ASAN UT: 636 tests from 36 related suites passed; zero failures, errors or skips. The two new expansion-scoring tests fail against the original production implementation and pass with this fix, including equality with independently collected TermQuery contributions.
  • Full ASAN and Release FE/BE builds completed. After synchronizing the newer Boolean-scoring base, both BE types were rebuilt and the unchanged FE artifacts were reused. Both binaries retain GLIBC compatibility (maximum required GLIBC 2.10); ASAN instrumentation was verified in the ASAN binary.
  • ASAN and Release each passed 16 related SEARCH/inverted-index suites, with zero failed suites, fatal scripts or skips. Coverage includes scoring/cache/TopN/range predicates, Boolean branches, regexp and array regexp, phrase/prefix/edge, custom analyzers and raw indexes.
  • Each binary passed 24 direct score-comparison SQL queries across V2/V3, standard/keyword indexes and WAND on/off. Regexp and wildcard scores match explicit expanded-term queries. The new regression has 48 framework-generated golden cases, replayed on both binaries, including NULL input, empty matches, ordering and unscored queries.
  • Runtime logs contain no sanitizer errors. All task-owned test instances were stopped. Full merged-tree clang-format 16 and English-description checks passed.

Additional static-check limit: clang-tidy 16 cannot parse the compiler-20 JNI assertions; clang-tidy 20 is blocked by an unmatched NOLINTEND in the unchanged baseline. An overall clang-tidy pass is not claimed.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (score-comparison matrix described above)
  • Behavior changed:
    • Yes. Scored regexp/wildcard expansions return BM25 contributions.
  • Does this need documentation?
    • No. No syntax or setting is added.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Backport apache#69052 through the existing query_v2 term scorers. Sum BM25
contributions for scored expansions and retain the unscored bitmap path.
Add independent term-score assertions and V2/V3 SQL regression coverage.

(cherry picked from commit 34a123b)
@airborne12
airborne12 requested a review from yiguolei as a code owner October 19, 2026 07:47
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 83.33% (30/36) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.10% (32049/42675)
Line Coverage 59.48% (359521/604475)
Region Coverage 56.13% (299504/533626)
Branch Coverage 57.05% (136022/238419)

@yiguolei
yiguolei merged commit 8cf932a into apache:branch-4.1 Oct 20, 2026
29 of 30 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Oct 30, 2026
…ant like Elasticsearch (#69825)

### What problem does this PR solve?

Issue Number: None

Related PR: #69815, #69269

Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself on branch-4.1:

    -- 'Inverted indexes are suitable for full text search',
    -- 'The parser splits text into multiple terms', ...
SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25
sum
SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum
    SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0

Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.

#69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard
sum the BM25 of every expanded term, i.e. scoring_boolean always on,
with idf and avgdl taken from the current segment rather than the tablet
statistics MATCH uses, so identical rows in two segments scored
differently. MATCH_REGEXP never requested a scoring context, so it
scored 0 like a filter.

Fix, the same as #69815 on master:
1. Revert #69269: RegexpWeight scores every pattern with the constant
   scorer again; its suite that pinned the BM25 sums is removed.
2. MATCH_REGEXP requests scoring on every index, and the legacy
   RegexpQuery records 1 for every matching row when score() is
   requested. The MATCH statistics collector skips it, since a constant
   reads no statistics and the pattern is not text to analyze.

A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the
keyword reader requests scoring (#69791); until then it keeps 0.

### Release note

score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP on a tokenized index is 1 instead of 0.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - Regression test: new search/test_search_score_pattern_constant
      (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3
      blocks; 16 existing score suites in inverted_index_p0 and search
      pass unchanged
- Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*,
      RegexpQueryTest and the query_v2 regexp/wildcard tests restored by
      the revert)
    - Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment
table now scores 1 for every pattern hit on a branch-4.1 ASAN build
- Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP
  scores change as described in the release note.
- Does this need documentation: No


🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants