Skip to content

branch-4.1: [fix](be) Support BM25 scoring on keyword SEARCH indexes #69051 - #69159

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
airborne12:backport/69051-branch-4.1
Oct 16, 2026
Merged

yiguolei merged 1 commit into
apache:branch-4.1from
airborne12:backport/69051-branch-4.1

Conversation

@airborne12

@airborne12 airborne12 commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #69051

Problem Summary: Backport #69051 to branch-4.1.

Legacy keyword SEARCH indexes lack norms and total-term-frequency metadata. Scoring could use a zero average length or zero norm and return invalid scores. Preserve whole keyword values, use unit document length when metadata is absent, and keep WAND upper bounds conservative. Numeric BKD scoring remains unsupported.

Branch adaptations

Backport the final net change from f22f161. Keep the branch-4.1 collection-statistics and predicate-collector paths and TermInfo representation. Select the same reader for plain-field statistics and execution when keyword and analyzed indexes coexist; retain the branch-specific variant binding path. The branch has no SNII, so this PR includes V2/V3 coverage and omits SNII-only changes.

Release note

SEARCH on legacy keyword indexes can return valid BM25 scores.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test
  • Behavior changed:
    • Yes. SEARCH on legacy keyword indexes can return valid BM25 scores.
  • Does this need documentation?
    • No.

Validation

  • PASS: Clang 16 across the complete candidate merged with origin/branch-4.1 (0993d5e5512b3d27649e6c44f38b13836e03ee97).
  • PASS: English code check for the complete PR diff.
  • PASS: 69 targeted ASAN BE unit tests from 5 suites on this backport.
  • PASS: ASAN BE/FE build with ./build.sh --be --fe -j48.
  • PASS: test_search_score_raw_index and test_search_score_cache, each replayed on this backport with one suite executed and zero failures, fatal scripts, or skips.
  • PASS: the BE binary requires GLIBC no newer than 2.10 (baseline 2.17).
  • PASS: the two backport commits merge together without conflicts.

Validation completed on the committed backport head. The temporary test instances are shut down after the run.

The unit run selected relevant test sources and existing repository helpers in a temporary local CMake manifest. The original manifest has been restored and is not included in this PR.

### What problem does this PR solve?

Related PR: apache#69051

Problem Summary: Backport keyword SEARCH scoring to branch-4.1. Use unit document length when legacy V2/V3 indexes lack norms and total term frequency, preserve complete keyword values, and keep WAND bounds conservative. Select matching query/statistics readers for plain dual-index fields. Reject numeric BKD scoring.

The target branch has no SNII implementation. Adapt the collector to the branch's TermInfo and index-selection interfaces without importing master refactors.

### Release note

SEARCH queries on legacy keyword indexes can return valid BM25 scores.

### Check List (For Author)

- Test: Includes targeted unit and V2/V3 regression coverage; execution results are recorded in the pull request.
- Behavior changed: Yes, enable keyword BM25 scoring.
- Does this need documentation: No.

Source-PR: apache#69051
Source-Head: f22f161
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
airborne12 marked this pull request as ready for review October 16, 2026 03:46
@airborne12
airborne12 requested a review from yiguolei as a code owner October 16, 2026 03:46
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 93.88% (46/49) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.49% (26983/43880)
Line Coverage 46.17% (281286/609254)
Region Coverage 41.81% (222609/532447)
Branch Coverage 43.33% (103402/238626)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (48/48) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.10% (32047/42670)
Line Coverage 59.48% (359457/604372)
Region Coverage 56.15% (299548/533504)
Branch Coverage 57.07% (136002/238327)

@yiguolei
yiguolei merged commit 9800bee into apache:branch-4.1 Oct 16, 2026
49 of 50 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 17, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69159

Problem Summary: The raw SEARCH score suite requires expression pushdown and inverted-index execution. Fuzzy sessions can disable expression pushdown, leaving SEARCH to row evaluation and causing an internal error. Set both options explicitly before the suite runs.

### Release note

None

### Check List (For Author)

- Test: The identical suite file passed CloudP0 in apache#69160 (build 1072387); git diff --check passed. CI for this PR is pending.
- Behavior changed: No. Test setup only.
- Does this need documentation: No.
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Oct 17, 2026
### What problem does this PR solve?

Issue Number: DORIS-29489

Problem Summary:

`SEARCH` with `score()` over a VARIANT sub-column fails on this branch with

```
ERROR 1105 (HY000): errCode = 2, detailMessage = Index statistics collection failed:
Not such column 2.v.content
```

`SearchPredicateCollector::collect_from_leaf` resolved the clause's field name only through
`tablet_schema->field_index(field_name)`. A VARIANT path such as `v.content` is not a
tablet-schema column while the path is not materialized, so the clause returned early and the
collection statistics were gathered for no field at all. The read side then asks the statistics
for the same path (`<parent uid>.<parent>.<path>`, the Lucene field the index files were written
with) and throws `INVERTED_INDEX_CLUCENE_ERROR` as `Not such column`.

`MatchPredicateCollector` already resolves the same shapes from schema: materialized/typed
paths, `field_pattern` templates, and a plain parent index inherited by the dynamic VARIANT
placeholder. This PR moves that resolution into one helper
(`resolve_scoring_index_candidates`) and uses it from the SEARCH collector too. When the clause
field is not a tablet-schema column, the collector now builds the same synthetic VARIANT column
the scan uses for a dynamic path (parent unique id + full path) and resolves it through the
parent index, keeping the cloned index metadata alive through `CollectInfo::owned_index_meta`.
Unresolvable names keep the previous skip, so only VARIANT paths gain collection.

The helper also carries the leaf type of the resolved shape, so the reader-selection and BKD
rejection this branch gained from the apache#69051 backport (apache#69159) keep working for plain fields and
for an inherited VARIANT parent index, which must be scored as text rather than as a BKD leaf.

### Release note

`score()` now works over `SEARCH` on VARIANT sub-columns served by a parent VARIANT inverted
index.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Unit Test
      `CollectionStatisticsTest.SearchCollectsVariantParentIndexStatistics` covers the collector
      resolution; run through `run-be-ut.sh`.
    - [x] Regression test
      `regression-test/suites/search/test_search_variant_subcolumn_bm25.groovy` runs the reported
      query plus the phrase and `field_pattern` variants.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why below to give a good reason
- Behavior changed: Yes (SEARCH + score() on VARIANT sub-columns returns scored rows instead of
  failing with `Not such column`)
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants