Repository navigation
branch-4.1: [fix](be) Support BM25 scoring on keyword SEARCH indexes #69051 - #69159
Merged
Merged
Conversation
### What problem does this PR solve? Related PR: apache#69051 Problem Summary: Backport keyword SEARCH scoring to branch-4.1. Use unit document length when legacy V2/V3 indexes lack norms and total term frequency, preserve complete keyword values, and keep WAND bounds conservative. Select matching query/statistics readers for plain dual-index fields. Reject numeric BKD scoring. The target branch has no SNII implementation. Adapt the collector to the branch's TermInfo and index-selection interfaces without importing master refactors. ### Release note SEARCH queries on legacy keyword indexes can return valid BM25 scores. ### Check List (For Author) - Test: Includes targeted unit and V2/V3 regression coverage; execution results are recorded in the pull request. - Behavior changed: Yes, enable keyword BM25 scoring. - Does this need documentation: No. Source-PR: apache#69051 Source-Head: f22f161
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
airborne12
marked this pull request as ready for review
October 16, 2026 03:46
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Oct 16, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Oct 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#69159 Problem Summary: The raw SEARCH score suite requires expression pushdown and inverted-index execution. Fuzzy sessions can disable expression pushdown, leaving SEARCH to row evaluation and causing an internal error. Set both options explicitly before the suite runs. ### Release note None ### Check List (For Author) - Test: The identical suite file passed CloudP0 in apache#69160 (build 1072387); git diff --check passed. CI for this PR is pending. - Behavior changed: No. Test setup only. - Does this need documentation: No.
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Oct 17, 2026
### What problem does this PR solve? Issue Number: DORIS-29489 Problem Summary: `SEARCH` with `score()` over a VARIANT sub-column fails on this branch with ``` ERROR 1105 (HY000): errCode = 2, detailMessage = Index statistics collection failed: Not such column 2.v.content ``` `SearchPredicateCollector::collect_from_leaf` resolved the clause's field name only through `tablet_schema->field_index(field_name)`. A VARIANT path such as `v.content` is not a tablet-schema column while the path is not materialized, so the clause returned early and the collection statistics were gathered for no field at all. The read side then asks the statistics for the same path (`<parent uid>.<parent>.<path>`, the Lucene field the index files were written with) and throws `INVERTED_INDEX_CLUCENE_ERROR` as `Not such column`. `MatchPredicateCollector` already resolves the same shapes from schema: materialized/typed paths, `field_pattern` templates, and a plain parent index inherited by the dynamic VARIANT placeholder. This PR moves that resolution into one helper (`resolve_scoring_index_candidates`) and uses it from the SEARCH collector too. When the clause field is not a tablet-schema column, the collector now builds the same synthetic VARIANT column the scan uses for a dynamic path (parent unique id + full path) and resolves it through the parent index, keeping the cloned index metadata alive through `CollectInfo::owned_index_meta`. Unresolvable names keep the previous skip, so only VARIANT paths gain collection. The helper also carries the leaf type of the resolved shape, so the reader-selection and BKD rejection this branch gained from the apache#69051 backport (apache#69159) keep working for plain fields and for an inherited VARIANT parent index, which must be scored as text rather than as a BKD leaf. ### Release note `score()` now works over `SEARCH` on VARIANT sub-columns served by a parent VARIANT inverted index. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Unit Test `CollectionStatisticsTest.SearchCollectsVariantParentIndexStatistics` covers the collector resolution; run through `run-be-ut.sh`. - [x] Regression test `regression-test/suites/search/test_search_variant_subcolumn_bm25.groovy` runs the reported query plus the phrase and `field_pattern` variants. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why below to give a good reason - Behavior changed: Yes (SEARCH + score() on VARIANT sub-columns returns scored rows instead of failing with `Not such column`) - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69051
Problem Summary: Backport #69051 to
branch-4.1.Legacy keyword SEARCH indexes lack norms and total-term-frequency metadata. Scoring could use a zero average length or zero norm and return invalid scores. Preserve whole keyword values, use unit document length when metadata is absent, and keep WAND upper bounds conservative. Numeric BKD scoring remains unsupported.
Branch adaptations
Backport the final net change from f22f161. Keep the branch-4.1 collection-statistics and predicate-collector paths and TermInfo representation. Select the same reader for plain-field statistics and execution when keyword and analyzed indexes coexist; retain the branch-specific variant binding path. The branch has no SNII, so this PR includes V2/V3 coverage and omits SNII-only changes.
Release note
SEARCH on legacy keyword indexes can return valid BM25 scores.
Check List (For Author)
Validation
origin/branch-4.1(0993d5e5512b3d27649e6c44f38b13836e03ee97)../build.sh --be --fe -j48.test_search_score_raw_indexandtest_search_score_cache, each replayed on this backport with one suite executed and zero failures, fatal scripts, or skips.Validation completed on the committed backport head. The temporary test instances are shut down after the run.
The unit run selected relevant test sources and existing repository helpers in a temporary local CMake manifest. The original manifest has been restored and is not included in this PR.