Repository navigation
[fix](nereids) Support minimum score filters without TopN - #69292
Conversation
### What problem does this PR solve? Problem Summary: A WHERE-only score() predicate misses the score TopN rewrite and can evaluate the zero-valued scalar placeholder. Bind supported minimum score predicates to the existing score virtual column and full-scoring mode. Retain hidden score slots during physical translation and preserve projection, sorting, LIMIT, and OFFSET behavior. Share validation with score TopN, reject lossy score casts and raw aggregate score expressions, and support SEARCH SQL rendering during DEBUG plan display. ### Release note Support one score() > literal or score() >= literal filter with MATCH or SEARCH without requiring ORDER BY or LIMIT. FLOAT and DOUBLE casts remain supported; lossy score casts and unsupported score contexts fail analysis. Existing index scoring formulas and norm encodings are preserved. ### Check List (For Author) - Test: 141 FE unit tests; six score regression suites, including 146 matrix result groups; V2/SNII MATCH and SEARCH probes consuming 50,000 scored rows each from 100,000 documents. - Behavior changed: Yes. Minimum score filtering no longer requires SQL TopN; unsupported score usage fails analysis. - Does this need documentation: Yes. Standalone minimum score filtering should be documented; no documentation PR has been opened.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed all 11 changed paths at bd79ec1 using the supplied authoritative diff, the repository review skill and required guides, plus independent FE, test/semantics and scoring-risk reviews.
Three verified new inline findings: one P1 where a projected MATCH changes the rows admitted by the minimum-score filter; one P2 where enabled virtual CSE leaves nested projected score() calls evaluating the zero placeholder; and one P2 where an existing p0 regression still expects the newly supported query to fail. There were no existing inline threads or independently confirmed existing P0/P1 comments to carry forward. Changes are requested for the P1 finding.
Critical checkpoint conclusions:
- Goal and proof: the ordinary minimum-score path materializes a hidden score slot and uses limit zero for full scoring; unit assertions and regression goldens cover it. The two virtual-expression interactions in the inline findings prevent complete correctness.
- Scope and reuse: the change is focused on five FE production files plus supporting tests/results, reusing existing scan scoring metadata and BE implementations.
- Concurrency and locks: no threads, lock order or concurrent mutable state are introduced. The projected-MATCH finding concerns score accumulation within one segment's shared evaluation context.
- Lifecycle and initialization: the existing query/segment scoring lifecycle is retained; no new static initialization dependencies or ownership cycles are introduced.
- Configuration: no configuration items are added; the CSE finding is reachable through an existing optional experimental session setting.
- Compatibility: existing optional score sort/range fields are reused, with no new function symbol, storage format or persistent schema. Mixed-version execution was not performed.
- Thrift/protobuf: neither directory changes, so cross-release field-ID comparison is inapplicable.
- Parallel paths: traced standalone range and score TopN, project and filter-only shapes, MATCH and SEARCH, CLucene V2/V3 and SNII, deletes/MoW, cache/WAND, ANN and lazy materialization. The CSE and projected-MATCH paths need fixes; lazy slot loss and user-output widening were ruled out by upstream guards and explicit sink outputs.
- Conditional checks and errors: top-level single minimum comparisons, reversed forms and FLOAT/DOUBLE casts have concrete validation paths; unsupported contexts and lossy casts raise analysis errors. The CSE interaction bypasses intended score binding.
- Test coverage: the added suites exercise boundaries, casts, no hits, NULL thresholds, multiple rowsets, cache/WAND, deletes, MoW, ordinary/multiple sorts, offsets and ANN. They omit both reported virtual-expression combinations, and the older test_bm25_score assertion needs updating.
- Changed expected results: all 146 matrix groups and seven usage groups were reviewed; paired range/TopN controls agree. Independent float32 BM25 arithmetic reproduced every distinct added score, including the intentional V2/V3 versus SNII long-norm difference. Golden generation was not rerun.
- Observability: existing explain score sort/range metadata, index profiles and explicit diagnostics support investigation; no additional logging or metrics is required for this FE extension.
- Transactions and persistence: no production transaction, EditLog, replay, failover or persistence path changes.
- Writes and atomicity: production changes are read/planning-only. Test inserts, updates and deletes exercise the existing visibility paths; no new write atomicity or crash-recovery mechanism is introduced.
- FE/BE transfer: traced virtual slots, score ordering, zero score limit and double threshold from logical scan through physical translation and OlapScanNode to ScoreRuntime and range materialization. No new protocol variable is added.
- Performance: full scoring is proportional to candidate rows as needed for a standalone threshold; eligible score TopN retains its bounded path. ANN metadata clearing uses scalar distance fallback, whose runtime cost was not benchmarked.
- Other issues: no further substantiated finding after the assigned coverage and missed-case rechecks. No additional user review focus was supplied.
This was a static-only review: no builds, Doris tests, cloud execution, mixed-version execution or performance benchmarks were run, and source files were not modified. Those validation limits do not represent unreviewed changed paths.
Review converged after 2 rounds: all three reviewers returned NO_NEW_VALUABLE_FINDINGS on the identical frozen findings, coverage is complete, and every candidate/risk has an evidence-based disposition. No unresolved review question remains within the static review scope.
Review converged after 2 round(s).
TPC-H: Total hot run time: 28375 ms |
TPC-DS: Total hot run time: 150241 ms |
ClickBench: Total hot run time: 24.51 s |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#69292 Problem Summary: Follow-up fixes for the standalone minimum score filter (score() > x without a score TopN): 1. Binding the score range replaced the scan's ANN TopN with an empty one, so WHERE content MATCH 'x' AND score() > 0.5 ORDER BY l2_distance_approximate() LIMIT k no longer used the ANN index and computed every distance by brute force. The scan applies the score range before the ANN search, so the binding now keeps the ANN order keys and limit. On a local cluster the profile of this query shows AnnIndexTopNSearchCnt=1 and no brute-force fallback, and the nearest row that fails the score filter is not returned. 2. score() has no input slots, so join rules treated score() > x like a constant. PushDownFilterThroughJoin copied it into both children, PushFilterInsideJoin turned it into a join condition and PushDownJoinOtherCondition then pushed it into the left child. A query with an IN or EXISTS subquery failed with a misleading "must contain MATCH" error, and an inner join silently filtered whichever table was on the left. The predicate now goes only into the child whose rows all reach the output of a left/right outer, semi or anti join, as ReorderJoin already does. Inner, cross and full joins keep it above the join, where CheckScoreUsage rejects it with a message that suggests a subquery; score() in a JOIN condition fails analysis. 3. test_bm25_score still expected an error for a score() > 0.5 filter without ORDER BY and LIMIT, which is now supported. It now checks that the row count matches the score TopN path. 4. EXPLAIN prints the pushed range (SCORE RANGE: score() > 0.5), because the predicate is no longer among the scan conjuncts. 5. The new filter rules are guarded by the cached containsType check, so filters without score() skip them. 6. SearchExpression.toSql keeps the options argument of search(dsl, options). ### Release note score() filters work with IN and EXISTS subqueries and on the preserved side of left or right outer joins. With inner, cross or full joins, or in a JOIN condition, they fail with a clear error instead of filtering an arbitrary table. MATCH queries with a score() filter and an ANN TopN keep using the ANN index. EXPLAIN shows the pushed score range. ### Check List (For Author) - Test: Unit Test and Regression test - FE unit tests: CheckScoreUsageTest, PushDownFilterThroughJoinTest, PushFilterInsideJoinTest, SearchExpressionTest, RewriteSearchToSlotsTest, PullUpProjectExprUnderTopNTest, CheckAnalysisTest, ReorderJoinTest (180 tests). - Regression tests on a local ASAN cluster: test_bm25_score, test_score_filter_usage, test_score_filter_matrix, test_bm25_score_range_filter, test_search_score_topn_predicates and six more score suites in inverted_index_p0, all 36 suites in search, ann_with_fulltext and ann_topn_small_candidate_fallback. - Behavior changed: Yes. score() filters above inner, cross or full joins and in JOIN conditions are rejected; outer, semi and anti joins keep the filter on their output side; EXPLAIN shows SCORE RANGE. - Does this need documentation: Yes, with the score() documentation for standalone minimum score filters. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
run buildall |
|
/review |
There was a problem hiding this comment.
Converged in 2 review rounds. All three reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged final comment set; no unresolved static candidate or changed-path coverage gap remains.
Reviewed all 20 changed paths at 9c71f1f using static analysis only. One new P2 finding is inline: raw score predicates can cross a window before the new context validation and alter window results.
Existing review context was independently revalidated and deduplicated:
- P1 #discussion_r4315061210 still applies: projected MATCH expressions contaminate the WHERE score. It remains an existing blocker and is not reposted.
- P2 #discussion_r4315061221 still applies: score-containing CSE virtual columns retain raw Score calls and evaluate the placeholder.
- P2 #discussion_r4315061234 is fixed at this head: the old expected-error regression became positive standalone/TopN count controls.
Critical checkpoint conclusions:
- Goal and proof: the ordinary single-scan path materializes standalone minimum scores, retains hidden score slots, and reuses the existing TopN/range machinery. New FE and regression tests exercise it; the new and existing findings prevent a clean correctness verdict.
- Scope and reuse: production changes are focused on ten FE files, using existing score metadata and BE machinery; no unrelated source changes found.
- Concurrency and locking: no threads, locks, or shared mutable planner state are added. Plans are per-query; scan expression contexts are cloned per segment. No new lock-order or heavy-under-lock path found.
- Lifecycle/static initialization: existing ScoreRuntime preparation/reader ownership is retained, and virtual aliases propagate to scan descriptors. No new resource ownership or cross-translation-unit static dependency.
- Configuration: no new setting or dynamic-update requirement.
- Compatibility: existing score sort/range protocol fields and shared local/cloud scan paths are reused; no persisted format or function ABI change. Cloud and mixed-version execution were not validated.
- Thrift/protobuf IDs: no gensrc definitions changed.
- Parallel paths: checked MATCH/SEARCH, existing TopN and standalone scoring, ordinary sorts/offsets, count/ANN consumers, outer/semi/anti versus inner/cross/full join placement, aggregates, CSE and window normalization. Window placement and known virtual-column interactions are the identified gaps.
- Conditions/errors: supported GT/GE and reversed comparisons, one top-level range, FLOAT/DOUBLE casts, unsupported operators/lossy casts, overflow and missing scoring contexts have explicit checks. Late operator-context validation is incomplete as described inline.
- Test coverage: inspected all four changed FE test files and all six changed regression suite/result files. Coverage includes V2/V3/SNII, null/no-hit/boundary cases, aliases/sorts, residual filters, cache/WAND, rowsets/deletes/MoW and ANN. Window placement, projected MATCH plus scoring, and score-containing CSE combinations need coverage.
- Test results: new output blocks are consistent with paired controls and format-specific norm differences; ordered result comparisons and expected-error patterns are used. Output-generation provenance and runtime correctness cannot be independently proven by this static review. No builds or tests were run, as instructed.
- Observability: SCORE RANGE in EXPLAIN preserves visibility of the removed predicate; no additional metrics requirement found.
- Transactions/persistence: no EditLog, replay, version acquisition, failover or persisted metadata changes.
- Writes/atomicity: no production data-write or publish/delete-bitmap mutation changes; existing visible-version/delete restrictions precede score materialization.
- FE-BE variables: no new wire variable. Score order, zero limit, range and existing ANN metadata reach the existing reader/ScoreRuntime consumers; hidden score retention was verified.
- Performance: zero limit deliberately computes all candidate scores with existing O(candidate rows) materialization. No additional substantiated performance regression found.
- Other/core invariants: no new version, transaction visibility or ownership defect established; the identified score-binding/predicate-movement defects violate result correctness and unsupported-context failure behavior.
User focus: no additional focus was supplied. Review coverage is static; no runtime, cloud, or mixed-version validation is claimed.
Review converged after 2 round(s).
Existing P0/P1 findings confirmed for this head: #69292 (comment)
### What problem does this PR solve? Issue Number: None Related PR: apache#69292 Problem Summary: Fixes for the remaining review findings on standalone minimum score filters (score() > x without a score TopN): 1. A MATCH in the select list is pushed into the scan as a virtual column. SegmentIterator::_apply_index_expr evaluated it with the same CollectionSimilarity as the WHERE predicates, so its BM25 score was added to every row's score before the range was applied. On a three-row table where the real score is 0.4700036, SELECT id, c1 MATCH_PHRASE 'apple' AS flag FROM t WHERE c1 MATCH 'apple' AND score() > 0.6 returned two rows instead of none. Projection-only index evaluation now runs without the collection similarity, as VCompoundPred already does for its branches; scoring queries never use the index result cache, so the projection's cached bitmaps cannot answer a scoring query. 2. With experimental_enable_virtual_slot_for_cse, PushDownVirtualColumnsIntoOlapScan could extract a repeated expression such as abs(score()) + 1 into its own virtual column. That column kept the score() placeholder, which always returns zero, so sin(abs(score()) + 1) returned sin(1). Expressions that contain score() are no longer extracted; their score-free parts still are. 3. score() has no input slot, so filter pushdown treats it as a constant. An outer WHERE score() > x over a windowed subquery was pushed below the window and changed row_number() of the remaining rows; aggregations and LIMIT behaved the same way. CheckAnalysis now fails such a filter unless its input only passes rows of a scan through projections without window functions, filters, sorts, unions or the preserved side of an outer, semi or anti join. ### Release note A score() filter above a window, an aggregation, a LIMIT or an inner, cross or full join fails with a clear error. A MATCH in the select list no longer changes which rows pass a score() filter. ### Check List (For Author) - Test: Unit Test and Regression test - BE unit test: SegmentIteratorApplyIndexExprTest, including a new case that checks projections cannot see the collection similarity. - FE unit tests: CheckScoreUsageTest, PushDownVirtualColumnsIntoOlapScanTest, PushDownMatchProjectionAsVirtualColumnTest, PushDownFilterThroughJoinTest, PushFilterInsideJoinTest, SearchExpressionTest, RewriteSearchToSlotsTest, PullUpProjectExprUnderTopNTest, CheckAnalysisTest, ReorderJoinTest. - Regression tests on a local ASAN cluster: the score suites in inverted_index_p0 (including test_match_projection_virtual_column), all search suites, ann_with_fulltext and ann_topn_small_candidate_fallback. - Behavior changed: Yes. score() filters above windows, aggregations, LIMIT and inner, cross or full joins are rejected; projected MATCH no longer adds to score(). - Does this need documentation: Yes, with the score() documentation for standalone minimum score filters. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
run buildall |
|
/review |
There was a problem hiding this comment.
One new P2 correctness finding: the new early score-context check accepts a CTE consumer without checking its producer. Post-analysis inlining can expose a window and allow the score filter to move below it, changing row numbers. The inline comment includes the reduced query, plan, production rewrite path, and fix direction.
Reviewed all 23 changed paths at head 230a892344681c6014ffc87ecb301e94764beada using the supplied merge-base diff. The existing projected-MATCH P1 (4315061210), CSE P2 (4315061221), stale regression P2 (4315061234), and direct derived-window P2 (4321752024) are addressed for their original triggering cases. The new inline describes a distinct CTE bypass in the newly added LogicalRelation branch; those existing findings are not reposted. No existing P0/P1 is independently confirmed still applicable, so the existing-blocking list is empty.
Critical checkpoint conclusions:
| Checkpoint | Conclusion |
|---|---|
| 1. Goal and evidence | Standalone minimum-score filters now bind to scan scoring without SQL TopN. Added plan/unit/regression coverage supports the intended shapes; the CTE-window context remains incorrect. |
| 2. Scope and clarity | Changes are focused on score validation, binding/materialization, projection scoring, SEARCH rendering, and corresponding tests. |
| 3. Concurrency and locks | Scoring context is segment-local, and expression evaluation is synchronous. No new threads, locks, heavy locked work, or lock-order issue found. |
| 4. Lifecycle and static initialization | The saved similarity shared_ptr survives the projection loop and DEFER restores it on every exit before score materialization. No new ownership cycle or cross-TU static dependency. |
| 5. Configuration | No configuration added. Tests exercise existing cache, WAND, CSE, and count settings. |
| 6. Compatibility | Existing optional score protocol fields and zero-limit full scoring are reused; no storage encoding or external symbol change. V2/V3/SNII retain their existing numerical differences. Mixed-version execution was not performed. |
| 7. Thrift/protobuf IDs | No gensrc definition changes; cross-branch field-ID comparison is inapplicable. |
| 8. Parallel paths | Traced MATCH/SEARCH, V2/V3/SNII, ordinary and score sorts/TopN, ANN, count, deletes/MoW, and shared local/cloud segment paths. No further verified path defect. |
| 9. Conditional checks | Error cases fail analysis and suppression has a concrete rationale. The new CTE acceptance condition is the reported exception. |
| 10. Tests | New Java and BE unit tests plus positive/negative SQL cover bindings, aliases/CSE, casts/boundaries, sort/offset, joins/UNION, counts, cache/WAND, deletes/MoW, and ANN. CTE-window coverage is missing. No tests or builds were run in this review. |
| 11. Expected results | Statically inspected all changed outputs: 146 matrix groups, 19 usage groups, 30 matching standalone/TopN control pairs, boundary behavior, format differences, and ANN/projection scores. No incorrect expected result found; generation provenance cannot be proved statically. |
| 12. Observability | EXPLAIN now retains the removed score-range predicate. Existing scan/profile counters remain usable; no additional metric need identified. |
| 13. Transactions and persistence | No production EditLog, metadata persistence, version publication, or failover path changes. |
| 14. Writes and crash behavior | No production write/atomicity path changes. Regression DML exercises existing visibility. The temporary similarity state restores on failure. |
| 15. FE-to-BE propagation | Logical/physical scans forward score order, zero limit, range, and ANN metadata. Translation preserves the hidden score slot through tuple pruning; lazy-materialization alternatives were also inspected. |
| 16. Performance and memory | Full scoring is required for minimum-score semantics; projection suppression avoids redundant BM25 collection. No additional significant allocation, redundant copy, or hot-path regression established. |
| 17. Additional concerns | Further sweeps covered post-analysis CTE/alias expansion, outer-join simplification, mark/semi/anti joins, compound scoring, caches, lazy slots, and ordinary LIMIT timing. No further independent finding remains unresolved. |
No additional user focus was supplied. The release note describes the behavior change; the PR description acknowledges that the documentation PR is still pending.
Validation limits: static analysis only under the review contract; no builds, Doris test execution, fault injection, cloud deployment, or rolling-upgrade execution. No changed path, candidate, or risk focus remains without a disposition.
Review converged in 2 rounds: all three reviewers returned NO_NEW_VALUABLE_FINDINGS on the same frozen finding set; all candidates and risk focuses have evidence-backed dispositions.
Review converged after 2 round(s).
### What problem does this PR solve? Issue Number: None Related PR: apache#69292 Problem Summary: The analysis check for score() filters accepted a CTE consumer as a plain relation. CTEInline expands a single-use CTE after analysis, so for WITH w AS (SELECT id, row_number() OVER (ORDER BY id) AS rn FROM t WHERE c1 MATCH 'apple') SELECT * FROM w WHERE score() > 0.5 the window only became visible after the check, and filter pushdown moved score() > 0.5 below it, which changed rn of the remaining row. The check now follows a CTE consumer to its analyzed producer and applies the same row rules there. A recursive CTE reads itself and is rejected. ### Release note None ### Check List (For Author) - Test: Unit Test and Regression test - FE unit tests: CheckScoreUsageTest (a CTE over a window is rejected, a CTE over one scan is bound) and the related rule tests. - Regression tests on a local ASAN cluster: test_score_filter_usage, the score suites in inverted_index_p0, all search suites, ann_with_fulltext and ann_topn_small_candidate_fallback. - Behavior changed: Yes. A score() filter on a CTE whose producer contains a window, an aggregation, a LIMIT or an inner join is rejected. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
run buildall |
|
/review |
|
PR approved by at least one committer and no changes requested. |
…#69399 (#69560) ### What problem does this PR solve? Related PR: #69399 Problem Summary: Backport #69399 to `branch-4.2`. A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by `score()` or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and `LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in the output. #### Branch adaptations The branch has no table streams, so the change to `LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the constructor call of the branch, `LogicalPlanDeepCopier` imports `LogicalOlapScan`, which master already imported, and `LogicalPlanDeepCopierTest` keeps the tests of the branch and adds the two tests of #69399. ### Release note Fix "A expression contains slot not from children" for a CTE that orders by `score()` or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] No. - Does this need documentation? - [x] No. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.2` at ecbc5fe), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest`, `PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280 tests), and FE Checkstyle. - PASS: `./build.sh --fe --be`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 23 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`, with unchanged `.out` files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
…69292 (#69562) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit of #69560, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69560 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps the behavior of the branch and skips the push-down instead of failing. - `SearchDslParser` is imported from its package on the branch. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.2` at ecbc5fe), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest`, `PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280 tests), and FE Checkstyle. - PASS: `./build.sh --fe --be`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 23 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`, with unchanged `.out` files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
…cross column equalities #69419 (#69561) ### What problem does this PR solve? Related PR: #69419 Problem Summary: Backport #69419 to `branch-4.2`. Predicate inference copied a predicate on one column to every column that is equal to it, also when the predicate contained MATCH, multi_match or SEARCH. Their result depends on the inverted index of the column, not only on its value: with `c1 = c2` and different parsers on the two columns, the copy removed matching rows, and with the same index it doubled `score()`, so a minimum score let low-scored rows through. `InferPredicateByReplace.infer` now skips every input that contains `Match`, `MultiMatch`, `Search` or `SearchExpression`. #### Branch adaptations The branch does not have the `NoneMovableFunction` check that master has next to the new check, so only the new check is added. The test imports `SearchDslParser` from its package on the branch. ### Release note Fix wrong results and scores when predicate inference copied a MATCH, multi_match or SEARCH predicate to an equal column. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. Predicate inference no longer copies a predicate that contains MATCH, multi_match or SEARCH to an equal column. - Does this need documentation? - [x] No. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.2` at ecbc5fe), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest`, `PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280 tests), and FE Checkstyle. - PASS: `./build.sh --fe --be`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 23 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`, with unchanged `.out` files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve? Related PR: apache#69292 Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at random for each session. With it off, MATCH is not evaluated with the inverted index in the scan, so no BM25 score is collected and score() returns 0. test_score_filter_usage and test_score_filter_matrix got no rows for a minimum score, and test_cte_inline_virtual_column got score 0.0 for its TopN. Like the score suites of the branch, these suites set enable_common_expr_pushdown = true. ### Release note None ### Check List (For Author) - Test: Regression test. With use_fuzzy_session_variable = true and the pull request number of CI, the three suites failed in sessions that drew enable_common_expr_pushdown = false (1 to 3 suites per run); with the setting pinned, four runs passed. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
…#69399 (#69563) ### What problem does this PR solve? Related PR: #69399 Problem Summary: Backport #69399 to `branch-4.1`. A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by `score()` or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and `LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in the output. #### Branch adaptations The branch has no table streams, so the change to `LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the constructor call of the branch, and `LogicalPlanDeepCopierTest` and `CTEInlineTest` keep the tests of the branch and add the tests of #69399. `test_cte_inline_virtual_column` sets `enable_common_expr_pushdown = true`, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and `score()` is 0. ### Release note Fix "A expression contains slot not from children" for a CTE that orders by `score()` or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] No. - Does this need documentation? - [x] No. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.1` at 3f911d0), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE Checkstyle. - PASS: `./build.sh --be` and `./build.sh --fe`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 16 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`. The framework regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other `.out` files are unchanged. - PASS: with the fuzzy session variables of CI (`use_fuzzy_session_variable = true` and its pull request number), the score suites failed in sessions that drew `enable_common_expr_pushdown = false`; with the setting pinned, four runs of `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column`, and four of `infer_match_predicate`, passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
…cross column equalities #69419 (#69564) ### What problem does this PR solve? Related PR: #69419 Problem Summary: Backport #69419 to `branch-4.1`. Predicate inference copied a predicate on one column to every column that is equal to it, also when the predicate contained MATCH, multi_match or SEARCH. Their result depends on the inverted index of the column, not only on its value: with `c1 = c2` and different parsers on the two columns, the copy removed matching rows, and with the same index it doubled `score()`, so a minimum score let low-scored rows through. `InferPredicateByReplace.infer` now skips every input that contains `Match`, `MultiMatch`, `Search` or `SearchExpression`. #### Branch adaptations The branch does not have the `NoneMovableFunction` check that master has next to the new check, so only the new check is added. The test imports `SearchDslParser` from its package on the branch. `infer_match_predicate` sets `enable_common_expr_pushdown = true`, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and `score()` is 0. ### Release note Fix wrong results and scores when predicate inference copied a MATCH, multi_match or SEARCH predicate to an equal column. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. Predicate inference no longer copies a predicate that contains MATCH, multi_match or SEARCH to an equal column. - Does this need documentation? - [x] No. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.1` at 3f911d0), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE Checkstyle. - PASS: `./build.sh --be` and `./build.sh --fe`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 16 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`. The framework regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other `.out` files are unchanged. - PASS: with the fuzzy session variables of CI (`use_fuzzy_session_variable = true` and its pull request number), the score suites failed in sessions that drew `enable_common_expr_pushdown = false`; with the setting pinned, four runs of `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column`, and four of `infer_match_predicate`, passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
…69292 (#69565) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit of #69563, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69563 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - The branch evaluates MATCH projections with `evaluate_inverted_index` directly, and that loop runs without the score collector. - `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch for `limit + offset`, which fails like master does. - `SearchDslParser` and `CollectionSimilarity` are included from their packages on the branch. - The branch has no SNII storage format, so `BindScoreToOlapScanTest` and `test_score_filter_matrix` cover V2 and V3, and the `.out` of `test_score_filter_matrix` is regenerated without the SNII cases. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. - `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column` set `enable_common_expr_pushdown = true`, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and `score()` is 0. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.1` at 3f911d0), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE Checkstyle. - PASS: `./build.sh --be` and `./build.sh --fe`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 16 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`. The framework regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other `.out` files are unchanged. - PASS: with the fuzzy session variables of CI (`use_fuzzy_session_variable = true` and its pull request number), the score suites failed in sessions that drew `enable_common_expr_pushdown = false`; with the setting pinned, four runs of `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column`, and four of `infer_match_predicate`, passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
### Background SEARCH fields are bound during analysis before score binding runs in rewrite. Bound SEARCH predicates use SearchExpression instead of the scalar Search function. ### What problem does this PR solve? Issue Number: None Related PR: apache#69344 (early SEARCH binding), apache#69292 (score binding) PR Type: Bugfix Problem Summary: Valid SEARCH queries that call score() are rejected because score binding only recognizes the unbound Search type. Recognize SearchExpression in the WHERE predicate check, scoring alias propagation and HAVING/QUALIFY validation. ### Details - **Frequency:** SEARCH scoring queries after early SEARCH binding. - **Root cause:** The downstream score binder checks the expression type from the previous rule ordering. - **Why existing tests missed it:** The targeted SEARCH suites did not cover score(); existing score regression suites expose the failure. - **Fix:** Replace all three Search type checks with SearchExpression and update the import. - **Other paths and branches:** MATCH recognition and score usage restrictions remain unchanged. - **Behavior change:** Valid SEARCH scoring queries can pass score binding. ### Testing FE Checkstyle and git diff --check passed. Built FE with DISABLE_BUILD_UI=ON ./build.sh --fe and deployed it to the development cluster. The five targeted FE test classes executed 147 tests with zero failures, errors or skips, including 82 BindScoreToOlapScanTest cases. Ran 21 targeted regression suites: 20 passed, one failed because the existing BE lacks the keyword MATCH scoring change from apache#69790. Its actual 0.290716 score matches the BE-version expected output, while the current expected output is 0.983863. Re-running that suite against the existing generated expected output from the running BE commit passed. Repository expected outputs were not changed. The previous SEARCH score binding error was not reproduced. The initial local FE unit-test attempt stopped during dependency resolution due to Maven-cache sandbox permissions; remote tests provide the validation above. ### Release note Fix planning failures when using score() with SEARCH predicates. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: current keyword MATCH expected scores require a BE containing apache#69790; validation used the existing BE ### AI Disclosure - Tool: OpenAI Codex - Generated: Code and commit description - Human review: Pending
Doris 4.0.0 through 4.0.3 reject `score()` predicates in `WHERE`, while 4.0.4 and later support direct minimum-score filtering. Document both query forms in the English and Chinese dev/4.x relevance guides, with an indexed sample table and input data. Explain supported score predicates, threshold semantics, and the candidate limit required by the older-version workaround. Fixes #3264. The dev guides also describe the current implementation: an outermost single-table query can compute and filter scores without Top-N, while a scored SELECT nested in a subquery, view, CTE, or set operation still needs score ordering and LIMIT. Update the activation conditions and FAQ to match. The 4.x guides retain the Top-N requirement verified for 4.0.x and 4.1.0 through 4.1.5. Validation: - Compared the score-filter implementation in the 4.0.0 through 4.0.4 release tags and 4.1.5-release, checked the 4.0.4 release notes and [#60997](apache/doris#60997), and checked current master’s `BindScoreToOlapScan` and regression examples from [#69292](apache/doris#69292). - All four pages pass MDX parsing after legacy HTML-comment removal, front matter preservation, English/Chinese SQL example comparison, link checks, locale/version sync checks, and `git diff --check`. - No live cluster SQL or full website build was run. Existing page routes, front matter, and navigation are preserved. --------- Co-authored-by: morningman <[email protected]>
What problem does this PR solve?
Issue Number: N/A
Related PR: #69399 (merged; the plan copier fix this PR needs), #69419 (merged; fixes the pre-existing MATCH predicate inference that review of this PR found)
Problem Summary:
score()only worked asORDER BY score() LIMIT ndirectly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unboundscore()placeholder as zero and silently dropped every row:score()is computed by the olap scan that evaluates the MATCH and SEARCH predicates of its SELECT. This PR bindsscore()to that scan at the start of rewrite, makes the scan's expressions read the score, and rejects every other use with an error that names the rule.Supported. A SELECT that reads one table and has a MATCH or SEARCH predicate in its WHERE clause can use
score():score() > xorscore() >= x(alsox < score(),x <= score(), and with a FLOAT or DOUBLE cast);score()can only be the minimum score).A scored SELECT in a subquery, view, CTE or set operation still needs
ORDER BY score() LIMIT n, as before. The TopN keeps the predicates of the outer query out of its scan, e.g. an outer MATCH cannot add to its score. A correlated subquery cannot callscore(), because decorrelation removes that TopN.Rejected. A join, a subquery or a lateral view in the scored SELECT; a view, CTE or other relation that is not an olap table (a table stream fails during planning, because its scan is replaced by scans of the table and of its changes); any other filter on the score (
score() < x,score() = x, lossy casts, ORs, a second bound);score()in aggregates, GROUP BY, HAVING or QUALIFY; MATCH or SEARCH in HAVING or QUALIFY of a scored SELECT, also through an alias;score()in a correlated subquery; a SELECT without its own MATCH or SEARCH; a nested scored SELECT without the TopN.How.
SegmentIteratorprepares the score of every row of its row bitmap before the first batch. It marked only stored columns of a pushed-down expression as common-expression columns, so an expression on the score alone, e.g.score() > 0.5, was never evaluated (and the scanner had already removed it from its conjuncts), and an expression that read the score next to other columns, e.g. a TopN filter onCAST(score() AS INT), evaluatedScore()itself, which returns 0. Now the score column is read with the other common-expression columns, before the expressions run.BindScoreToOlapScanreplacesCheckScoreUsage. It runs first in rewrite, while subqueries, views and CTEs still have their own plan nodes, so the analysis classes are as on master. For each SELECT that callsscore(), it checks the rules above, adds a score column to the scan (score limit 0: the scan scores every row it returns) and replaces thescore()calls of the SELECT with that column. A table is a relation, also under the filters analysis puts on it (the delete sign filter of a UNIQUE table, row policies). Binding after analysis also covers* REPLACE (score() AS c), alias functions overscore()andORDER BYa position, and a UDF namedscoreis never mistaken for it.PushDownScoreTopNIntoOlapScanpushes a TopN on the score into the scan, and with it the minimum score, exactly as before this PR (with a zero score limit when SEARCH has another predicate), because the scan keeps the best rows of a segment only when no predicate follows. TopN filter pushdown and MV rewrite need noscore()cases: a TopN filter reads the score like any other column, and a materialized view without the score cannot answer a predicate on it.SegmentIteratorevaluates them without the score collector.SearchExpressioncan be rendered as SQL, which the shape plan logged at debug level andEXPLAIN SHAPE PLANneed.Release note
score()works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum scorescore() > xorscore() >= x, withoutORDER BY score() LIMIT. A scored SELECT in a subquery, view, CTE or set operation still needsORDER BY score() LIMIT. Other uses ofscore()are rejected instead of evaluating it as 0.Check List (For Author)
Test
Run locally on an ASAN build of the final commit, which merges master with [fix](nereids) Do not infer MATCH and SEARCH predicates across column equalities #69419:
BindScoreToOlapScanTest(80 cases: the supported shapes for V2, V3 and SNII with MATCH and SEARCH,*and* REPLACE, alias functions, one score column per SELECT, the main query of a WITH clause, nested TopNs ordered by alias or position, a UDF namedscore, and every rejected shape with its message),CTEInlineTest,LogicalPlanDeepCopierTest,TopNRuntimeFilterTest,PullUpProjectExprUnderTopNTest,MaterializedViewUtilsTest,SearchExpressionTest, and the predicate inference testsInferPredicateByReplaceTest,InferPredicatesTestandUnequalPredicateInferTest(317 tests). Checkstyle passes.SegmentIteratorScoreCommonExprTest(an expression on the score marks the score column as a common-expression column; another virtual column is still computed from its expression), withSegmentIteratorApplyIndexExprTestandSegmentIteratorCandidatePushdownTest(35 tests). The merge does not change the BE files of this PR, and master's BE unit tests do not compile at the moment (function_string_test.cpp, from [improvement](be) Optimize string functions and ARM byte checks #69223).score()ininverted_index_p0,inverted_index_p0/storage_format,search,ann_index_p0andnereids_rules_p0/cte,infer_match_predicatefrom [fix](nereids) Do not infer MATCH and SEARCH predicates across column equalities #69419, and the five suites from master that usescore()(test_bm25_score_parallel_scan_statistics,test_search_document_boolean_contract,test_search_null_array_payload_contract,test_variant_search_historical_rowset_indexandtest_variant_search_missing_doc_path), with unchanged.outfiles except those two cases.test_score_filter_usageandtest_score_filter_matrixcheck the new EXPLAIN output, andtest_score_filter_usagecovers a TopN filter onCAST(score() AS INT)over 20,000 rows read in many batches. Its.outwas regenerated by the framework for two new cases,select_star_replace_scoreandnested_score_topn_by_position.Cloud mode and mixed-version clusters were not tested.
Behavior changed:
score()withoutORDER BY score() LIMIT; unsupported uses ofscore()fail planning instead of evaluating it as 0. TopN filters on keys computed fromscore()now reach the scan. Without a TopN on the score, EXPLAIN shows the minimum score among the scan predicates instead ofSCORE RANGE. A minimum score without a TopN needs the BE change of this PR, so BEs must be upgraded before FEs, as usual; queries that master supports keep their plans.Does this need documentation?
score()documentation should describe wherescore()can be used. A documentation PR has not been opened.Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code