Repository navigation
branch-4.1: [fix](nereids) Fix deep copy of scans with virtual columns #69399 - #69563
Merged
yiguolei merged 2 commits intoOct 26, 2026
Merged
Conversation
### What problem does this PR solve? Related PR: apache#69399 Problem Summary: Backport apache#69399 to branch-4.1. A CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite, failed with "A expression contains slot not from children". LogicalPlanDeepCopier gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the output. The branch has no table streams, so the change to LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the constructor call of the branch, and LogicalPlanDeepCopierTest and CTEInlineTest keep the tests of the branch and add the tests of apache#69399. ### Release note Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. ### Check List (For Author) - Test: Includes the unit and regression tests of apache#69399; results are recorded in the pull request. - Behavior changed: No. - Does this need documentation: No. Source-PR: apache#69399 Source-Head: 5be9ff5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
4 of 5 tasks
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Related PR: apache#69399 Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at random for each session. With it off, MATCH is not evaluated with the inverted index in the scan, so no BM25 score is collected and score() returns 0. test_cte_inline_virtual_column got score 0.0 for its TopN. Like the score suites of the branch, the suite sets enable_common_expr_pushdown = true. ### Release note None ### Check List (For Author) - Test: Regression test. With use_fuzzy_session_variable = true and the pull request number of CI, the suite failed in sessions that drew enable_common_expr_pushdown = false; with the setting pinned, four runs passed. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run p0 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Oct 26, 2026
yiguolei
pushed a commit
that referenced
this pull request
Oct 26, 2026
…69292 (#69565) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit of #69563, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69563 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - The branch evaluates MATCH projections with `evaluate_inverted_index` directly, and that loop runs without the score collector. - `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch for `limit + offset`, which fails like master does. - `SearchDslParser` and `CollectionSimilarity` are included from their packages on the branch. - The branch has no SNII storage format, so `BindScoreToOlapScanTest` and `test_score_filter_matrix` cover V2 and V3, and the `.out` of `test_score_filter_matrix` is regenerated without the SNII cases. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. - `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column` set `enable_common_expr_pushdown = true`, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and `score()` is 0. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.1` at 3f911d0), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE Checkstyle. - PASS: `./build.sh --be` and `./build.sh --fe`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 16 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`. The framework regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other `.out` files are unchanged. - PASS: with the fuzzy session variables of CI (`use_fuzzy_session_variable = true` and its pull request number), the score suites failed in sessions that drew `enable_common_expr_pushdown = false`; with the setting pinned, four runs of `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column`, and four of `infer_match_predicate`, passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69399
Problem Summary: Backport #69399 to
branch-4.1.A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.LogicalPlanDeepCopiergave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, andLogicalOlapScan.withVirtualColumnsreplaces the old virtual slots in the output.Branch adaptations
The branch has no table streams, so the change to
LogicalOlapTableStreamScanis dropped.withVirtualColumnskeeps the constructor call of the branch, andLogicalPlanDeepCopierTestandCTEInlineTestkeep the tests of the branch and add the tests of #69399.test_cte_inline_virtual_columnsetsenable_common_expr_pushdown = true, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan andscore()is 0.Release note
Fix "A expression contains slot not from children" for a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.Check List (For Author)
Validation
Validated together with the other two backports (#69399, #69419 and #69292 on
branch-4.1at 3f911d0), in a local ASAN build:LogicalPlanDeepCopierTest,CTEInlineTest,InferPredicateByReplaceTest,InferPredicatesTest,UnequalPredicateInferTest,BindScoreToOlapScanTest,SearchExpressionTest,TopNRuntimeFilterTestandMaterializedViewUtilsTest(228 tests), and FE Checkstyle../build.sh --beand./build.sh --fe.SegmentIteratorScoreCommonExprTest,SegmentIteratorApplyIndexExprTestandVirtualSlotRefTest(31 tests); clang-format 16 on the changed BE files.score(), includingtest_cte_inline_virtual_column,infer_match_predicate,test_score_filter_usageandtest_score_filter_matrix. The framework regenerated the.outoftest_score_filter_matrixon this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other.outfiles are unchanged.use_fuzzy_session_variable = trueand its pull request number), the score suites failed in sessions that drewenable_common_expr_pushdown = false; with the setting pinned, four runs oftest_score_filter_usage,test_score_filter_matrixandtest_cte_inline_virtual_column, and four ofinfer_match_predicate, passed.🤖 Generated with Claude Code