Skip to content

branch-4.1: [fix](nereids) Fix deep copy of scans with virtual columns #69399 - #69563

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:backport/69399-branch-4.1
Oct 26, 2026
Merged

yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:backport/69399-branch-4.1

Conversation

@airborne12

@airborne12 airborne12 commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to branch-4.1.

A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. LogicalPlanDeepCopier gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the output.

Branch adaptations

The branch has no table streams, so the change to LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the constructor call of the branch, and LogicalPlanDeepCopierTest and CTEInlineTest keep the tests of the branch and add the tests of #69399. test_cte_inline_virtual_column sets enable_common_expr_pushdown = true, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and score() is 0.

Release note

Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test
  • Behavior changed:
    • No.
  • Does this need documentation?
    • No.

Validation

Validated together with the other two backports (#69399, #69419 and #69292 on branch-4.1 at 3f911d0), in a local ASAN build:

  • PASS: FE unit tests LogicalPlanDeepCopierTest, CTEInlineTest, InferPredicateByReplaceTest, InferPredicatesTest, UnequalPredicateInferTest, BindScoreToOlapScanTest, SearchExpressionTest, TopNRuntimeFilterTest and MaterializedViewUtilsTest (228 tests), and FE Checkstyle.
  • PASS: ./build.sh --be and ./build.sh --fe.
  • PASS: BE unit tests SegmentIteratorScoreCommonExprTest, SegmentIteratorApplyIndexExprTest and VirtualSlotRefTest (31 tests); clang-format 16 on the changed BE files.
  • PASS: the 16 regression suites of the branch that use score(), including test_cte_inline_virtual_column, infer_match_predicate, test_score_filter_usage and test_score_filter_matrix. The framework regenerated the .out of test_score_filter_matrix on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other .out files are unchanged.
  • PASS: with the fuzzy session variables of CI (use_fuzzy_session_variable = true and its pull request number), the score suites failed in sessions that drew enable_common_expr_pushdown = false; with the setting pinned, four runs of test_score_filter_usage, test_score_filter_matrix and test_cte_inline_virtual_column, and four of infer_match_predicate, passed.

🤖 Generated with Claude Code

### What problem does this PR solve?

Related PR: apache#69399

Problem Summary: Backport apache#69399 to branch-4.1. A CTE that orders by
score() or by an ANN distance with a LIMIT, or that has MATCH in its
select list and is inlined after rewrite, failed with "A expression
contains slot not from children". LogicalPlanDeepCopier gave each virtual
column two ids, copied them in list order, kept the old slot in the
output of the copied scan and left the score and ANN order keys on the
old slots. The copier now maps every virtual column to its new id before
copying their expressions and remaps the order keys, and
LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the
output.

The branch has no table streams, so the change to
LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the
constructor call of the branch, and LogicalPlanDeepCopierTest and
CTEInlineTest keep the tests of the branch and add the tests of apache#69399.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by score() or by an ANN distance with a LIMIT, or that has MATCH in its
select list and is inlined after rewrite.

### Check List (For Author)

- Test: Includes the unit and regression tests of apache#69399; results are
  recorded in the pull request.
- Behavior changed: No.
- Does this need documentation: No.

Source-PR: apache#69399
Source-Head: 5be9ff5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Related PR: apache#69399

Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at
random for each session. With it off, MATCH is not evaluated with the
inverted index in the scan, so no BM25 score is collected and score()
returns 0. test_cte_inline_virtual_column got score 0.0 for its
TopN. Like the score suites of the branch, the suite sets
enable_common_expr_pushdown = true.

### Release note

None

### Check List (For Author)

- Test: Regression test. With use_fuzzy_session_variable = true and
the pull request number of CI, the suite failed in sessions that
drew enable_common_expr_pushdown = false; with the setting pinned,
four runs passed.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

@airborne12

Copy link
Copy Markdown
Member Author

run p0

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit 7af5d0d into apache:branch-4.1 Oct 26, 2026
28 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…69292 (#69565)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit
of #69563, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69563
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- The branch evaluates MATCH projections with `evaluate_inverted_index`
directly, and that loop runs without the score collector.
- `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch
for `limit + offset`, which fails like master does.
- `SearchDslParser` and `CollectionSimilarity` are included from their
packages on the branch.
- The branch has no SNII storage format, so `BindScoreToOlapScanTest`
and `test_score_filter_matrix` cover V2 and V3, and the `.out` of
`test_score_filter_matrix` is regenerated without the SNII cases.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.
- `test_score_filter_usage`, `test_score_filter_matrix` and
`test_cte_inline_virtual_column` set `enable_common_expr_pushdown =
true`, as the score suites of the branch do: CI on branch-4.1 draws it
at random for each session, and without it MATCH is not evaluated with
the index in the scan and `score()` is 0.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
@airborne12
airborne12 deleted the backport/69399-branch-4.1 branch October 26, 2026 10:35
@yujun777 yujun777 added dev/4.1.x usercase Important user case type label dev/4.1.6-merged and removed usercase Important user case type label dev/4.1.x labels Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants