Skip to content

[fix](nereids) Fix deep copy of scans with virtual columns - #69399

Merged
airborne12 merged 2 commits into
apache:masterfrom
airborne12:fix-deep-copy-virtual-columns
Oct 22, 2026
Merged

airborne12 merged 2 commits into
apache:masterfrom
airborne12:fix-deep-copy-virtual-columns

Conversation

@airborne12

@airborne12 airborne12 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: N/A

Related PR: #69292 (depends on this fix)

Problem Summary:

A CTE fails to plan when the optimizer inlines it after one of its scans got a virtual column:

WITH c AS (
    SELECT id, score() AS s FROM t WHERE c1 MATCH 'apple'
    ORDER BY s DESC LIMIT 2
)
SELECT * FROM c;
-- ERROR: A expression contains slot not from children

A score or ANN TopN pushed into a scan, and a MATCH in the select list, add a virtual column to the scan. After rewrite, the optimizer inlines a CTE that has one consumer left (Optimizer.normalizeCtePlan → CTEInliner) by deep copying its rewritten body, so the copy includes these scans:

  • score() and l2_distance_approximate() are nondeterministic, so the rewrite does not inline a CTE that calls them, even with one consumer. Once the TopN pushdown has replaced them with a virtual column of the scan, the optimizer inlines the CTE. So every CTE with ORDER BY score() LIMIT n or an ANN TopN fails, with default settings.
  • A CTE with MATCH in its select list reaches the optimizer when the rewrite removes all but one of its consumers, e.g. ... UNION ALL SELECT * FROM c WHERE 1 = 0.

LogicalPlanDeepCopier copied a scan with virtual columns wrongly:

  • it gave each virtual column two new ids: the slots above the scan were mapped to the first one, and the copied scan output the second one;
  • it copied the virtual columns in list order, so a virtual column that refers to a later one (CSE with experimental_enable_virtual_slot_for_cse=true, e.g. [v2 := v1 * v1, v1 := x + y]) kept the old id of the later one;
  • the copied scan still output the original virtual column, with its original id, next to the new one;
  • the score and ANN order keys of the copied scan still used the original slots.

So the plan above the copied scan read a slot that the scan does not output.

The fix:

  • The copier gives every virtual column its new id first and then copies their expressions, which reuse those ids, so virtual columns may refer to each other in any order. It also maps the score and ANN order keys to the new slots. Relations without virtual columns are copied as before.
  • LogicalOlapScan.withVirtualColumns replaces the virtual columns of the scan, but kept their old slots in the output next to the new ones. It now replaces them there too and keeps the other output slots, which the copier has already mapped. Computing the output of the copy again would not work: it gives VARIANT subpath slots new ids. AdjustNullable, the other caller that replaces existing virtual columns, no longer keeps two slots for a virtual column whose nullability changes.

#69292 binds score() to its scan before CTEs are inlined, so it needs this fix; it carries the same change until this PR is merged.

Release note

Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.

Check List (For Author)

  • Test

    • Regression test

    • Unit Test

    • Manual test (add detailed scripts or steps below)

    • No need to test or manual test. Explain why:

      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
    • Unit tests: LogicalPlanDeepCopierTest.testDeepCopyOlapScanWithVirtualColumn, LogicalPlanDeepCopierTest.testDeepCopyOlapScanWithDependentVirtualColumns (virtual columns that refer to each other, in both orders) and CTEInlineTest.inlineCteWhoseScanHasVirtualColumns (CTEs with a score TopN, an ANN TopN, a MATCH projection, and a VARIANT subpath with a MATCH projection, with one consumer and with a second consumer removed by WHERE 1 = 0). They fail on master (4 output slots instead of 3; a copied virtual column that refers to the old one; A expression contains slot not from children) and pass with this change. OrExpansionTest, SplitMultiDistinctTest, DecomposeRepeatWithPreAggregationTest, LogicalOlapScanTest, IvmDeltaRewriteHelperTest and PushDownVirtualColumnsIntoOlapScanTest pass. Checkstyle passes.

    • Regression test: nereids_rules_p0/cte/test_cte_inline_virtual_column, with its .out generated by the framework, on an FE built from this branch and a BE built from 41a58af plus [fix](nereids) Support minimum score filters without TopN #69292's SegmentIterator change (this change is FE only). The VARIANT case is only in the unit test, because that BE reads v['x'] as NULL even without a CTE.

  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

🤖 Generated with Claude Code

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292

Problem Summary:

A CTE fails to plan when the optimizer inlines it after one of its scans got
a virtual column:

    WITH c AS (
        SELECT id, score() AS s FROM t WHERE c1 MATCH 'apple'
        ORDER BY s DESC LIMIT 2
    )
    SELECT * FROM c;
    -- A expression contains slot not from children

A score or ANN TopN pushed into a scan, and a MATCH in the select list, add a
virtual column to the scan. After rewrite, the optimizer inlines a CTE that
has one consumer left (Optimizer.normalizeCtePlan) by deep copying its
rewritten body, so the copy includes these scans:

- score() and l2_distance_approximate() are nondeterministic, so the rewrite
  does not inline a CTE that calls them, even with one consumer. Once the
  TopN pushdown has replaced them with a virtual column of the scan, the
  optimizer inlines the CTE.
- A CTE with MATCH in its select list reaches the optimizer when the rewrite
  removes all but one of its consumers, e.g. a UNION ALL branch with
  WHERE 1 = 0.

LogicalPlanDeepCopier copied a scan with virtual columns wrongly:

- it gave each virtual column two new ids: the slots above the scan were
  mapped to the first one, and the copied scan output the second one;
- the copied scan still output the original virtual column, with its
  original id, next to the new one;
- the score and ANN order keys of the copied scan still used the original
  slots.

So the plan above the copied scan read a slot that the scan does not output.

The copier now gives each virtual column one new id, maps the original id to
it and maps the score and ANN order keys to the new slots. Relations without
virtual columns are copied as before.

LogicalOlapScan.withVirtualColumns replaces the virtual columns of the scan,
but kept their old slots in the output next to the new ones. It now replaces
them there too and keeps the other output slots, which the copier has
already mapped. Computing the output of the copy again would not work: it
gives VARIANT subpath slots new ids. AdjustNullable, the other caller that
replaces existing virtual columns, no longer keeps two slots for a virtual
column whose nullability changes.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders by
score() or by an ANN distance with a LIMIT, or that has MATCH in its select
list and is inlined after rewrite.

### Check List (For Author)

- Test: Unit Test and Regression test
    - Unit tests: LogicalPlanDeepCopierTest.testDeepCopyOlapScanWithVirtualColumn
      and CTEInlineTest.inlineCteWhoseScanHasVirtualColumns (CTEs with a score
      TopN, an ANN TopN, a MATCH projection, and a VARIANT subpath with a MATCH
      projection, with one consumer and with a second consumer removed by
      WHERE 1 = 0). Both fail on master (4 output slots instead of 3; "A
      expression contains slot not from children") and pass with this change.
      OrExpansionTest, SplitMultiDistinctTest,
      DecomposeRepeatWithPreAggregationTest, LogicalOlapScanTest,
      IvmDeltaRewriteHelperTest and PushDownVirtualColumnsIntoOlapScanTest
      pass. Checkstyle passes.
    - Regression test: nereids_rules_p0/cte/test_cte_inline_virtual_column,
      with its .out generated by the framework, on an FE built from this
      branch and a BE built from 41a58af plus apache#69292's SegmentIterator
      change (this change is FE only). The VARIANT case is only in the unit
      test, because that BE reads v['x'] as NULL even without a CTE.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 21, 2026
…ANT slots in scan copies

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292, apache#69399

Problem Summary:

Review of 8fc7d7c found two problems:

- A minimum score under GROUPING SETS, ROLLUP or CUBE, e.g.
  SELECT c2, count(*) FROM t WHERE c1 MATCH 'apple' AND score() > 0.00001
  GROUP BY GROUPING SETS ((c2), ()), was rejected as an unsupported SELECT:
  the binder did not walk through the LogicalRepeat that analysis puts below
  the aggregate. It now does, so the minimum score works as with any other
  GROUP BY, and score() in the select list of such a SELECT gets the
  aggregate error.
- The plan copier computed the output of a copied scan again after replacing
  its virtual columns. That gives VARIANT subpath slots new ids, which the
  copier had already mapped, so a CTE that reads v['x'] and has a MATCH in
  its select list failed with "A expression contains slot not from children"
  when the optimizer inlined it. The copier fix is now the one of apache#69399:
  LogicalOlapScan.withVirtualColumns replaces the slots of the old virtual
  columns in the output and keeps the other slots, and the copier no longer
  computes the output again. This PR carries the same change until apache#69399 is
  merged.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression test
    - BindScoreToOlapScanTest: GROUPING SETS, ROLLUP and CUBE with a minimum
      score, and max(score()) with ROLLUP; all 73 tests pass. The copier
      tests are the ones of apache#69399, whose CTEInlineTest covers the VARIANT
      case. LogicalPlanDeepCopierTest, SearchCteRewriteTest,
      PushDownVirtualColumnsIntoOlapScanTest, PhysicalPlanTranslatorTest,
      PushDownMatchProjectionAsVirtualColumnTest, CTEInlineTest,
      LogicalOlapScanTest and SearchExpressionTest pass (157 tests with
      BindScoreToOlapScanTest). Checkstyle passes.
    - Regression tests: test_score_filter_usage gets a GROUPING SETS case,
      with its .out regenerated by the framework. The 19 suites that use
      score() pass on a private cluster.
- Behavior changed: Yes. A minimum score is supported with GROUPING SETS,
  ROLLUP and CUBE.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all seven changed files at be40091 and traced virtual-column production, early/late/CBO CTE inlining, other deep-copy callers, scan replacement, and physical translation. No newly introduced issue was verified. There are no existing inline findings to carry forward and no additional user focus points.

Critical checkpoint conclusions:

  • Goal and evidence: the new mapping/output/order-key handling repairs the stated score TopN, ANN TopN and MATCH projection cases. Added unit and planner tests exercise these cases, late inlining, and VARIANT subpaths; execution was not performed in this review.
  • Scope and reuse: changes are focused; both scan classes share output replacement and TopN copying reuses the order-key helper.
  • Concurrency and locking: no new threads or shared mutable execution state; copy context remains local. No new lock order or heavy work under locks.
  • Lifecycle: existing planner object ownership is preserved; no new resources, cleanup obligations, or static initialization dependency.
  • Configuration: no configuration is added or changed.
  • Compatibility: no persisted format, public wire contract, or rolling-upgrade behavior changes.
  • Thrift/protobuf: no definitions changed.
  • Parallel paths: regular/stream scans, no-virtual scans, score/ANN/MATCH/CSE producers, nullability adjustment, operative slots and other copier callers were checked. Stream subtype and read mode are preserved.
  • Conditions: the nonempty-virtual guard skips unnecessary reconstruction; replacement removes the current virtual ids while retaining ordinary/VARIANT slot identities.
  • Test coverage: direct id/order-key/output invariants and eight planner query shapes are covered; regression cases use ordered output and preserve tables for debugging. Stream/nullability edge cases and dependent CSE aliases lack new dedicated tests.
  • Expected results: the three new result sets agree with BM25 arithmetic, distinct ANN distances and MATCH truth values. The author reports framework generation using this FE and a separate BE revision plus a related patch; that patch does not affect these three query shapes. Exact-head BE execution and output generation were not independently reproduced.
  • Observability: existing planning diagnostics suffice for this local identity repair; no new distributed operation needs metrics or logging.
  • Transactions and persistence: no transaction, EditLog, replay, or failover path changes.
  • Data writes and crashes: no production data-write or atomicity behavior changes.
  • FE-BE variables: existing virtual-column and TopN metadata is forwarded; no new transmitted variable or send path.
  • Performance: output replacement is linear in output plus virtual-column count; no material regression was substantiated.
  • Other issues and limits: dependent CSE virtual aliases can still retain a forward dependency on an old id with experimental_enable_virtual_slot_for_cse enabled. The baseline already fails the same case through its intermediate/final alias-id mismatch, so this is recorded as a pre-existing limitation, with no new inline finding attributed to this PR. No unresolved introduced-behavior candidate remains.

Validation was static only, as required: no builds, tests, or source modifications.

Review converged after 2 round(s).

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed be40091. The independent score/ANN/MATCH remapping and output replacement look correct, but the inline comment describes a remaining dependent-virtual-column case in the copier. Source review only; I did not run builds or tests.

…them

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69399

Problem Summary:

With experimental_enable_virtual_slot_for_cse=true, a virtual column can
refer to another one, and PushDownVirtualColumnsIntoOlapScan does not list
them in dependency order, e.g. [v2 := v1 * v1, v1 := x + y]. The copier
copied the virtual columns in list order. When it copied v2, the id of v1
still mapped to itself, so the copy of v2 referred to the old v1, which the
copied scan does not output. The copier now maps every virtual column to its
new id first and then copies their expressions, which reuse those ids.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - LogicalPlanDeepCopierTest.testDeepCopyOlapScanWithDependentVirtualColumns
      copies a scan whose virtual columns refer to each other, in both orders.
      It fails with the previous commit, where "(v1#10002 * v1#10002) AS v2"
      refers to the old v1, and passes now. LogicalPlanDeepCopierTest,
      CTEInlineTest, OrExpansionTest, SplitMultiDistinctTest,
      DecomposeRepeatWithPreAggregationTest, LogicalOlapScanTest,
      IvmDeltaRewriteHelperTest and PushDownVirtualColumnsIntoOlapScanTest
      pass. Checkstyle passes. The regression test was not run again: it
      creates no virtual columns that refer to each other.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 21, 2026
…pache#69399

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292, apache#69399

Problem Summary:

Review of apache#69399 found that the plan copier still mishandled virtual
columns that refer to each other, which CSE creates with
experimental_enable_virtual_slot_for_cse=true: copying them in list order
kept the old id of a virtual column that comes later in the list. apache#69399 now
maps every virtual column to its new id before copying their expressions.
This PR carries the identical change until apache#69399 is merged.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - LogicalPlanDeepCopierTest, with the new test for virtual columns that
      refer to each other, BindScoreToOlapScanTest, SearchCteRewriteTest,
      PushDownVirtualColumnsIntoOlapScanTest, PhysicalPlanTranslatorTest,
      PushDownMatchProjectionAsVirtualColumnTest, CTEInlineTest,
      LogicalOlapScanTest and SearchExpressionTest pass. Checkstyle passes.
      The regression suites were not run again: they create no virtual
      columns that refer to each other.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all seven changed files at 5be9ff5372e41aeb75170c85454cd59ba657ced0. No new actionable findings from this static review.

The previously reported dependent-virtual-column issue in thread 4338098220 is fixed: every virtual ExprId is allocated before any dependent expression is copied, and both alias orders are covered by the new unit test. No existing P0/P1 finding remains to carry forward. The user-focus file supplies no additional concerns.

Traced virtual-column creation through CTE normalization/inlining, relation and expression copying, output replacement, physical implementation, and descriptor/order-key translation. Both complete-review agents and the additional mechanism reviewer returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty finding set in round 1. All initial risks and subsidiary candidates were dismissed with source evidence; no changed path or concrete code question remains uncovered.

Critical-checkpoint conclusions:

  1. Goal and evidence: two-pass ExprId allocation fixes dependent aliases; replacement removes old virtual outputs while preserving mapped ordinary/VARIANT slots; score/ANN keys use the final mapping. Added copier, full-planner, and regression cases target the reported failures.
  2. Scope and reuse: changes are focused; ordinary/stream scans share the output helper and TopN copying reuses the expression copier.
  3. Concurrency and locks: no new thread, shared concurrent state, or locking; replacement maps remain per-copy contexts.
  4. Lifecycle: no new resource or static-initialization dependency; relation-cache reuse preserves the completed copy within its context.
  5. Configuration: none added or changed.
  6. Compatibility: no wire/storage-format change or external function-symbol change.
  7. Thrift/protobuf: no gensrc definitions changed.
  8. Parallel paths: ordinary and stream builders preserve their state; schema frontend-conjunct copying remains outside the virtual-column guard. Score, ANN, MATCH, CSE, and nullable replacement callers were checked.
  9. Conditional checks: nonempty-virtual guarding is supported by score/ANN producers that install aliases with their keys; CSE replacement requires an empty existing virtual list. Dependency preallocation is documented.
  10. Test coverage: JUnit 5 tests check fresh IDs, output cardinality, removed old slots, score-key alignment, dependent aliases in both orders, and score/ANN/MATCH/VARIANT CTE planning with single and eliminated consumers. Regression cases cover score, ANN, and MATCH with deterministic output ordering.
  11. Expected results: every added output row was inspected; MATCH booleans, ANN nearest IDs, and BM25 score ordering/values are consistent with the fixtures and scoring formula. Tables are dropped before use, names are hardcoded, and the output is marked as generated.
  12. Observability and errors: existing planner exceptions/plan diagnostics remain adequate for this local fix; no swallowed error or new distributed boundary was introduced.
  13. Transactions and persistence: no EditLog, replay, visible-version, committed-data visibility, or delete-bitmap behavior changes.
  14. Data writes and atomicity: no write path, transaction boundary, or crash-cleanup change.
  15. FE/BE propagation: no new transmitted fields; existing physical implementation and translation receive the copied virtual expressions and remapped keys.
  16. Performance and memory: the added ID/output/key passes are linear in their respective lists; replacement prevents virtual-output accumulation. No material performance or native-memory ownership issue found.
  17. Additional recheck: operative-slot positions, repeated relation reuse, cached-output recomputation, stream read modes, mixed VARIANT output, limits, range payloads, and downstream descriptor availability yielded no distinct new defect.

Validation limits: static inspection only, as required; no builds, tests, or runtime queries were run. Runtime correctness and .out generation provenance were not independently reproduced. Stream modes, combined metadata/range/zero-limit cases, and repeated-context identity were traced statically rather than covered by new direct assertions. These execution limits do not leave an unresolved code candidate in the reviewed scope.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28274 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17600	5087	5019	5019
q2	2111	299	254	254
q3	10227	1366	822	822
q4	4679	456	340	340
q5	7433	788	525	525
q6	179	182	151	151
q7	727	788	618	618
q8	9251	1310	1069	1069
q9	5553	4162	4159	4159
q10	6824	1273	1033	1033
q11	428	239	225	225
q12	645	397	288	288
q13	18111	3636	2986	2986
q14	272	279	245	245
q15	q16	725	729	671	671
q17	1207	747	574	574
q18	7052	6199	6129	6129
q19	1245	959	594	594
q20	392	340	220	220
q21	5220	2104	2284	2104
q22	347	299	248	248
Total cold run time: 100228 ms
Total hot run time: 28274 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5582	5575	5569	5569
q2	709	557	539	539
q3	5207	5401	4925	4925
q4	1989	2124	1539	1539
q5	5242	5189	5557	5189
q6	254	211	156	156
q7	2326	2079	1902	1902
q8	2981	2822	2528	2528
q9	7992	8064	8079	8064
q10	3743	3687	3211	3211
q11	595	445	400	400
q12	710	670	511	511
q13	3506	3854	3319	3319
q14	288	325	282	282
q15	q16	740	710	643	643
q17	8234	7439	7317	7317
q18	12977	12200	12898	12200
q19	923	831	777	777
q20	2120	2123	1915	1915
q21	5503	4560	4674	4560
q22	531	460	403	403
Total cold run time: 72152 ms
Total hot run time: 65949 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150056 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

query5	4315	591	444	444
query6	422	206	198	198
query7	4796	453	223	223
query8	316	166	147	147
query9	8779	3077	3076	3076
query10	481	313	258	258
query11	6012	3840	3557	3557
query12	145	91	85	85
query13	1255	442	309	309
query14	6035	3920	3624	3624
query14_1	3410	3346	3397	3346
query15	199	192	175	175
query16	964	426	396	396
query17	835	625	515	515
query18	2410	406	304	304
query19	196	158	126	126
query20	81	81	80	80
query21	207	129	103	103
query22	12874	12937	12705	12705
query23	13412	12741	12398	12398
query23_1	12493	12449	12596	12449
query24	7388	816	457	457
query24_1	470	485	469	469
query25	540	413	360	360
query26	1257	259	139	139
query27	2787	434	272	272
query28	4672	1945	1956	1945
query29	1558	610	456	456
query30	306	227	185	185
query31	1005	929	823	823
query32	150	95	93	93
query33	530	306	247	247
query34	1007	845	494	494
query35	738	780	645	645
query36	797	829	741	741
query37	133	106	88	88
query38	1750	1737	1732	1732
query39	805	812	738	738
query39_1	728	737	738	737
query40	221	114	99	99
query41	75	68	68	68
query42	89	82	82	82
query43	347	366	314	314
query44	1311	682	675	675
query45	183	177	171	171
query46	818	953	560	560
query47	2975	2961	2639	2639
query48	299	311	228	228
query49	588	402	298	298
query50	672	278	207	207
query51	10331	10593	10215	10215
query52	81	86	82	82
query53	185	207	159	159
query54	263	201	185	185
query55	71	71	66	66
query56	236	227	220	220
query57	1656	1618	1551	1551
query58	293	276	270	270
query59	2132	2140	1960	1960
query60	295	250	235	235
query61	166	168	170	168
query62	390	337	284	284
query63	193	161	158	158
query64	2837	911	754	754
query65	3332	3259	3281	3259
query66	1795	408	292	292
query67	20184	20311	20042	20042
query68	3223	993	617	617
query69	392	287	244	244
query70	828	769	751	751
query71	321	245	198	198
query72	2786	2614	2301	2301
query73	520	515	289	289
query74	4634	4558	4315	4315
query75	2368	2274	1920	1920
query76	2406	1005	567	567
query77	350	360	264	264
query78	9699	9651	9177	9177
query79	966	825	514	514
query80	1204	419	331	331
query81	587	313	270	270
query82	620	137	101	101
query83	303	193	182	182
query84	305	126	104	104
query85	867	416	399	399
query86	397	241	228	228
query87	2012	1930	1823	1823
query88	3647	2655	2647	2647
query89	331	282	242	242
query90	1911	167	165	165
query91	167	150	130	130
query92	91	89	89	89
query93	1026	991	564	564
query94	642	297	277	277
query95	569	417	302	302
query96	662	490	234	234
query97	2424	2449	2310	2310
query98	158	156	149	149
query99	770	758	640	640
Total cold run time: 235314 ms
Total hot run time: 150056 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.04
query3	0.26	0.09	0.09
query4	1.61	0.09	0.09
query5	0.23	0.25	0.24
query6	1.16	0.70	0.67
query7	0.03	0.01	0.00
query8	0.05	0.04	0.04
query9	0.40	0.33	0.35
query10	0.58	0.56	0.61
query11	0.20	0.14	0.14
query12	0.18	0.15	0.14
query13	0.51	0.49	0.49
query14	1.04	1.01	1.01
query15	0.62	0.60	0.60
query16	0.30	0.34	0.33
query17	1.11	1.12	1.18
query18	0.19	0.18	0.18
query19	2.08	1.98	1.92
query20	0.02	0.01	0.01
query21	15.51	0.20	0.13
query22	4.90	0.05	0.05
query23	16.27	0.32	0.12
query24	2.71	0.71	0.46
query25	0.12	0.09	0.07
query26	1.66	0.21	0.16
query27	0.05	0.05	0.04
query28	4.06	0.77	0.35
query29	12.49	4.12	3.28
query30	0.29	0.14	0.10
query31	2.77	0.56	0.32
query32	3.23	0.60	0.49
query33	3.13	3.23	3.31
query34	15.35	4.02	3.42
query35	3.36	3.34	3.35
query36	0.55	0.46	0.43
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 97.65 s
Total hot run time: 24.33 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28345 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17592	5134	5036	5036
q2	2119	309	261	261
q3	10223	1368	825	825
q4	4680	466	337	337
q5	7432	790	532	532
q6	179	182	158	158
q7	750	793	605	605
q8	9244	1324	1050	1050
q9	5483	4175	4189	4175
q10	6802	1289	1030	1030
q11	427	246	233	233
q12	637	405	279	279
q13	18069	3552	3008	3008
q14	270	264	240	240
q15	q16	701	713	665	665
q17	1193	742	578	578
q18	7012	6195	6126	6126
q19	1109	950	585	585
q20	369	326	226	226
q21	4672	2143	2303	2143
q22	341	291	253	253
Total cold run time: 99304 ms
Total hot run time: 28345 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5617	5601	5606	5601
q2	674	571	541	541
q3	5219	5380	4923	4923
q4	2005	2138	1531	1531
q5	5258	5168	5412	5168
q6	260	196	159	159
q7	2374	2014	1898	1898
q8	3008	2669	2741	2669
q9	7947	8103	7998	7998
q10	3706	3701	3213	3213
q11	608	492	462	462
q12	690	682	534	534
q13	3557	4032	3207	3207
q14	326	327	306	306
q15	q16	699	706	649	649
q17	8220	7501	6886	6886
q18	13000	12183	12848	12183
q19	903	798	791	791
q20	2133	2163	1917	1917
q21	5524	4544	4706	4544
q22	482	437	409	409
Total cold run time: 72210 ms
Total hot run time: 65589 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151279 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

query5	4334	574	440	440
query6	429	216	196	196
query7	4815	436	226	226
query8	314	166	141	141
query9	8767	3152	3134	3134
query10	482	310	293	293
query11	6022	3884	3603	3603
query12	153	93	90	90
query13	1253	439	339	339
query14	6066	3936	3687	3687
query14_1	3436	3375	3424	3375
query15	202	197	183	183
query16	952	437	429	429
query17	861	655	542	542
query18	2416	426	311	311
query19	202	165	134	134
query20	91	86	87	86
query21	209	127	104	104
query22	13013	13141	12867	12867
query23	13470	12822	12614	12614
query23_1	12774	12657	12715	12657
query24	7384	813	462	462
query24_1	474	478	468	468
query25	533	419	359	359
query26	1265	263	142	142
query27	2771	431	268	268
query28	4633	1961	1938	1938
query29	1566	597	451	451
query30	303	221	185	185
query31	1004	935	824	824
query32	146	90	96	90
query33	535	305	241	241
query34	910	842	498	498
query35	742	754	666	666
query36	808	797	725	725
query37	131	104	85	85
query38	1781	1752	1697	1697
query39	814	816	744	744
query39_1	740	726	742	726
query40	230	118	97	97
query41	71	74	73	73
query42	91	92	89	89
query43	353	372	312	312
query44	1324	684	671	671
query45	173	177	172	172
query46	816	965	569	569
query47	3030	2989	2843	2843
query48	300	357	212	212
query49	574	394	283	283
query50	660	273	212	212
query51	10449	10634	10288	10288
query52	78	79	72	72
query53	181	205	147	147
query54	254	211	183	183
query55	68	65	66	65
query56	221	226	203	203
query57	1681	1654	1598	1598
query58	290	268	253	253
query59	2084	2083	1983	1983
query60	281	240	220	220
query61	144	145	144	144
query62	384	335	282	282
query63	191	158	159	158
query64	2732	966	752	752
query65	3334	3274	3291	3274
query66	1774	408	300	300
query67	20245	20174	19867	19867
query68	2945	998	595	595
query69	409	281	257	257
query70	835	782	852	782
query71	304	243	215	215
query72	2682	2585	2277	2277
query73	499	510	288	288
query74	4650	4533	4376	4376
query75	2314	2280	1925	1925
query76	2324	1028	601	601
query77	344	377	272	272
query78	9879	9891	9201	9201
query79	964	920	531	531
query80	978	432	365	365
query81	585	324	273	273
query82	574	141	110	110
query83	295	199	179	179
query84	314	118	98	98
query85	991	424	367	367
query86	351	231	229	229
query87	2006	1950	1782	1782
query88	3674	2713	2691	2691
query89	330	283	244	244
query90	1716	172	172	172
query91	162	153	131	131
query92	101	91	94	91
query93	1009	1000	568	568
query94	512	298	275	275
query95	566	339	297	297
query96	655	486	237	237
query97	2473	2423	2323	2323
query98	162	170	149	149
query99	763	769	648	648
Total cold run time: 235043 ms
Total hot run time: 151279 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5be9ff5372e41aeb75170c85454cd59ba657ced0, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.10	0.09
query5	0.23	0.24	0.24
query6	1.17	0.70	0.68
query7	0.03	0.01	0.01
query8	0.05	0.04	0.03
query9	0.40	0.35	0.35
query10	0.59	0.60	0.60
query11	0.23	0.14	0.14
query12	0.20	0.15	0.15
query13	0.50	0.49	0.50
query14	1.04	1.01	1.01
query15	0.63	0.61	0.60
query16	0.36	0.34	0.35
query17	1.12	1.09	1.12
query18	0.22	0.20	0.19
query19	2.05	1.99	2.01
query20	0.02	0.01	0.01
query21	15.51	0.20	0.13
query22	4.90	0.06	0.05
query23	16.27	0.34	0.13
query24	3.08	0.59	0.31
query25	0.14	0.05	0.05
query26	0.87	0.21	0.16
query27	0.06	0.03	0.05
query28	3.21	0.78	0.37
query29	12.47	4.10	3.28
query30	0.29	0.14	0.10
query31	2.79	0.56	0.32
query32	3.22	0.60	0.51
query33	3.13	3.25	3.29
query34	15.34	4.00	3.42
query35	3.33	3.34	3.39
query36	0.57	0.46	0.42
query37	0.10	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 96.51 s
Total hot run time: 24.36 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

3 similar comments
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 92.59% (25/27) 🎉
Increment coverage report
Complete coverage report

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 5be9ff5, including the full PR diff and the changes since be40091. The previous dependent-virtual-column finding is fixed: all virtual IDs are allocated before copying expressions, and ExpressionDeepCopier reuses those IDs. The added test covers both dependency orders and checks copied input slots against scan output. The output replacement, VARIANT subpath preservation, stream-scan overrides, and score/ANN order-key remapping look consistent. No further actionable issue found. Source review only; I did not independently run builds or tests.

@airborne12
airborne12 merged commit 8851108 into apache:master Oct 22, 2026
43 of 44 checks passed
@airborne12
airborne12 deleted the fix-deep-copy-virtual-columns branch October 22, 2026 02:35
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 22, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292, apache#69399

Problem Summary:

apache#69399, the plan copier fix that this PR carried until it was merged, is now
on master. Merge master so that the copier change leaves the diff of this
PR.

### Release note

None

### Check List (For Author)

- Test: Unit Test (the FE tests around the copier and the binder are run
  after the merge)
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 22, 2026
… in scan expressions

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69399, apache#69419

Problem Summary:

The score() support of this PR special-cased generic code: TopN filter
pushdown skipped scans with a score, the MV rewrite check skipped them,
and the rewrite-time binder recognized the delete sign filter of UNIQUE
tables and rejected table streams by class. All of it came from one BE
limitation and from binding score() after analysis.

BE: the olap scan prepares the score of every row of its row bitmap
before the first batch, but SegmentIterator marked only the stored
columns of a pushed-down expression as common-expression columns. An
expression on the score alone, e.g. score() > 0.5, set no
_is_need_expr_eval, so it was never evaluated, while the scanner had
already removed it from its own conjuncts. An expression that read the
score next to other columns, e.g. a TopN filter on CAST(score() AS INT),
evaluated Score() itself, which returns 0. SegmentIterator now marks the
score column like a stored column: it is read with the other
common-expression columns, before the expressions run.

FE, analysis: BindScore replaces BindScoreToOlapScan. It runs first on
each plan the analyzer binds (a statement, and on their own its views,
CTEs and subqueries), while the plan nodes of a SELECT are still its
clauses, so it needs no knowledge of what analysis adds later. For each
SELECT that calls score(), it checks the usage rules, asks the table in
its FROM clause for a score column and replaces the score() calls with
that column. BindRelation adds the column to the olap scan and rejects a
relation that cannot provide it, e.g. a view, a CTE or a table stream.
The column is left out of *. CheckScoreInCorrelatedSubquery rejects a
scored scan under a correlated subquery after SubqueryToApply. Since
score() no longer reaches type coercion as a function, the check that it
is compared with numbers moves from ComparisonPredicate to BindScore,
which sees the type of a literal before binding.

FE, rewrite: the minimum score is an ordinary predicate of the scan.
PushDownScoreRangeIntoOlapScan is removed; PushDownScoreTopNIntoOlapScan
moves the minimum into the scan together with a TopN on the score, as on
master, because the scan keeps the best rows of a segment only when no
predicate follows. TopnFilterPushDownVisitor, the MV check and the
translator lose their score() cases: a TopN filter reads the score like
any column, and a materialized view without the score cannot answer a
predicate on it.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression test
    - BE: SegmentIteratorScoreCommonExprTest is new. With
      SegmentIteratorApplyIndexExprTest and
      SegmentIteratorCandidatePushdownTest, 35 tests pass.
    - FE: BindScoreTest (79 tests), the tests of nereids/rules/analysis,
      CTEInlineTest, LogicalPlanDeepCopierTest, TopNRuntimeFilterTest,
      PullUpProjectExprUnderTopNTest and MaterializedViewUtilsTest pass.
      Checkstyle passes.
    - Regression: the 20 suites that use score() (inverted_index_p0,
      search, ann_index_p0, nereids_rules_p0/cte) pass on an ASAN build
      of this commit. Their .out files are unchanged; two suites check
      the new EXPLAIN output.
- Behavior changed: Yes. TopN filters on keys computed from score() reach
  the scan. Without a TopN on the score, EXPLAIN shows the minimum score
  among the scan predicates instead of SCORE RANGE.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
airborne12 added a commit that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #69399 (merged; the plan copier fix this PR needs), #69419
(merged; fixes the pre-existing MATCH predicate inference that review of
this PR found)

Problem Summary:

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents
WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`score()` is computed by the olap scan that evaluates the MATCH and
SEARCH predicates of its SELECT. This PR binds `score()` to that scan at
the start of rewrite, makes the scan's expressions read the score, and
rejects every other use with an error that names the rule.

**Supported.** A SELECT that reads one table and has a MATCH or SEARCH
predicate in its WHERE clause can use `score()`:
- in the WHERE clause, as one minimum score: `score() > x` or `score()
>= x` (also `x < score()`, `x <= score()`, and with a FLOAT or DOUBLE
cast);
- in the select list, window functions and ORDER BY, if the SELECT does
not aggregate (with GROUP BY, including GROUPING SETS, ROLLUP and CUBE,
aggregate functions or DISTINCT, `score()` can only be the minimum
score).

A scored SELECT in a subquery, view, CTE or set operation still needs
`ORDER BY score() LIMIT n`, as before. The TopN keeps the predicates of
the outer query out of its scan, e.g. an outer MATCH cannot add to its
score. A correlated subquery cannot call `score()`, because
decorrelation removes that TopN.

**Rejected.** A join, a subquery or a lateral view in the scored SELECT;
a view, CTE or other relation that is not an olap table (a table stream
fails during planning, because its scan is replaced by scans of the
table and of its changes); any other filter on the score (`score() < x`,
`score() = x`, lossy casts, ORs, a second bound); `score()` in
aggregates, GROUP BY, HAVING or QUALIFY; MATCH or SEARCH in HAVING or
QUALIFY of a scored SELECT, also through an alias; `score()` in a
correlated subquery; a SELECT without its own MATCH or SEARCH; a nested
scored SELECT without the TopN.

**How.**
- BE: the scan reads the score in its own expressions. `SegmentIterator`
prepares the score of every row of its row bitmap before the first
batch. It marked only stored columns of a pushed-down expression as
common-expression columns, so an expression on the score alone, e.g.
`score() > 0.5`, was never evaluated (and the scanner had already
removed it from its conjuncts), and an expression that read the score
next to other columns, e.g. a TopN filter on `CAST(score() AS INT)`,
evaluated `Score()` itself, which returns 0. Now the score column is
read with the other common-expression columns, before the expressions
run.
- FE, binding: `BindScoreToOlapScan` replaces `CheckScoreUsage`. It runs
first in rewrite, while subqueries, views and CTEs still have their own
plan nodes, so the analysis classes are as on master. For each SELECT
that calls `score()`, it checks the rules above, adds a score column to
the scan (score limit 0: the scan scores every row it returns) and
replaces the `score()` calls of the SELECT with that column. A table is
a relation, also under the filters analysis puts on it (the delete sign
filter of a UNIQUE table, row policies). Binding after analysis also
covers `* REPLACE (score() AS c)`, alias functions over `score()` and
`ORDER BY` a position, and a UDF named `score` is never mistaken for it.
- FE, rewrite: the minimum score is an ordinary predicate of the scan.
`PushDownScoreTopNIntoOlapScan` pushes a TopN on the score into the
scan, and with it the minimum score, exactly as before this PR (with a
zero score limit when SEARCH has another predicate), because the scan
keeps the best rows of a segment only when no predicate follows. TopN
filter pushdown and MV rewrite need no `score()` cases: a TopN filter
reads the score like any other column, and a materialized view without
the score cannot answer a predicate on it.
- This PR needs the plan copier fix of #69399, which is merged: copying
a scan with virtual columns gave each one two ids and kept the old slot
in the output.
- The BE keeps the score unchanged by MATCH projections:
`SegmentIterator` evaluates them without the score collector.
- `SearchExpression` can be rendered as SQL, which the shape plan logged
at debug level and `EXPLAIN SHAPE PLAN` need.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…#69399 (#69560)

### What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to `branch-4.2`.

A CTE fails to plan with "A expression contains slot not from children"
when the optimizer inlines it after one of its scans got a virtual
column: a CTE that orders by `score()` or by an ANN distance with a
LIMIT, or that has MATCH in its select list and is inlined after
rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids,
copied them in list order, kept the old slot in the output of the copied
scan and left the score and ANN order keys on the old slots. The copier
now maps every virtual column to its new id before copying their
expressions and remaps the order keys, and
`LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in
the output.

#### Branch adaptations

The branch has no table streams, so the change to
`LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the
constructor call of the branch, `LogicalPlanDeepCopier` imports
`LogicalOlapScan`, which master already imported, and
`LogicalPlanDeepCopierTest` keeps the tests of the branch and adds the
two tests of #69399.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by `score()` or by an ANN distance with a LIMIT, or that has MATCH in
its select list and is inlined after rewrite.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…69292 (#69562)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit
of #69560, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69560
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps
the behavior of the branch and skips the push-down instead of failing.
- `SearchDslParser` is imported from its package on the branch.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…cross column equalities #69419 (#69561)

### What problem does this PR solve?

Related PR: #69419

Problem Summary: Backport #69419 to `branch-4.2`.

Predicate inference copied a predicate on one column to every column
that is equal to it, also when the predicate contained MATCH,
multi_match or SEARCH. Their result depends on the inverted index of the
column, not only on its value: with `c1 = c2` and different parsers on
the two columns, the copy removed matching rows, and with the same index
it doubled `score()`, so a minimum score let low-scored rows through.
`InferPredicateByReplace.infer` now skips every input that contains
`Match`, `MultiMatch`, `Search` or `SearchExpression`.

#### Branch adaptations

The branch does not have the `NoneMovableFunction` check that master has
next to the new check, so only the new check is added. The test imports
`SearchDslParser` from its package on the branch.

### Release note

Fix wrong results and scores when predicate inference copied a MATCH,
multi_match or SEARCH predicate to an equal column.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. Predicate inference no longer copies a predicate that
contains MATCH, multi_match or SEARCH to an equal column.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 24, 2026
### What problem does this PR solve?

Related PR: apache#69399

Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at
random for each session. With it off, MATCH is not evaluated with the
inverted index in the scan, so no BM25 score is collected and score()
returns 0. test_cte_inline_virtual_column got score 0.0 for its
TopN. Like the score suites of the branch, the suite sets
enable_common_expr_pushdown = true.

### Release note

None

### Check List (For Author)

- Test: Regression test. With use_fuzzy_session_variable = true and
the pull request number of CI, the suite failed in sessions that
drew enable_common_expr_pushdown = false; with the setting pinned,
four runs passed.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…#69399 (#69563)

### What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to `branch-4.1`.

A CTE fails to plan with "A expression contains slot not from children"
when the optimizer inlines it after one of its scans got a virtual
column: a CTE that orders by `score()` or by an ANN distance with a
LIMIT, or that has MATCH in its select list and is inlined after
rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids,
copied them in list order, kept the old slot in the output of the copied
scan and left the score and ANN order keys on the old slots. The copier
now maps every virtual column to its new id before copying their
expressions and remaps the order keys, and
`LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in
the output.

#### Branch adaptations

The branch has no table streams, so the change to
`LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the
constructor call of the branch, and `LogicalPlanDeepCopierTest` and
`CTEInlineTest` keep the tests of the branch and add the tests of
#69399. `test_cte_inline_virtual_column` sets
`enable_common_expr_pushdown = true`, as the score suites of the branch
do: CI on branch-4.1 draws it at random for each session, and without it
MATCH is not evaluated with the index in the scan and `score()` is 0.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by `score()` or by an ANN distance with a LIMIT, or that has MATCH in
its select list and is inlined after rewrite.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…cross column equalities #69419 (#69564)

### What problem does this PR solve?

Related PR: #69419

Problem Summary: Backport #69419 to `branch-4.1`.

Predicate inference copied a predicate on one column to every column
that is equal to it, also when the predicate contained MATCH,
multi_match or SEARCH. Their result depends on the inverted index of the
column, not only on its value: with `c1 = c2` and different parsers on
the two columns, the copy removed matching rows, and with the same index
it doubled `score()`, so a minimum score let low-scored rows through.
`InferPredicateByReplace.infer` now skips every input that contains
`Match`, `MultiMatch`, `Search` or `SearchExpression`.

#### Branch adaptations

The branch does not have the `NoneMovableFunction` check that master has
next to the new check, so only the new check is added. The test imports
`SearchDslParser` from its package on the branch.
`infer_match_predicate` sets `enable_common_expr_pushdown = true`, as
the score suites of the branch do: CI on branch-4.1 draws it at random
for each session, and without it MATCH is not evaluated with the index
in the scan and `score()` is 0.

### Release note

Fix wrong results and scores when predicate inference copied a MATCH,
multi_match or SEARCH predicate to an equal column.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. Predicate inference no longer copies a predicate that
contains MATCH, multi_match or SEARCH to an equal column.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…69292 (#69565)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit
of #69563, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69563
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- The branch evaluates MATCH projections with `evaluate_inverted_index`
directly, and that loop runs without the score collector.
- `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch
for `limit + offset`, which fails like master does.
- `SearchDslParser` and `CollectionSimilarity` are included from their
packages on the branch.
- The branch has no SNII storage format, so `BindScoreToOlapScanTest`
and `test_score_filter_matrix` cover V2 and V3, and the `.out` of
`test_score_filter_matrix` is regenerated without the SNII cases.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.
- `test_score_filter_usage`, `test_score_filter_matrix` and
`test_cte_inline_virtual_column` set `enable_common_expr_pushdown =
true`, as the score suites of the branch do: CI on branch-4.1 draws it
at random for each session, and without it MATCH is not evaluated with
the index in the scan and `score()` is 0.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.6-merged dev/4.2.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants