Skip to content

[fix](nereids) Support minimum score filters without TopN - #69292

Merged
airborne12 merged 23 commits into
apache:masterfrom
airborne12:fix-score-filter-without-topn
Oct 23, 2026
Merged

airborne12 merged 23 commits into
apache:masterfrom
airborne12:fix-score-filter-without-topn

Conversation

@airborne12

@airborne12 airborne12 commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: N/A

Related PR: #69399 (merged; the plan copier fix this PR needs), #69419 (merged; fixes the pre-existing MATCH predicate inference that review of this PR found)

Problem Summary:

score() only worked as ORDER BY score() LIMIT n directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound score() placeholder as zero and silently dropped every row:

SELECT id FROM documents
WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing

score() is computed by the olap scan that evaluates the MATCH and SEARCH predicates of its SELECT. This PR binds score() to that scan at the start of rewrite, makes the scan's expressions read the score, and rejects every other use with an error that names the rule.

Supported. A SELECT that reads one table and has a MATCH or SEARCH predicate in its WHERE clause can use score():

  • in the WHERE clause, as one minimum score: score() > x or score() >= x (also x < score(), x <= score(), and with a FLOAT or DOUBLE cast);
  • in the select list, window functions and ORDER BY, if the SELECT does not aggregate (with GROUP BY, including GROUPING SETS, ROLLUP and CUBE, aggregate functions or DISTINCT, score() can only be the minimum score).

A scored SELECT in a subquery, view, CTE or set operation still needs ORDER BY score() LIMIT n, as before. The TopN keeps the predicates of the outer query out of its scan, e.g. an outer MATCH cannot add to its score. A correlated subquery cannot call score(), because decorrelation removes that TopN.

Rejected. A join, a subquery or a lateral view in the scored SELECT; a view, CTE or other relation that is not an olap table (a table stream fails during planning, because its scan is replaced by scans of the table and of its changes); any other filter on the score (score() < x, score() = x, lossy casts, ORs, a second bound); score() in aggregates, GROUP BY, HAVING or QUALIFY; MATCH or SEARCH in HAVING or QUALIFY of a scored SELECT, also through an alias; score() in a correlated subquery; a SELECT without its own MATCH or SEARCH; a nested scored SELECT without the TopN.

How.

  • BE: the scan reads the score in its own expressions. SegmentIterator prepares the score of every row of its row bitmap before the first batch. It marked only stored columns of a pushed-down expression as common-expression columns, so an expression on the score alone, e.g. score() > 0.5, was never evaluated (and the scanner had already removed it from its conjuncts), and an expression that read the score next to other columns, e.g. a TopN filter on CAST(score() AS INT), evaluated Score() itself, which returns 0. Now the score column is read with the other common-expression columns, before the expressions run.
  • FE, binding: BindScoreToOlapScan replaces CheckScoreUsage. It runs first in rewrite, while subqueries, views and CTEs still have their own plan nodes, so the analysis classes are as on master. For each SELECT that calls score(), it checks the rules above, adds a score column to the scan (score limit 0: the scan scores every row it returns) and replaces the score() calls of the SELECT with that column. A table is a relation, also under the filters analysis puts on it (the delete sign filter of a UNIQUE table, row policies). Binding after analysis also covers * REPLACE (score() AS c), alias functions over score() and ORDER BY a position, and a UDF named score is never mistaken for it.
  • FE, rewrite: the minimum score is an ordinary predicate of the scan. PushDownScoreTopNIntoOlapScan pushes a TopN on the score into the scan, and with it the minimum score, exactly as before this PR (with a zero score limit when SEARCH has another predicate), because the scan keeps the best rows of a segment only when no predicate follows. TopN filter pushdown and MV rewrite need no score() cases: a TopN filter reads the score like any other column, and a materialized view without the score cannot answer a predicate on it.
  • This PR needs the plan copier fix of [fix](nereids) Fix deep copy of scans with virtual columns #69399, which is merged: copying a scan with virtual columns gave each one two ids and kept the old slot in the output.
  • The BE keeps the score unchanged by MATCH projections: SegmentIterator evaluates them without the score collector.
  • SearchExpression can be rendered as SQL, which the shape plan logged at debug level and EXPLAIN SHAPE PLAN need.

Release note

score() works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score score() > x or score() >= x, without ORDER BY score() LIMIT. A scored SELECT in a subquery, view, CTE or set operation still needs ORDER BY score() LIMIT. Other uses of score() are rejected instead of evaluating it as 0.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    Run locally on an ASAN build of the final commit, which merges master with [fix](nereids) Do not infer MATCH and SEARCH predicates across column equalities #69419:

    • FE unit tests: BindScoreToOlapScanTest (80 cases: the supported shapes for V2, V3 and SNII with MATCH and SEARCH, * and * REPLACE, alias functions, one score column per SELECT, the main query of a WITH clause, nested TopNs ordered by alias or position, a UDF named score, and every rejected shape with its message), CTEInlineTest, LogicalPlanDeepCopierTest, TopNRuntimeFilterTest, PullUpProjectExprUnderTopNTest, MaterializedViewUtilsTest, SearchExpressionTest, and the predicate inference tests InferPredicateByReplaceTest, InferPredicatesTest and UnequalPredicateInferTest (317 tests). Checkstyle passes.
    • BE unit tests, run before that merge: the new SegmentIteratorScoreCommonExprTest (an expression on the score marks the score column as a common-expression column; another virtual column is still computed from its expression), with SegmentIteratorApplyIndexExprTest and SegmentIteratorCandidatePushdownTest (35 tests). The merge does not change the BE files of this PR, and master's BE unit tests do not compile at the moment (function_string_test.cpp, from [improvement](be) Optimize string functions and ARM byte checks #69223).
    • Regression tests: the 20 suites that use score() in inverted_index_p0, inverted_index_p0/storage_format, search, ann_index_p0 and nereids_rules_p0/cte, infer_match_predicate from [fix](nereids) Do not infer MATCH and SEARCH predicates across column equalities #69419, and the five suites from master that use score() (test_bm25_score_parallel_scan_statistics, test_search_document_boolean_contract, test_search_null_array_payload_contract, test_variant_search_historical_rowset_index and test_variant_search_missing_doc_path), with unchanged .out files except those two cases. test_score_filter_usage and test_score_filter_matrix check the new EXPLAIN output, and test_score_filter_usage covers a TopN filter on CAST(score() AS INT) over 20,000 rows read in many batches. Its .out was regenerated by the framework for two new cases, select_star_replace_score and nested_score_topn_by_position.

    Cloud mode and mixed-version clusters were not tested.

  • Behavior changed:

    • No.
    • Yes. The outermost SELECT can use score() without ORDER BY score() LIMIT; unsupported uses of score() fail planning instead of evaluating it as 0. TopN filters on keys computed from score() now reach the scan. Without a TopN on the score, EXPLAIN shows the minimum score among the scan predicates instead of SCORE RANGE. A minimum score without a TopN needs the BE change of this PR, so BEs must be upgraded before FEs, as usual; queries that master supports keep their plans.
  • Does this need documentation?

    • No.
    • Yes. The score() documentation should describe where score() can be used. A documentation PR has not been opened.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

🤖 Generated with Claude Code

### What problem does this PR solve?

Problem Summary: A WHERE-only score() predicate misses the score TopN rewrite and can evaluate the zero-valued scalar placeholder. Bind supported minimum score predicates to the existing score virtual column and full-scoring mode. Retain hidden score slots during physical translation and preserve projection, sorting, LIMIT, and OFFSET behavior. Share validation with score TopN, reject lossy score casts and raw aggregate score expressions, and support SEARCH SQL rendering during DEBUG plan display.

### Release note

Support one score() > literal or score() >= literal filter with MATCH or SEARCH without requiring ORDER BY or LIMIT. FLOAT and DOUBLE casts remain supported; lossy score casts and unsupported score contexts fail analysis. Existing index scoring formulas and norm encodings are preserved.

### Check List (For Author)

- Test: 141 FE unit tests; six score regression suites, including 146 matrix result groups; V2/SNII MATCH and SEARCH probes consuming 50,000 scored rows each from 100,000 documents.
- Behavior changed: Yes. Minimum score filtering no longer requires SQL TopN; unsupported score usage fails analysis.
- Does this need documentation: Yes. Standalone minimum score filtering should be documented; no documentation PR has been opened.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 11 changed paths at bd79ec1 using the supplied authoritative diff, the repository review skill and required guides, plus independent FE, test/semantics and scoring-risk reviews.

Three verified new inline findings: one P1 where a projected MATCH changes the rows admitted by the minimum-score filter; one P2 where enabled virtual CSE leaves nested projected score() calls evaluating the zero placeholder; and one P2 where an existing p0 regression still expects the newly supported query to fail. There were no existing inline threads or independently confirmed existing P0/P1 comments to carry forward. Changes are requested for the P1 finding.

Critical checkpoint conclusions:

  • Goal and proof: the ordinary minimum-score path materializes a hidden score slot and uses limit zero for full scoring; unit assertions and regression goldens cover it. The two virtual-expression interactions in the inline findings prevent complete correctness.
  • Scope and reuse: the change is focused on five FE production files plus supporting tests/results, reusing existing scan scoring metadata and BE implementations.
  • Concurrency and locks: no threads, lock order or concurrent mutable state are introduced. The projected-MATCH finding concerns score accumulation within one segment's shared evaluation context.
  • Lifecycle and initialization: the existing query/segment scoring lifecycle is retained; no new static initialization dependencies or ownership cycles are introduced.
  • Configuration: no configuration items are added; the CSE finding is reachable through an existing optional experimental session setting.
  • Compatibility: existing optional score sort/range fields are reused, with no new function symbol, storage format or persistent schema. Mixed-version execution was not performed.
  • Thrift/protobuf: neither directory changes, so cross-release field-ID comparison is inapplicable.
  • Parallel paths: traced standalone range and score TopN, project and filter-only shapes, MATCH and SEARCH, CLucene V2/V3 and SNII, deletes/MoW, cache/WAND, ANN and lazy materialization. The CSE and projected-MATCH paths need fixes; lazy slot loss and user-output widening were ruled out by upstream guards and explicit sink outputs.
  • Conditional checks and errors: top-level single minimum comparisons, reversed forms and FLOAT/DOUBLE casts have concrete validation paths; unsupported contexts and lossy casts raise analysis errors. The CSE interaction bypasses intended score binding.
  • Test coverage: the added suites exercise boundaries, casts, no hits, NULL thresholds, multiple rowsets, cache/WAND, deletes, MoW, ordinary/multiple sorts, offsets and ANN. They omit both reported virtual-expression combinations, and the older test_bm25_score assertion needs updating.
  • Changed expected results: all 146 matrix groups and seven usage groups were reviewed; paired range/TopN controls agree. Independent float32 BM25 arithmetic reproduced every distinct added score, including the intentional V2/V3 versus SNII long-norm difference. Golden generation was not rerun.
  • Observability: existing explain score sort/range metadata, index profiles and explicit diagnostics support investigation; no additional logging or metrics is required for this FE extension.
  • Transactions and persistence: no production transaction, EditLog, replay, failover or persistence path changes.
  • Writes and atomicity: production changes are read/planning-only. Test inserts, updates and deletes exercise the existing visibility paths; no new write atomicity or crash-recovery mechanism is introduced.
  • FE/BE transfer: traced virtual slots, score ordering, zero score limit and double threshold from logical scan through physical translation and OlapScanNode to ScoreRuntime and range materialization. No new protocol variable is added.
  • Performance: full scoring is proportional to candidate rows as needed for a standalone threshold; eligible score TopN retains its bounded path. ANN metadata clearing uses scalar distance fallback, whose runtime cost was not benchmarked.
  • Other issues: no further substantiated finding after the assigned coverage and missed-case rechecks. No additional user review focus was supplied.

This was a static-only review: no builds, Doris tests, cloud execution, mixed-version execution or performance benchmarks were run, and source files were not modified. Those validation limits do not represent unreviewed changed paths.

Review converged after 2 rounds: all three reviewers returned NO_NEW_VALUABLE_FINDINGS on the identical frozen findings, coverage is complete, and every candidate/risk has an evidence-based disposition. No unresolved review question remains within the static review scope.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28375 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd79ec172d8f9e74e7de00dc1b6688db99fd82b6, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	5099	5045	5045
q2	2101	320	275	275
q3	10223	1376	818	818
q4	4679	466	336	336
q5	7430	797	523	523
q6	185	187	155	155
q7	740	794	587	587
q8	9254	1454	1084	1084
q9	5480	4214	4170	4170
q10	6803	1280	1001	1001
q11	428	252	225	225
q12	637	396	281	281
q13	18099	3668	3012	3012
q14	269	275	246	246
q15	q16	738	713	653	653
q17	1212	755	577	577
q18	7027	6162	6109	6109
q19	1115	964	609	609
q20	373	330	222	222
q21	5567	2201	2315	2201
q22	352	300	246	246
Total cold run time: 100308 ms
Total hot run time: 28375 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5607	5617	5614	5614
q2	669	584	536	536
q3	5226	5367	4899	4899
q4	1986	2135	1532	1532
q5	5253	5146	5636	5146
q6	254	206	152	152
q7	2310	2079	1921	1921
q8	3103	2970	2750	2750
q9	8116	8026	8071	8026
q10	3720	3708	3255	3255
q11	627	452	419	419
q12	695	696	509	509
q13	3529	3845	3212	3212
q14	302	331	301	301
q15	q16	679	710	661	661
q17	8324	7544	7319	7319
q18	12975	12154	12906	12154
q19	931	792	782	782
q20	2142	2146	1906	1906
q21	5621	4630	4646	4630
q22	522	434	408	408
Total cold run time: 72591 ms
Total hot run time: 66132 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150241 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd79ec172d8f9e74e7de00dc1b6688db99fd82b6, data reload: false

query5	4305	574	437	437
query6	427	208	208	208
query7	4796	442	226	226
query8	322	162	152	152
query9	8752	3049	3081	3049
query10	494	306	244	244
query11	6035	3836	3560	3560
query12	146	93	87	87
query13	1257	447	318	318
query14	6111	3893	3634	3634
query14_1	3405	3366	3423	3366
query15	206	196	179	179
query16	949	436	419	419
query17	854	652	530	530
query18	2418	435	310	310
query19	203	164	138	138
query20	85	84	83	83
query21	239	128	104	104
query22	12896	13105	12780	12780
query23	13494	12895	12549	12549
query23_1	12556	12631	12449	12449
query24	7394	807	454	454
query24_1	454	458	455	455
query25	509	385	342	342
query26	1260	251	136	136
query27	2769	436	267	267
query28	4628	1920	1936	1920
query29	1536	575	434	434
query30	300	219	184	184
query31	1006	928	819	819
query32	138	86	89	86
query33	522	289	229	229
query34	858	842	495	495
query35	726	757	641	641
query36	821	787	751	751
query37	121	95	77	77
query38	1751	1731	1652	1652
query39	793	796	733	733
query39_1	729	736	727	727
query40	209	113	93	93
query41	66	62	63	62
query42	84	84	78	78
query43	339	366	315	315
query44	1306	674	693	674
query45	185	178	162	162
query46	828	964	571	571
query47	3039	2846	2946	2846
query48	291	304	209	209
query49	570	395	290	290
query50	689	303	195	195
query51	10307	10427	10339	10339
query52	81	86	69	69
query53	188	204	148	148
query54	230	210	178	178
query55	83	75	68	68
query56	238	225	214	214
query57	1626	1615	1554	1554
query58	295	268	262	262
query59	2093	2120	1998	1998
query60	287	247	221	221
query61	147	143	142	142
query62	391	325	274	274
query63	191	156	162	156
query64	2725	935	720	720
query65	3343	3272	3248	3248
query66	1785	418	301	301
query67	20218	20046	20016	20016
query68	2986	988	602	602
query69	380	287	240	240
query70	832	739	770	739
query71	292	236	219	219
query72	2752	2563	2230	2230
query73	514	526	295	295
query74	4558	4517	4339	4339
query75	2318	2283	1895	1895
query76	2343	1015	589	589
query77	338	380	257	257
query78	10043	9643	9142	9142
query79	886	842	518	518
query80	466	415	350	350
query81	514	328	266	266
query82	214	131	107	107
query83	193	196	179	179
query84	285	122	100	100
query85	766	415	373	373
query86	253	239	227	227
query87	1972	1958	1774	1774
query88	3585	2647	2635	2635
query89	320	275	242	242
query90	2096	166	170	166
query91	163	146	124	124
query92	96	88	89	88
query93	991	984	572	572
query94	443	312	267	267
query95	568	399	303	303
query96	670	489	236	236
query97	2421	2420	2347	2347
query98	162	148	147	147
query99	760	765	662	662
Total cold run time: 233090 ms
Total hot run time: 150241 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd79ec172d8f9e74e7de00dc1b6688db99fd82b6, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.10	0.11
query5	0.23	0.23	0.23
query6	1.17	0.68	0.66
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.35	0.33
query10	0.56	0.57	0.59
query11	0.19	0.15	0.15
query12	0.18	0.16	0.15
query13	0.48	0.49	0.50
query14	1.04	1.03	1.00
query15	0.61	0.61	0.62
query16	0.31	0.35	0.36
query17	1.09	1.16	1.17
query18	0.21	0.20	0.20
query19	2.10	2.01	2.01
query20	0.02	0.01	0.02
query21	15.51	0.19	0.13
query22	4.83	0.06	0.05
query23	16.28	0.32	0.12
query24	3.04	0.60	0.40
query25	0.10	0.07	0.06
query26	1.03	0.23	0.16
query27	0.04	0.06	0.06
query28	2.50	0.73	0.38
query29	12.47	4.15	3.29
query30	0.28	0.14	0.10
query31	2.77	0.56	0.32
query32	3.22	0.59	0.51
query33	3.21	3.20	3.19
query34	15.34	4.00	3.41
query35	3.35	3.32	3.36
query36	0.57	0.43	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 95.71 s
Total hot run time: 24.51 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 85.00% (68/80) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 88.75% (71/80) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292

Problem Summary:

Follow-up fixes for the standalone minimum score filter (score() > x without
a score TopN):

1. Binding the score range replaced the scan's ANN TopN with an empty one, so
   WHERE content MATCH 'x' AND score() > 0.5 ORDER BY l2_distance_approximate()
   LIMIT k no longer used the ANN index and computed every distance by brute
   force. The scan applies the score range before the ANN search, so the
   binding now keeps the ANN order keys and limit. On a local cluster the
   profile of this query shows AnnIndexTopNSearchCnt=1 and no brute-force
   fallback, and the nearest row that fails the score filter is not returned.
2. score() has no input slots, so join rules treated score() > x like a
   constant. PushDownFilterThroughJoin copied it into both children,
   PushFilterInsideJoin turned it into a join condition and
   PushDownJoinOtherCondition then pushed it into the left child. A query with
   an IN or EXISTS subquery failed with a misleading "must contain MATCH"
   error, and an inner join silently filtered whichever table was on the left.
   The predicate now goes only into the child whose rows all reach the output
   of a left/right outer, semi or anti join, as ReorderJoin already does.
   Inner, cross and full joins keep it above the join, where CheckScoreUsage
   rejects it with a message that suggests a subquery; score() in a JOIN
   condition fails analysis.
3. test_bm25_score still expected an error for a score() > 0.5 filter without
   ORDER BY and LIMIT, which is now supported. It now checks that the row count
   matches the score TopN path.
4. EXPLAIN prints the pushed range (SCORE RANGE: score() > 0.5), because the
   predicate is no longer among the scan conjuncts.
5. The new filter rules are guarded by the cached containsType check, so
   filters without score() skip them.
6. SearchExpression.toSql keeps the options argument of search(dsl, options).

### Release note

score() filters work with IN and EXISTS subqueries and on the preserved side of
left or right outer joins. With inner, cross or full joins, or in a JOIN
condition, they fail with a clear error instead of filtering an arbitrary
table. MATCH queries with a score() filter and an ANN TopN keep using the ANN
index. EXPLAIN shows the pushed score range.

### Check List (For Author)

- Test: Unit Test and Regression test
    - FE unit tests: CheckScoreUsageTest, PushDownFilterThroughJoinTest,
      PushFilterInsideJoinTest, SearchExpressionTest, RewriteSearchToSlotsTest,
      PullUpProjectExprUnderTopNTest, CheckAnalysisTest, ReorderJoinTest
      (180 tests).
    - Regression tests on a local ASAN cluster: test_bm25_score,
      test_score_filter_usage, test_score_filter_matrix,
      test_bm25_score_range_filter, test_search_score_topn_predicates and six
      more score suites in inverted_index_p0, all 36 suites in search,
      ann_with_fulltext and ann_topn_small_candidate_fallback.
- Behavior changed: Yes. score() filters above inner, cross or full joins and
  in JOIN conditions are rejected; outer, semi and anti joins keep the filter
  on their output side; EXPLAIN shows SCORE RANGE.
- Does this need documentation: Yes, with the score() documentation for
  standalone minimum score filters.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converged in 2 review rounds. All three reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged final comment set; no unresolved static candidate or changed-path coverage gap remains.

Reviewed all 20 changed paths at 9c71f1f using static analysis only. One new P2 finding is inline: raw score predicates can cross a window before the new context validation and alter window results.

Existing review context was independently revalidated and deduplicated:

  • P1 #discussion_r4315061210 still applies: projected MATCH expressions contaminate the WHERE score. It remains an existing blocker and is not reposted.
  • P2 #discussion_r4315061221 still applies: score-containing CSE virtual columns retain raw Score calls and evaluate the placeholder.
  • P2 #discussion_r4315061234 is fixed at this head: the old expected-error regression became positive standalone/TopN count controls.

Critical checkpoint conclusions:

  • Goal and proof: the ordinary single-scan path materializes standalone minimum scores, retains hidden score slots, and reuses the existing TopN/range machinery. New FE and regression tests exercise it; the new and existing findings prevent a clean correctness verdict.
  • Scope and reuse: production changes are focused on ten FE files, using existing score metadata and BE machinery; no unrelated source changes found.
  • Concurrency and locking: no threads, locks, or shared mutable planner state are added. Plans are per-query; scan expression contexts are cloned per segment. No new lock-order or heavy-under-lock path found.
  • Lifecycle/static initialization: existing ScoreRuntime preparation/reader ownership is retained, and virtual aliases propagate to scan descriptors. No new resource ownership or cross-translation-unit static dependency.
  • Configuration: no new setting or dynamic-update requirement.
  • Compatibility: existing score sort/range protocol fields and shared local/cloud scan paths are reused; no persisted format or function ABI change. Cloud and mixed-version execution were not validated.
  • Thrift/protobuf IDs: no gensrc definitions changed.
  • Parallel paths: checked MATCH/SEARCH, existing TopN and standalone scoring, ordinary sorts/offsets, count/ANN consumers, outer/semi/anti versus inner/cross/full join placement, aggregates, CSE and window normalization. Window placement and known virtual-column interactions are the identified gaps.
  • Conditions/errors: supported GT/GE and reversed comparisons, one top-level range, FLOAT/DOUBLE casts, unsupported operators/lossy casts, overflow and missing scoring contexts have explicit checks. Late operator-context validation is incomplete as described inline.
  • Test coverage: inspected all four changed FE test files and all six changed regression suite/result files. Coverage includes V2/V3/SNII, null/no-hit/boundary cases, aliases/sorts, residual filters, cache/WAND, rowsets/deletes/MoW and ANN. Window placement, projected MATCH plus scoring, and score-containing CSE combinations need coverage.
  • Test results: new output blocks are consistent with paired controls and format-specific norm differences; ordered result comparisons and expected-error patterns are used. Output-generation provenance and runtime correctness cannot be independently proven by this static review. No builds or tests were run, as instructed.
  • Observability: SCORE RANGE in EXPLAIN preserves visibility of the removed predicate; no additional metrics requirement found.
  • Transactions/persistence: no EditLog, replay, version acquisition, failover or persisted metadata changes.
  • Writes/atomicity: no production data-write or publish/delete-bitmap mutation changes; existing visible-version/delete restrictions precede score materialization.
  • FE-BE variables: no new wire variable. Score order, zero limit, range and existing ANN metadata reach the existing reader/ScoreRuntime consumers; hidden score retention was verified.
  • Performance: zero limit deliberately computes all candidate scores with existing O(candidate rows) materialization. No additional substantiated performance regression found.
  • Other/core invariants: no new version, transaction visibility or ownership defect established; the identified score-binding/predicate-movement defects violate result correctness and unsupported-context failure behavior.

User focus: no additional focus was supplied. Review coverage is static; no runtime, cloud, or mixed-version validation is claimed.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #69292 (comment)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292

Problem Summary:

Fixes for the remaining review findings on standalone minimum score filters
(score() > x without a score TopN):

1. A MATCH in the select list is pushed into the scan as a virtual column.
   SegmentIterator::_apply_index_expr evaluated it with the same
   CollectionSimilarity as the WHERE predicates, so its BM25 score was added to
   every row's score before the range was applied. On a three-row table where
   the real score is 0.4700036,
   SELECT id, c1 MATCH_PHRASE 'apple' AS flag FROM t
   WHERE c1 MATCH 'apple' AND score() > 0.6
   returned two rows instead of none. Projection-only index evaluation now runs
   without the collection similarity, as VCompoundPred already does for its
   branches; scoring queries never use the index result cache, so the
   projection's cached bitmaps cannot answer a scoring query.
2. With experimental_enable_virtual_slot_for_cse, PushDownVirtualColumnsIntoOlapScan
   could extract a repeated expression such as abs(score()) + 1 into its own
   virtual column. That column kept the score() placeholder, which always
   returns zero, so sin(abs(score()) + 1) returned sin(1). Expressions that
   contain score() are no longer extracted; their score-free parts still are.
3. score() has no input slot, so filter pushdown treats it as a constant. An
   outer WHERE score() > x over a windowed subquery was pushed below the
   window and changed row_number() of the remaining rows; aggregations and
   LIMIT behaved the same way. CheckAnalysis now fails such a filter unless
   its input only passes rows of a scan through projections without window
   functions, filters, sorts, unions or the preserved side of an outer, semi
   or anti join.

### Release note

A score() filter above a window, an aggregation, a LIMIT or an inner, cross or
full join fails with a clear error. A MATCH in the select list no longer
changes which rows pass a score() filter.

### Check List (For Author)

- Test: Unit Test and Regression test
    - BE unit test: SegmentIteratorApplyIndexExprTest, including a new case
      that checks projections cannot see the collection similarity.
    - FE unit tests: CheckScoreUsageTest, PushDownVirtualColumnsIntoOlapScanTest,
      PushDownMatchProjectionAsVirtualColumnTest, PushDownFilterThroughJoinTest,
      PushFilterInsideJoinTest, SearchExpressionTest, RewriteSearchToSlotsTest,
      PullUpProjectExprUnderTopNTest, CheckAnalysisTest, ReorderJoinTest.
    - Regression tests on a local ASAN cluster: the score suites in
      inverted_index_p0 (including test_match_projection_virtual_column), all
      search suites, ann_with_fulltext and ann_topn_small_candidate_fallback.
- Behavior changed: Yes. score() filters above windows, aggregations, LIMIT and
  inner, cross or full joins are rejected; projected MATCH no longer adds to
  score().
- Does this need documentation: Yes, with the score() documentation for
  standalone minimum score filters.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One new P2 correctness finding: the new early score-context check accepts a CTE consumer without checking its producer. Post-analysis inlining can expose a window and allow the score filter to move below it, changing row numbers. The inline comment includes the reduced query, plan, production rewrite path, and fix direction.

Reviewed all 23 changed paths at head 230a892344681c6014ffc87ecb301e94764beada using the supplied merge-base diff. The existing projected-MATCH P1 (4315061210), CSE P2 (4315061221), stale regression P2 (4315061234), and direct derived-window P2 (4321752024) are addressed for their original triggering cases. The new inline describes a distinct CTE bypass in the newly added LogicalRelation branch; those existing findings are not reposted. No existing P0/P1 is independently confirmed still applicable, so the existing-blocking list is empty.

Critical checkpoint conclusions:

Checkpoint Conclusion
1. Goal and evidence Standalone minimum-score filters now bind to scan scoring without SQL TopN. Added plan/unit/regression coverage supports the intended shapes; the CTE-window context remains incorrect.
2. Scope and clarity Changes are focused on score validation, binding/materialization, projection scoring, SEARCH rendering, and corresponding tests.
3. Concurrency and locks Scoring context is segment-local, and expression evaluation is synchronous. No new threads, locks, heavy locked work, or lock-order issue found.
4. Lifecycle and static initialization The saved similarity shared_ptr survives the projection loop and DEFER restores it on every exit before score materialization. No new ownership cycle or cross-TU static dependency.
5. Configuration No configuration added. Tests exercise existing cache, WAND, CSE, and count settings.
6. Compatibility Existing optional score protocol fields and zero-limit full scoring are reused; no storage encoding or external symbol change. V2/V3/SNII retain their existing numerical differences. Mixed-version execution was not performed.
7. Thrift/protobuf IDs No gensrc definition changes; cross-branch field-ID comparison is inapplicable.
8. Parallel paths Traced MATCH/SEARCH, V2/V3/SNII, ordinary and score sorts/TopN, ANN, count, deletes/MoW, and shared local/cloud segment paths. No further verified path defect.
9. Conditional checks Error cases fail analysis and suppression has a concrete rationale. The new CTE acceptance condition is the reported exception.
10. Tests New Java and BE unit tests plus positive/negative SQL cover bindings, aliases/CSE, casts/boundaries, sort/offset, joins/UNION, counts, cache/WAND, deletes/MoW, and ANN. CTE-window coverage is missing. No tests or builds were run in this review.
11. Expected results Statically inspected all changed outputs: 146 matrix groups, 19 usage groups, 30 matching standalone/TopN control pairs, boundary behavior, format differences, and ANN/projection scores. No incorrect expected result found; generation provenance cannot be proved statically.
12. Observability EXPLAIN now retains the removed score-range predicate. Existing scan/profile counters remain usable; no additional metric need identified.
13. Transactions and persistence No production EditLog, metadata persistence, version publication, or failover path changes.
14. Writes and crash behavior No production write/atomicity path changes. Regression DML exercises existing visibility. The temporary similarity state restores on failure.
15. FE-to-BE propagation Logical/physical scans forward score order, zero limit, range, and ANN metadata. Translation preserves the hidden score slot through tuple pruning; lazy-materialization alternatives were also inspected.
16. Performance and memory Full scoring is required for minimum-score semantics; projection suppression avoids redundant BM25 collection. No additional significant allocation, redundant copy, or hot-path regression established.
17. Additional concerns Further sweeps covered post-analysis CTE/alias expansion, outer-join simplification, mark/semi/anti joins, compound scoring, caches, lazy slots, and ordinary LIMIT timing. No further independent finding remains unresolved.

No additional user focus was supplied. The release note describes the behavior change; the PR description acknowledges that the documentation PR is still pending.

Validation limits: static analysis only under the review contract; no builds, Doris test execution, fault injection, cloud deployment, or rolling-upgrade execution. No changed path, candidate, or risk focus remains without a disposition.

Review converged in 2 rounds: all three reviewers returned NO_NEW_VALUABLE_FINDINGS on the same frozen finding set; all candidates and risk focuses have evidence-backed dispositions.

Review converged after 2 round(s).

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69292

Problem Summary:

The analysis check for score() filters accepted a CTE consumer as a plain
relation. CTEInline expands a single-use CTE after analysis, so for

    WITH w AS (SELECT id, row_number() OVER (ORDER BY id) AS rn FROM t
               WHERE c1 MATCH 'apple')
    SELECT * FROM w WHERE score() > 0.5

the window only became visible after the check, and filter pushdown moved
score() > 0.5 below it, which changed rn of the remaining row. The check now
follows a CTE consumer to its analyzed producer and applies the same row rules
there. A recursive CTE reads itself and is rejected.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression test
    - FE unit tests: CheckScoreUsageTest (a CTE over a window is rejected, a
      CTE over one scan is bound) and the related rule tests.
    - Regression tests on a local ASAN cluster: test_score_filter_usage, the
      score suites in inverted_index_p0, all search suites, ann_with_fulltext
      and ann_topn_small_candidate_fallback.
- Behavior changed: Yes. A score() filter on a CTE whose producer contains a
  window, an aggregation, a LIMIT or an inner join is rejected.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@airborne12

Copy link
Copy Markdown
Member Author

/review

@airborne12
airborne12 merged commit 2524893 into apache:master Oct 23, 2026
53 of 54 checks passed
@airborne12
airborne12 deleted the fix-score-filter-without-topn branch October 23, 2026 10:36
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…#69399 (#69560)

### What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to `branch-4.2`.

A CTE fails to plan with "A expression contains slot not from children"
when the optimizer inlines it after one of its scans got a virtual
column: a CTE that orders by `score()` or by an ANN distance with a
LIMIT, or that has MATCH in its select list and is inlined after
rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids,
copied them in list order, kept the old slot in the output of the copied
scan and left the score and ANN order keys on the old slots. The copier
now maps every virtual column to its new id before copying their
expressions and remaps the order keys, and
`LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in
the output.

#### Branch adaptations

The branch has no table streams, so the change to
`LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the
constructor call of the branch, `LogicalPlanDeepCopier` imports
`LogicalOlapScan`, which master already imported, and
`LogicalPlanDeepCopierTest` keeps the tests of the branch and adds the
two tests of #69399.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by `score()` or by an ANN distance with a LIMIT, or that has MATCH in
its select list and is inlined after rewrite.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…69292 (#69562)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit
of #69560, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69560
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps
the behavior of the branch and skips the push-down instead of failing.
- `SearchDslParser` is imported from its package on the branch.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…cross column equalities #69419 (#69561)

### What problem does this PR solve?

Related PR: #69419

Problem Summary: Backport #69419 to `branch-4.2`.

Predicate inference copied a predicate on one column to every column
that is equal to it, also when the predicate contained MATCH,
multi_match or SEARCH. Their result depends on the inverted index of the
column, not only on its value: with `c1 = c2` and different parsers on
the two columns, the copy removed matching rows, and with the same index
it doubled `score()`, so a minimum score let low-scored rows through.
`InferPredicateByReplace.infer` now skips every input that contains
`Match`, `MultiMatch`, `Search` or `SearchExpression`.

#### Branch adaptations

The branch does not have the `NoneMovableFunction` check that master has
next to the new check, so only the new check is added. The test imports
`SearchDslParser` from its package on the branch.

### Release note

Fix wrong results and scores when predicate inference copied a MATCH,
multi_match or SEARCH predicate to an equal column.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. Predicate inference no longer copies a predicate that
contains MATCH, multi_match or SEARCH to an equal column.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 24, 2026
### What problem does this PR solve?

Related PR: apache#69292

Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at
random for each session. With it off, MATCH is not evaluated with the
inverted index in the scan, so no BM25 score is collected and score()
returns 0. test_score_filter_usage and test_score_filter_matrix got no
rows for a minimum score, and test_cte_inline_virtual_column got score
0.0 for its TopN. Like the score suites of the branch, these suites set
enable_common_expr_pushdown = true.

### Release note

None

### Check List (For Author)

- Test: Regression test. With use_fuzzy_session_variable = true and
the pull request number of CI, the three suites failed in sessions
that drew enable_common_expr_pushdown = false (1 to 3 suites per run);
with the setting pinned, four runs passed.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…#69399 (#69563)

### What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to `branch-4.1`.

A CTE fails to plan with "A expression contains slot not from children"
when the optimizer inlines it after one of its scans got a virtual
column: a CTE that orders by `score()` or by an ANN distance with a
LIMIT, or that has MATCH in its select list and is inlined after
rewrite. `LogicalPlanDeepCopier` gave each virtual column two ids,
copied them in list order, kept the old slot in the output of the copied
scan and left the score and ANN order keys on the old slots. The copier
now maps every virtual column to its new id before copying their
expressions and remaps the order keys, and
`LogicalOlapScan.withVirtualColumns` replaces the old virtual slots in
the output.

#### Branch adaptations

The branch has no table streams, so the change to
`LogicalOlapTableStreamScan` is dropped. `withVirtualColumns` keeps the
constructor call of the branch, and `LogicalPlanDeepCopierTest` and
`CTEInlineTest` keep the tests of the branch and add the tests of
#69399. `test_cte_inline_virtual_column` sets
`enable_common_expr_pushdown = true`, as the score suites of the branch
do: CI on branch-4.1 draws it at random for each session, and without it
MATCH is not evaluated with the index in the scan and `score()` is 0.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by `score()` or by an ANN distance with a LIMIT, or that has MATCH in
its select list and is inlined after rewrite.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…cross column equalities #69419 (#69564)

### What problem does this PR solve?

Related PR: #69419

Problem Summary: Backport #69419 to `branch-4.1`.

Predicate inference copied a predicate on one column to every column
that is equal to it, also when the predicate contained MATCH,
multi_match or SEARCH. Their result depends on the inverted index of the
column, not only on its value: with `c1 = c2` and different parsers on
the two columns, the copy removed matching rows, and with the same index
it doubled `score()`, so a minimum score let low-scored rows through.
`InferPredicateByReplace.infer` now skips every input that contains
`Match`, `MultiMatch`, `Search` or `SearchExpression`.

#### Branch adaptations

The branch does not have the `NoneMovableFunction` check that master has
next to the new check, so only the new check is added. The test imports
`SearchDslParser` from its package on the branch.
`infer_match_predicate` sets `enable_common_expr_pushdown = true`, as
the score suites of the branch do: CI on branch-4.1 draws it at random
for each session, and without it MATCH is not evaluated with the index
in the scan and `score()` is 0.

### Release note

Fix wrong results and scores when predicate inference copied a MATCH,
multi_match or SEARCH predicate to an equal column.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. Predicate inference no longer copies a predicate that
contains MATCH, multi_match or SEARCH to an equal column.
- Does this need documentation?
    - [x] No.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…69292 (#69565)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit
of #69563, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69563
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- The branch evaluates MATCH projections with `evaluate_inverted_index`
directly, and that loop runs without the score collector.
- `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch
for `limit + offset`, which fails like master does.
- `SearchDslParser` and `CollectionSimilarity` are included from their
packages on the branch.
- The branch has no SNII storage format, so `BindScoreToOlapScanTest`
and `test_score_filter_matrix` cover V2 and V3, and the `.out` of
`test_score_filter_matrix` is regenerated without the SNII cases.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.
- `test_score_filter_usage`, `test_score_filter_matrix` and
`test_cte_inline_virtual_column` set `enable_common_expr_pushdown =
true`, as the score suites of the branch do: CI on branch-4.1 draws it
at random for each session, and without it MATCH is not evaluated with
the index in the scan and `score()` is 0.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Nov 6, 2026
### Background

SEARCH fields are bound during analysis before score binding runs in rewrite. Bound SEARCH predicates use SearchExpression instead of the scalar Search function.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69344 (early SEARCH binding), apache#69292 (score binding)
PR Type: Bugfix

Problem Summary: Valid SEARCH queries that call score() are rejected because score binding only recognizes the unbound Search type. Recognize SearchExpression in the WHERE predicate check, scoring alias propagation and HAVING/QUALIFY validation.

### Details

- **Frequency:** SEARCH scoring queries after early SEARCH binding.
- **Root cause:** The downstream score binder checks the expression type from the previous rule ordering.
- **Why existing tests missed it:** The targeted SEARCH suites did not cover score(); existing score regression suites expose the failure.
- **Fix:** Replace all three Search type checks with SearchExpression and update the import.
- **Other paths and branches:** MATCH recognition and score usage restrictions remain unchanged.
- **Behavior change:** Valid SEARCH scoring queries can pass score binding.

### Testing

FE Checkstyle and git diff --check passed. Built FE with DISABLE_BUILD_UI=ON ./build.sh --fe and deployed it to the development cluster.

The five targeted FE test classes executed 147 tests with zero failures, errors or skips, including 82 BindScoreToOlapScanTest cases.

Ran 21 targeted regression suites: 20 passed, one failed because the existing BE lacks the keyword MATCH scoring change from apache#69790. Its actual 0.290716 score matches the BE-version expected output, while the current expected output is 0.983863. Re-running that suite against the existing generated expected output from the running BE commit passed. Repository expected outputs were not changed. The previous SEARCH score binding error was not reproduced.

The initial local FE unit-test attempt stopped during dependency resolution due to Maven-cache sandbox permissions; remote tests provide the validation above.

### Release note

Fix planning failures when using score() with SEARCH predicates.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: current keyword MATCH expected scores require a BE containing apache#69790; validation used the existing BE

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Code and commit description
- Human review: Pending
morningman added a commit to apache/doris-website that referenced this pull request Nov 13, 2026
Doris 4.0.0 through 4.0.3 reject `score()` predicates in `WHERE`, while
4.0.4 and later support direct minimum-score filtering. Document both
query forms in the English and Chinese dev/4.x relevance guides, with an
indexed sample table and input data. Explain supported score predicates,
threshold semantics, and the candidate limit required by the
older-version workaround. Fixes #3264.

The dev guides also describe the current implementation: an outermost
single-table query can compute and filter scores without Top-N, while a
scored SELECT nested in a subquery, view, CTE, or set operation still
needs score ordering and LIMIT. Update the activation conditions and FAQ
to match. The 4.x guides retain the Top-N requirement verified for 4.0.x
and 4.1.0 through 4.1.5.

Validation:
- Compared the score-filter implementation in the 4.0.0 through 4.0.4
release tags and 4.1.5-release, checked the 4.0.4 release notes and
[#60997](apache/doris#60997), and checked
current master’s `BindScoreToOlapScan` and regression examples from
[#69292](apache/doris#69292).
- All four pages pass MDX parsing after legacy HTML-comment removal,
front matter preservation, English/Chinese SQL example comparison, link
checks, locale/version sync checks, and `git diff --check`.
- No live cluster SQL or full website build was run. Existing page
routes, front matter, and navigation are preserved.

---------

Co-authored-by: morningman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.6-merged dev/4.2.0-merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants