What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to branch-4.1. It contains the commit of #69563, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69563 first.

score() only worked as ORDER BY score() LIMIT n directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound score() placeholder as zero and silently dropped every row:

SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing

BindScoreToOlapScan, which replaces CheckScoreUsage, binds score() to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and PushDownScoreTopNIntoOlapScan still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating Score(), and MATCH projections do not add to the score. See #69292 for the supported and rejected uses.

Branch adaptations

• The SegmentIterator of the branch collects the columns of pushed-down expressions in _extract_common_expr_columns instead of per-column states, so the score column is marked there, and the new BE test checks _common_expr_columns and _is_common_expr_column. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read.
• The branch evaluates MATCH projections with evaluate_inverted_index directly, and that loop runs without the score collector.
• PushDownScoreTopNIntoOlapScan keeps the overflow check of the branch for limit + offset, which fails like master does.
• SearchDslParser and CollectionSimilarity are included from their packages on the branch.
• The branch has no SNII storage format, so BindScoreToOlapScanTest and test_score_filter_matrix cover V2 and V3, and the .out of test_score_filter_matrix is regenerated without the SNII cases.
• The branch does not have [fix](cache) Share cache eligibility for nondeterministic expressions #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, test_score_filter_matrix lost the rows of the first rowset. VirtualSlotRef::get_digest returns 0 as on master, the part of [fix](cache) Share cache eligibility for nondeterministic expressions #68672 that this needs, with the VirtualSlotRef tests of [fix](cache) Share cache eligibility for nondeterministic expressions #68672 that the branch can run.
• test_score_filter_usage, test_score_filter_matrix and test_cte_inline_virtual_column set enable_common_expr_pushdown = true, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and score() is 0.

Release note

score() works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score score() > x or score() >= x, without ORDER BY score() LIMIT. A scored SELECT in a subquery, view, CTE or set operation still needs ORDER BY score() LIMIT. Other uses of score() are rejected instead of evaluating it as 0.

Check List (For Author)

• Test
  • Regression test
  • Unit Test
  • Manual test
• Behavior changed:
  • Yes. A minimum score works without ORDER BY score() LIMIT, and unsupported uses of score() fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs.
• Does this need documentation?
  • Yes. The score() documentation should describe where score() can be used.

Validation

Validated together with the other two backports (#69399, #69419 and #69292 on branch-4.1 at 3f911d0), in a local ASAN build:

• PASS: FE unit tests LogicalPlanDeepCopierTest, CTEInlineTest, InferPredicateByReplaceTest, InferPredicatesTest, UnequalPredicateInferTest, BindScoreToOlapScanTest, SearchExpressionTest, TopNRuntimeFilterTest and MaterializedViewUtilsTest (228 tests), and FE Checkstyle.
• PASS: ./build.sh --be and ./build.sh --fe.
• PASS: BE unit tests SegmentIteratorScoreCommonExprTest, SegmentIteratorApplyIndexExprTest and VirtualSlotRefTest (31 tests); clang-format 16 on the changed BE files.
• PASS: the 16 regression suites of the branch that use score(), including test_cte_inline_virtual_column, infer_match_predicate, test_score_filter_usage and test_score_filter_matrix. The framework regenerated the .out of test_score_filter_matrix on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other .out files are unchanged.
• PASS: with the fuzzy session variables of CI (use_fuzzy_session_variable = true and its pull request number), the score suites failed in sessions that drew enable_common_expr_pushdown = false; with the setting pinned, four runs of test_score_filter_usage, test_score_filter_matrix and test_cte_inline_virtual_column, and four of infer_match_predicate, passed.

🤖 Generated with Claude Code