…he#62222)

This pull request introduces significant improvements to scan operator
logic, particularly enhancing the correctness and efficiency of LIMIT
and predicate pushdown in OLAP scans. The main changes include a robust
mechanism for sharing LIMIT counters among scanners, more precise
control over predicate pushdown, and stricter validation for residual
predicates. These changes help ensure that queries with LIMIT and TopN
semantics return accurate results and avoid incorrect or inefficient
execution paths.

**Improvements to LIMIT handling and scanner coordination:**

- Introduced a shared atomic counter (`_shared_scan_limit`) for LIMIT
queries, allowing all scanners to collectively respect the global row
limit. Scanners now check and update this counter to stop early when the
LIMIT is reached, preventing over-scanning and improving efficiency.
TopN scans bypass this mechanism as required.
[[1]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR50-R55)
[[2]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR135-R143)
[[3]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR177-R183)
[[4]](diffhunk://#diff-7fe2376680f9ab835053a0a93c7f9458e6d526acfc180318be0ad936e0c2dedeR196)
[[5]](diffhunk://#diff-9c777f60e1d076777f042a0b702967bd5864bcfe9ee1eee9b456bb67c6fdf817R266-R269)
[[6]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R71)
[[7]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R239)
[[8]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262L283-R285)
[[9]](diffhunk://#diff-3c4794a864169735a628d8fcb1de986523a9516aad318aa0a893913f4ce031d4R1043-R1050)

- Adjusted scanner and scan local state interfaces to support the new
shared LIMIT mechanism, including new virtual methods and atomic counter
management.
[[1]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R71)
[[2]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262R239)
[[3]](diffhunk://#diff-a45108106b12759815ac5991d56a9d6ead7ffb9c915fb0cf7b6f0b8123b30262L283-R285)

**Predicate pushdown and validation enhancements:**

- Refactored `_should_push_down_common_expr` to require the expression
as an argument and added logic to ensure only eligible expressions are
pushed down, considering storage merge requirements and key columns.
[[1]](diffhunk://#diff-780cdfbcbb1d30adb63c61521e6b681cad347bebeaa5332a781d05bb51543527L46-R46)
[[2]](diffhunk://#diff-235cc51f4698ebf0ebe796e7253912907bb7c1caf4efcac92538ac1ff2eac171L104-R107)
[[3]](diffhunk://#diff-3ddc75656071d9c0e6b0be450e152a1c94559f7e70ea820e7f0c80a7078e3292L465-R547)
[[4]](diffhunk://#diff-3c4794a864169735a628d8fcb1de986523a9516aad318aa0a893913f4ce031d4L313-R313)

- Added a validation step (`validate_residual_scan_conjuncts`) to
prevent unsupported residual predicates (like SEARCH or disabled MATCH)
and to ensure correctness when using COUNT_ON_INDEX pushdown.

**OLAP scan and segment limit logic improvements:**

- Improved the logic for pushing down segment limits and TopN
optimizations, ensuring that such optimizations are only enabled when
all necessary conditions are met (e.g., no residual predicates, no
runtime filters, storage does not require merging). The code now asserts
correct states and disables shared LIMITs for TopN scans as appropriate.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bL446-R502)
[[2]](diffhunk://#diff-3ddc75656071d9c0e6b0be450e152a1c94559f7e70ea820e7f0c80a7078e3292R1155-R1156)

**Code cleanup and maintenance:**

- Removed unused or obsolete logic related to filter block conjuncts and
the old limit quota acquisition method.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bL90)
[[2]](diffhunk://#diff-0c9a817d45d8130ea3211189e1321d1275e22fd4a9a3fac2bd707b1cfeefa5e5L115-L135)

**Dependency and include updates:**

- Added missing includes for expression types to support new logic.


```sql
CREATE DATABASE IF NOT EXISTS lm_bench;
USE lm_bench;

DROP TABLE IF EXISTS lm_wide_fact_10m;
CREATE TABLE lm_wide_fact_10m (
    id BIGINT NOT NULL,
    filter_key INT NOT NULL,
    sort_key BIGINT NOT NULL,
    metric BIGINT NOT NULL,
    payload_a VARCHAR(1024) NOT NULL,
    payload_b VARCHAR(1024) NOT NULL,
    long_payload_a VARCHAR(4096) NOT NULL,
    long_payload_b VARCHAR(4096) NOT NULL
)
ENGINE=OLAP
DUPLICATE KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 32
PROPERTIES (
    "replication_num" = "1"
);

INSERT INTO lm_wide_fact_10m
SELECT
    number AS id,
    CAST(number % 1000000 AS INT) AS filter_key,
    (number * 2654435761) % 1000000007 AS sort_key,
    (number * 1103515245 + 12345) % 2147483647 AS metric,
    REPEAT(CONCAT('payload_a_', CAST(number AS STRING), '_',
        CAST((number * 1315423911) % 1000000007 AS STRING), '_'), 16) AS payload_a,
    REPEAT(CONCAT('payload_b_', CAST(number AS STRING), '_',
        CAST((number * 2654435761) % 1000000009 AS STRING), '_'), 16) AS payload_b,
    REPEAT(CONCAT('long_payload_a_', CAST(number AS STRING), '_',
        CAST((number * 48271) % 2147483647 AS STRING), '_'), 64) AS long_payload_a,
    REPEAT(CONCAT('long_payload_b_', CAST(number AS STRING), '_',
        CAST((number * 69621) % 2147483629 AS STRING), '_'), 64) AS long_payload_b
FROM numbers("number" = "10000000");

ANALYZE TABLE lm_wide_fact_10m WITH SYNC;


-- ============================================================
-- Case 1: TopN next + LIMIT + high-filter-rate pushed-down predicate
-- Expected observation:
--   filter_key is a narrow pushed-down predicate and keeps about 1% rows.
--   TopN works on narrow metric/sort_key, while long strings are final output.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
WHERE filter_key < 10000
ORDER BY metric DESC, sort_key ASC
LIMIT 10;
-- Result: 0.12 sec -> 0.10 sec

-- ============================================================
-- Case 2: Normal scan + LIMIT
-- Expected observation:
--   No ORDER BY and no predicate. This isolates ordinary scan limit behavior
--   while still returning long string columns.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
LIMIT 10;
-- Result: 0.06 sec -> 0.01 sec

-- ============================================================
-- Case 3: Normal scan + LIMIT + high-filter-rate pushed-down predicate
-- Expected observation:
--   No ORDER BY. This isolates scan predicate + limit behavior with long
--   string result columns and a predicate that filters about 99% of rows.
-- ============================================================

SELECT
    id,
    filter_key,
    metric,
    sort_key,
    payload_a,
    payload_b,
    long_payload_a,
    long_payload_b
FROM lm_wide_fact_10m
WHERE filter_key < 10000
LIMIT 10;
-- Result: 0.07 sec -> 0.02 sec

```

---------

Co-authored-by: Copilot <[email protected]>

### Backport notes (branch-4.2)

Partial pick. This branch keeps the enable_common_expr_pushdown switch and has no
shared scan LIMIT counter (apache#61617 was reverted here by apache#62191), so:

- Not picked: scanner.cpp/.h, scanner_context.cpp/.h and scanner_scheduler.cpp (the
  shared counter), ScanLocalState::shared_scan_limit_ptr(), and the
  collection_statistics_test.cpp hunk (the file does not exist on this branch).
- Not picked: pushing key-only common expressions into SegmentIterator for tables
  merged on read (AGG, UNIQUE without merge-on-write). _should_push_down_common_expr()
  keeps the rule this branch had: switch on, no storage merge, expression acts on a
  slot (VExpr::is_acting_on_a_slot stays). The follow-up fix apache#63279 was only needed by
  that pushdown and is not picked either.
- RuntimeState keeps enable_common_expr_pushdown() and
  enable_common_expr_pushdown_for_inverted_index(). The OlapScanner assertion that
  every slot-based conjunct was pushed down only runs with the switch on. The thrift
  comment on field 107 is unchanged.
- Not picked: validate_residual_scan_conjuncts (the SEARCH / MATCH / COUNT_ON_INDEX
  residual-predicate errors). This branch still has enable_common_expr_pushdown, and
  fuzzy sessions and existing tests such as test_compound_1 turn it off. Those
  predicates then stay in the scanner, which this branch evaluates correctly, so the
  errors would reject valid queries. On master the switch is gone (apache#64285) and the
  tests no longer turn it off (apache#64479).
- Key TopN on tables merged on read keeps the behavior this branch had: without
  runtime filters the scanner sets read_orderby_key* and keeps _limit = sort_limit, so
  it stops after its own N key-ordered rows (master resets _limit to -1 for them).
  Storage TopN (read_orderby_key_limit) is only set for physical DUP_KEYS and
  merge-on-write UNIQUE_KEYS tablets; for a MOR table read as DUP the reader does not
  prepare the key columns that VCollectIterator::_topn_next() compares, so that case
  falls back to reading every row.
- TQueryOptions field id is 231; 219 and 230 are already used on this branch. apache#68785
  gives master the same id.
- SessionVariable keeps the VariableMgr.VarAttr annotation style. The "Legacy session
  variable" comment is not added.
- segment_iterator.cpp: the count fast path check stays ahead of the read_limit EOF
  check, the shrink_char_type_column_suffix_zero() call does not exist here, and
  count_on_index_fastpath's has_read_limit now reads _opts.read_limit (the SNII
  backport apache#68584 pinned it to false because this branch had no read_limit yet).
- segment_iterator_limit_opt_test builds its Schema from (columns, col_ids); this
  branch has no Schema(TabletSchemaSPtr) constructor.

Verified on this branch: BE (ASAN) and FE build with 0 checkstyle violations; the 23
related BE UT suites (183 cases, including SegmentIteratorLimitOptTest) pass; the
regression suites query_p0/limit (18 suites), test_search_slash_in_term,
infer_predicate and test_variant_predefine_index_type pass against the picked .out
files. Without validate_residual_scan_conjuncts, the suites that failed in CI with
residual-predicate errors pass with enable_common_expr_pushdown both off and on, except
test_multi_tokenize_index_not_built and test_search_score_topn_predicates, which fail
with the switch off on this branch's own errors (E-6001, SearchExpr without inverted
index), not on anything from this pick. The kept scanner-level stop was measured on a Release build: on an AGG table
built from 4M input rows, ORDER BY key LIMIT 10 scans 131k rows in 19 ms; the same
binary with master's behavior (_limit reset to -1 for tables merged on read) scans 2.9M
rows in 114 ms. A MOR UNIQUE table goes 98k rows / 20 ms against 2.2M rows / 84 ms.
Results are identical.

(cherry picked from commit 28cc2bf)

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

# Conflicts:
#	gensrc/thrift/PaloInternalService.thrift

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# Author:    Pxl <[email protected]>
#
# interactive rebase in progress; onto cabcf45
# Last commands done (2 commands done):
#    pick f6d4a8d4292 # [Improvement](scan) update limit pushdown to tablet reader (apache#61713)
#    pick 428d832fcb0 # [Improvement](scan) support push down limit to segment iterator (apache#62222)
# No commands remaining.
# You are currently rebasing branch 'pick-4.2/segment-limit-pushdown' on 'cabcf45c1af'.
#
# Changes to be committed:
#	modified:   be/src/exec/operator/mock_scan_operator.h
#	modified:   be/src/exec/operator/olap_scan_operator.cpp
#	modified:   be/src/exec/operator/olap_scan_operator.h
#	modified:   be/src/exec/operator/scan_operator.cpp
#	modified:   be/src/exec/operator/scan_operator.h
#	modified:   be/src/exec/scan/olap_scanner.cpp
#	modified:   be/src/runtime/runtime_state.h
#	modified:   be/src/storage/iterator/vcollect_iterator.cpp
#	modified:   be/src/storage/iterator/vcollect_iterator.h
#	modified:   be/src/storage/iterators.h
#	modified:   be/src/storage/rowset/beta_rowset_reader.cpp
#	modified:   be/src/storage/rowset/rowset_reader_context.h
#	modified:   be/src/storage/segment/segment_iterator.cpp
#	modified:   be/src/storage/segment/segment_iterator.h
#	modified:   be/src/storage/tablet/tablet_reader.cpp
#	modified:   be/src/storage/tablet/tablet_reader.h
#	new file:   be/test/storage/segment/segment_iterator_limit_opt_test.cpp
#	modified:   fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
#	modified:   gensrc/thrift/PaloInternalService.thrift
#	new file:   regression-test/data/query_p0/limit/test_unified_limit_pushdown.out
#	modified:   regression-test/data/search/test_search_slash_in_term.out
#	modified:   regression-test/data/variant_p0/predefine/test_predefine_type_index.out
#	modified:   regression-test/suites/query_p0/limit/test_general_limit_pushdown.groovy
#	new file:   regression-test/suites/query_p0/limit/test_unified_limit_pushdown.groovy
#	modified:   regression-test/suites/search/test_search_slash_in_term.groovy
#	modified:   regression-test/suites/variant_p0/predefine/test_predefine_type_index.groovy
#
# Untracked files:
#	be/dict/kuromoji/
#	ui/tsconfig.app.tsbuildinfo
#	ui/tsconfig.node.tsbuildinfo
#

### What problem does this PR solve?

Backport to branch-4.2 of how master dropped enable_common_expr_pushdown and
enable_common_expr_pushdown_for_inverted_index (apache#62222, apache#64285, apache#64479).

- BE: RuntimeState::enable_common_expr_pushdown() and
  enable_common_expr_pushdown_for_inverted_index() are gone. Tables not merged on
  read always push slot-based conjuncts into SegmentIterator, the OlapScanner check
  that every such conjunct was pushed runs whenever segment LIMIT can be pushed, and
  SegmentIterator no longer returns early from
  _check_all_conditions_passed_inverted_index_for_column.
- BE: validate_residual_scan_conjuncts from apache#62222 comes back. A residual SEARCH, a
  residual MATCH with enable_match_without_inverted_index off, and COUNT_ON_INDEX with
  residual scan conjuncts now fail with INVALID_ARGUMENT. The SEARCH and MATCH messages
  also say that tables merged on read do not push expressions on this branch.
- FE: the two session variables are removed (apache#64285). VariableMgr takes
  REMOVED_SESSION_VAR_NAMES from apache#63744, listing only these two names, so SET,
  SET GLOBAL, SET_VAR and SELECT @@ on them still succeed and do nothing.
- FE: fuzzy mode now randomizes enable_segment_limit_pushdown (apache#64479).
- thrift: field 65 is marked deprecated. No field id changes.
- Regression cases follow apache#64479: a setting of either removed variable becomes
  enable_segment_limit_pushdown with the same value; two settings in one SET_VAR or on
  two adjacent SET lines become one, taking the enable_common_expr_pushdown value when
  they differ. 214 files take apache#64479's hunks without its trailing-whitespace cleanup,
  57 files that apache#64479 misses or that differ here are rewritten by the same rules (one
  file gets both), and 12 files keep the removed names exactly as master has them today.

Upgrade BE before FE. An old BE behind a new FE no longer receives
enable_common_expr_pushdown, reads it as false, and keeps conjuncts in the scanner
until it is upgraded.

Verified on this branch (ASAN BE):
- FE and BE build, 0 checkstyle violations, BE changes clean under clang-format 16.
- FE UT SessionVariablesTest, VariableMgrTest and SessionVariableBackendSelectionTest
  (42 cases) pass. The 23 related BE UT suites (183 cases, including the SegmentIterator
  apply-index-expr, short-circuit, expr-zonemap, candidate-pushdown and no-need-read-data
  suites) pass.
- SET, SET GLOBAL, SET_VAR and SELECT @@ on both removed names succeed; SELECT @@ returns
  an empty string, SHOW VARIABLES lists neither, and an unknown name still fails.
- With enable_segment_limit_pushdown set globally to false and then true:
  query_p0/limit (18 suites), test_search_slash_in_term, test_variant_predefine_index_type,
  infer_predicate, and 69 more suites (the ones that failed on residual predicates in this
  PR's earlier CI, the ones that turned the old switch off, the ones that still set the
  removed names, and migrated SNII and gram cases) pass, except
  test_search_score_topn_predicates. That suite fails the same way in both runs, and CI
  mutes it on other branch-4.2 PRs (apache#68213, apache#68774): the FE reports the score() usage
  error before the LIMIT overflow error the case expects.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve?

Pick of master PR apache#68804 (be261b7, still in review) to branch-4.2.

A merge-on-read UNIQUE table listed in read_mor_as_dup_tables is read without merging, so
OlapScanLocalState::_storage_no_merge() is true for it, but
TabletReader::_init_orderby_keys_param() sets up the key columns that
VCollectIterator::_topn_next() compares only for DUP_KEYS and merge-on-write tablets. The
apache#62222 pick on this branch already keeps segment LIMIT off for a key TopN on such a table,
through a separate condition next to can_push_down_segment_limit. This commit switches to
apache#68804's form of the same rule:

- OlapScanner folds storage TopN support into can_push_down_segment_limit, so the
  DORIS_CHECK that guards a pushed LIMIT also covers it. The behavior does not change: the
  scan returns every row and the TopN operator above sorts them.
- TabletReader::_init_orderby_keys_param() returns INTERNAL_ERROR when storage TopN is
  requested but no order-by key columns are set up, so a wrong pushdown fails the query at
  reader init instead of aborting the BE in _topn_next().

Branch-4.2 notes: be/test/storage/tablet_reader_test.cpp does not exist on this branch, so it
is added with only the two TopN cases from apache#68804 and a fixture that sets _return_columns
(this branch has no ReadSchema). The regression case and its .out are taken as is.

Verified on this branch (ASAN BE):
- BE builds, and the changes are clean under clang-format 16.
- 24 related BE UT suites (185 cases) pass, including the two new TabletReaderTest cases:
  storage TopN on a merge-on-read tablet, or with an empty key prefix, fails
  _init_orderby_keys_param() with INTERNAL_ERROR.
- test_read_mor_as_dup_key_topn, test_read_mor_as_dup and query_p0/limit (18 suites) pass
  with enable_segment_limit_pushdown set globally to false and then true.

Co-Authored-By: Claude Opus 5.5 <[email protected]>