Skip to content

Commit afb1601

Browse files
authored
[Improvement](scan) update limit pushdown to tablet reader (apache#61713)
This pull request implements a general limit pushdown optimization for storage-layer scans on DUP_KEYS and UNIQUE_KEYS (with Merge-on-Write, MOW) tables. The optimization ensures that LIMIT clauses are enforced after filtering, preventing the scan from returning fewer rows than requested when filters are present. The change is carefully scoped to avoid interfering with existing optimizations (like TopN or storage merge paths) and includes comprehensive regression tests to validate correctness across various scenarios. **General Limit Pushdown Implementation:** * Added a `general_read_limit` parameter to the storage layer (`TabletReader`, `RowsetReaderContext`, and `VCollectIterator`) to enforce a post-filter row limit in non-merge scan paths for DUP_KEYS and UNIQUE_KEYS with MOW. The iterator now tracks the number of rows returned and stops reading once the limit is reached, ensuring correct LIMIT semantics even in the presence of filters. [[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR494-R503) [[2]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR518-R531) [[3]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35L95-R102) [[4]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35R258-R294) [[5]](diffhunk://#diff-b7a74d17e270719c904e8c31fb1308cb82ec0b89459a245dce9cb5505021ac63R353-R358) [[6]](diffhunk://#diff-0025ea2b49c400923787cd77990575f94ba5d925efc95cb16336fbaec92ebfb1R106-R109) [[7]](diffhunk://#diff-979b99e23ca7a4bb73f5beaa00270494778545296929a3de64b2e87dc8299cd6R202-R203) [[8]](diffhunk://#diff-48c14ad6bacc0a0ed497ad0cde34756af85aa98dcb7dc909609a288a503be0c9R216-R220) **Testing and Validation:** * Added a new regression test suite (`test_general_limit_pushdown.groovy`) covering DUP_KEYS, UNIQUE_KEYS (MOW), AGG_KEYS, and MOR tables. The tests verify correct row counts and output for LIMIT queries with and without filters, as well as negative cases where the optimization must not apply. * Added corresponding expected output results for the new tests.
1 parent 92f75f3 commit afb1601

8 files changed

Lines changed: 514 additions & 16 deletions

File tree

‎be/src/exec/scan/olap_scanner.cpp‎

Lines changed: 35 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -491,20 +491,41 @@ Status OlapScanner::_init_tablet_reader_params(
491491
_tablet_reader_params.enable_mor_value_predicate_pushdown = true;
492492
}
493493

494-
// order by table keys optimization for topn
495-
// will only read head/tail of data file since it's already sorted by keys
496-
if (olap_scan_node.__isset.sort_info && !olap_scan_node.sort_info.is_asc_order.empty()) {
497-
_limit = _local_state->limit_per_scanner();
498-
_tablet_reader_params.read_orderby_key = true;
499-
if (!olap_scan_node.sort_info.is_asc_order[0]) {
500-
_tablet_reader_params.read_orderby_key_reverse = true;
501-
}
502-
_tablet_reader_params.read_orderby_key_num_prefix_columns =
503-
olap_scan_node.sort_info.is_asc_order.size();
504-
_tablet_reader_params.read_orderby_key_limit = _limit;
505-
506-
if (_tablet_reader_params.read_orderby_key_limit > 0 &&
507-
olap_scan_local_state->_storage_no_merge()) {
494+
// Skip topn / general-limit storage-layer optimizations when runtime
495+
// filters exist. Late-arriving filters would re-populate _conjuncts
496+
// at the scanner level while the storage layer has already committed
497+
// to a row budget counted before those filters, causing the scan to
498+
// return fewer rows than the limit requires.
499+
if (_total_rf_num == 0) {
500+
// order by table keys optimization for topn
501+
// will only read head/tail of data file since it's already sorted by keys
502+
if (olap_scan_node.__isset.sort_info &&
503+
!olap_scan_node.sort_info.is_asc_order.empty()) {
504+
_limit = _local_state->limit_per_scanner();
505+
_tablet_reader_params.read_orderby_key = true;
506+
if (!olap_scan_node.sort_info.is_asc_order[0]) {
507+
_tablet_reader_params.read_orderby_key_reverse = true;
508+
}
509+
_tablet_reader_params.read_orderby_key_num_prefix_columns =
510+
olap_scan_node.sort_info.is_asc_order.size();
511+
_tablet_reader_params.read_orderby_key_limit = _limit;
512+
513+
if (_tablet_reader_params.read_orderby_key_limit > 0 &&
514+
olap_scan_local_state->_storage_no_merge()) {
515+
_tablet_reader_params.filter_block_conjuncts = _conjuncts;
516+
_conjuncts.clear();
517+
}
518+
} else if (_limit > 0 && olap_scan_local_state->_storage_no_merge()) {
519+
// General limit pushdown for DUP_KEYS and UNIQUE_KEYS with MOW
520+
// (non-merge path). Only when topn optimization is NOT active.
521+
// NOTE: _limit is the global query limit (TPlanNode.limit), not a
522+
// per-scanner budget. With N scanners each scanner may read up to
523+
// _limit rows, so up to N * _limit rows are read in total before
524+
// the _shared_scan_limit coordinator stops them. This is
525+
// acceptable because _shared_scan_limit guarantees correctness,
526+
// and the over-read is bounded by (N-1) * _limit which is small
527+
// for typical LIMIT values.
528+
_tablet_reader_params.general_read_limit = _limit;
508529
_tablet_reader_params.filter_block_conjuncts = _conjuncts;
509530
_conjuncts.clear();
510531
}

‎be/src/storage/iterator/vcollect_iterator.cpp‎

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,14 @@ void VCollectIterator::init(TabletReader* reader, bool ori_data_overlapping, boo
9292
_topn_limit = _reader->_reader_context.read_orderby_key_limit;
9393
} else {
9494
_topn_limit = 0;
95-
DCHECK_EQ(_reader->_reader_context.filter_block_conjuncts.size(), 0);
95+
}
96+
97+
// General limit pushdown: only for non-merge path (DUP_KEYS or UNIQUE_KEYS with MOW).
98+
// The scanner already guards this with _storage_no_merge(), but we also check !_merge
99+
// here because _merge can be forced true by overlapping data (force_merge), in which
100+
// case limit pushdown is not safe.
101+
if (!_merge && _reader->_reader_context.general_read_limit > 0) {
102+
_general_read_limit = _reader->_reader_context.general_read_limit;
96103
}
97104
}
98105

@@ -248,8 +255,43 @@ Status VCollectIterator::next(Block* block) {
248255
return _topn_next(block);
249256
}
250257

258+
// Fast path: if general limit already reached, return EOF immediately
259+
if (_general_read_limit > 0 && _general_rows_returned >= _general_read_limit) {
260+
return Status::Error<END_OF_FILE>("");
261+
}
262+
251263
if (LIKELY(_inner_iter)) {
252-
return _inner_iter->next(block);
264+
auto st = _inner_iter->next(block);
265+
if (UNLIKELY(!st.ok())) {
266+
return st;
267+
}
268+
269+
// Apply filter_block_conjuncts that were moved from Scanner::_conjuncts.
270+
// This must happen BEFORE limit counting so that _general_rows_returned
271+
// reflects post-filter rows (same pattern as _topn_next).
272+
// Intentionally not gated by _general_read_limit > 0:
273+
// filter_block_conjuncts is only populated when the general-limit or
274+
// topn branches move conjuncts into the storage layer (topn takes
275+
// the _topn_next path and never reaches here).
276+
if (!_reader->_reader_context.filter_block_conjuncts.empty()) {
277+
RETURN_IF_ERROR(VExprContext::filter_block(
278+
_reader->_reader_context.filter_block_conjuncts, block, block->columns()));
279+
}
280+
281+
// Enforce general read limit: truncate block if needed
282+
if (_general_read_limit > 0) {
283+
_general_rows_returned += block->rows();
284+
if (_general_rows_returned > _general_read_limit) {
285+
// Truncate block to return exactly the remaining rows needed
286+
int64_t excess = _general_rows_returned - _general_read_limit;
287+
int64_t keep = block->rows() - excess;
288+
DCHECK_GT(keep, 0);
289+
block->set_num_rows(keep);
290+
_general_rows_returned = _general_read_limit;
291+
}
292+
}
293+
294+
return Status::OK();
253295
} else {
254296
return Status::Error<END_OF_FILE>("");
255297
}

‎be/src/storage/iterator/vcollect_iterator.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -350,6 +350,12 @@ class VCollectIterator {
350350
bool _topn_eof = false;
351351
std::vector<RowSetSplits> _rs_splits;
352352

353+
// General limit pushdown for DUP_KEYS and UNIQUE_KEYS with MOW (non-merge path).
354+
// When > 0, VCollectIterator will stop reading after returning this many rows.
355+
int64_t _general_read_limit = -1;
356+
// Number of rows already returned to the caller.
357+
int64_t _general_rows_returned = 0;
358+
353359
// Hold reader point to access read params, such as fetch conditions.
354360
TabletReader* _reader = nullptr;
355361

‎be/src/storage/rowset/rowset_reader_context.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,10 @@ struct RowsetReaderContext {
103103

104104
// When true, push down value predicates for MOR tables
105105
bool enable_mor_value_predicate_pushdown = false;
106+
107+
// General limit pushdown for DUP_KEYS and UNIQUE_KEYS with MOW.
108+
// Propagated from ReaderParams.general_read_limit.
109+
int64_t general_read_limit = -1;
106110
};
107111

108112
} // namespace doris

‎be/src/storage/tablet/tablet_reader.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ Status TabletReader::_capture_rs_readers(const ReaderParams& read_params) {
199199
_reader_context.all_access_paths = read_params.all_access_paths;
200200
_reader_context.predicate_access_paths = read_params.predicate_access_paths;
201201

202+
// Propagate general read limit for DUP_KEYS and UNIQUE_KEYS with MOW
203+
_reader_context.general_read_limit = read_params.general_read_limit;
202204
return Status::OK();
203205
}
204206

‎be/src/storage/tablet/tablet_reader.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,11 @@ class TabletReader {
213213
std::shared_ptr<segment_v2::AnnTopNRuntime> ann_topn_runtime;
214214

215215
uint64_t condition_cache_digest = 0;
216+
217+
// General limit pushdown for DUP_KEYS and UNIQUE_KEYS with MOW.
218+
// When > 0, the storage layer (VCollectIterator) will stop reading
219+
// after returning this many rows. -1 means no limit.
220+
int64_t general_read_limit = -1;
216221
};
217222

218223
TabletReader() = default;
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !dup_basic_limit --
3+
1 99
4+
10 90
5+
2 98
6+
3 97
7+
4 96
8+
5 95
9+
6 94
10+
7 93
11+
8 92
12+
9 91
13+
14+
-- !dup_filter_limit --
15+
11 89
16+
12 88
17+
13 87
18+
14 86
19+
15 85
20+
16 84
21+
17 83
22+
18 82
23+
19 81
24+
20 80
25+
21 79
26+
22 78
27+
23 77
28+
24 76
29+
25 75
30+
31+
-- !dup_filter_over_limit --
32+
46 54
33+
47 53
34+
48 52
35+
49 51
36+
50 50
37+
38+
-- !dup_complex_filter_limit --
39+
16 84
40+
17 83
41+
18 82
42+
19 81
43+
20 80
44+
21 79
45+
22 78
46+
23 77
47+
48+
-- !mow_basic_limit --
49+
1 99
50+
10 90
51+
2 98
52+
3 97
53+
4 96
54+
5 95
55+
6 94
56+
7 93
57+
8 92
58+
9 91
59+
60+
-- !mow_filter_limit --
61+
11 89
62+
12 88
63+
13 87
64+
14 86
65+
15 85
66+
16 84
67+
17 83
68+
18 82
69+
19 81
70+
20 80
71+
21 79
72+
22 78
73+
23 77
74+
24 76
75+
25 75
76+
77+
-- !mow_complex_filter_limit --
78+
16 84
79+
17 83
80+
18 82
81+
19 81
82+
20 80
83+
21 79
84+
22 78
85+
23 77
86+
87+
-- !agg_basic_limit --
88+
1 1 30
89+
2 2 70
90+
91+
-- !mor_filter_limit --
92+
11 89
93+
12 88
94+
13 87
95+
14 86
96+
15 85
97+
16 84
98+
17 83
99+
18 82
100+
19 81
101+
20 80
102+
103+
-- !mow_delete_limit --
104+
21 79
105+
22 78
106+
23 77
107+
24 76
108+
25 75
109+
26 74
110+
27 73
111+
28 72
112+
29 71
113+
30 70
114+
31 69
115+
32 68
116+
33 67
117+
34 66
118+
35 65
119+
120+
-- !dup_limit_offset --
121+
10 90
122+
11 89
123+
12 88
124+
13 87
125+
14 86
126+
15 85
127+
6 94
128+
7 93
129+
8 92
130+
9 91
131+

0 commit comments

Comments
 (0)