Skip to content

Commit 30a44d6

Browse files
authored
[improvement](be) Optimize filtered Parquet scans and condition cache (#65602)
[improvement](be) Optimize filtered Parquet scans and condition cache Problem Summary: Parquet v2 repeatedly advanced lazy non-predicate readers for every fully filtered batch, and external condition cache could not safely reuse entries when runtime filters were present. Accumulate skipped rows for lazy readers and flush once before the next surviving batch, discard the lag at row-group boundaries, expose Arrow skip time, and key scanner-driven condition cache entries with the exact predicate and runtime-filter snapshot. Runtime filters without a reliable digest and standalone callers without an explicit split digest remain uncached. Condition cache + runtime filter behavior Before this change, the presence of any runtime filter disabled condition cache, even when the filter was ready and had a stable payload. Runtime-filter pushdown still worked, but repeated scans could neither look up nor publish a granule survivor bitmap. After this change, scanner-driven splits rebuild the condition-cache digest from the exact conjunct snapshot used by that split. Ready IN, Bloom, and Min/Max runtime filters encode their probe expression, filter semantics, payload, and null behavior in the digest. For example, for static predicate P: price > 100: P AND customer_id IN {7, 9} uses a payload-specific key digest(P, RF{7,9}). After a cache miss reaches reader EOF, its survivor bitmap is published under that key. P AND customer_id IN {9, 7} has the same IN-set semantics and therefore reuses the same key. P AND customer_id IN {8, 10} produces a different key and cannot reuse the {7,9} bitmap. If split 0 starts with only P and RF {7,9} arrives before split 1, split 0 uses digest(P) while split 1 rebuilds and uses digest(P, RF{7,9}); the late RF is never stored under the stale P key. If any current conjunct returns digest 0, condition cache is disabled for that split while runtime-filter pushdown remains enabled. This is the correctness fallback for expressions that cannot represent their complete semantics in a stable digest—for example, an RF probe expression containing a digest-unsupported lambda, or a future/custom stateful RF whose payload is not digestible. Standalone TableReader callers with an RF also remain conservative unless they provide an explicit digest for the exact split snapshot. Problem Summary: Parquet v2 repeatedly advanced lazy non-predicate readers for every fully filtered batch, and external condition cache could not safely reuse entries when runtime filters were present. Accumulate skipped rows for lazy readers and flush once before the next surviving batch, discard the lag at row-group boundaries, expose Arrow skip time, and key scanner-driven condition cache entries with the exact predicate and runtime-filter snapshot. Runtime filters without a reliable digest and standalone callers without an explicit split digest remain uncached. #### Condition cache + runtime filter behavior Before this change, the presence of any runtime filter disabled condition cache, even when the filter was ready and had a stable payload. Runtime-filter pushdown still worked, but repeated scans could neither look up nor publish a granule survivor bitmap. After this change, scanner-driven splits rebuild the condition-cache digest from the exact conjunct snapshot used by that split. Ready IN, Bloom, and Min/Max runtime filters encode their probe expression, filter semantics, payload, and null behavior in the digest. For example, for static predicate `P: price > 100`: - `P AND customer_id IN {7, 9}` uses a payload-specific key `digest(P, RF{7,9})`. After a cache miss reaches reader EOF, its survivor bitmap is published under that key. - `P AND customer_id IN {9, 7}` has the same IN-set semantics and therefore reuses the same key. - `P AND customer_id IN {8, 10}` produces a different key and cannot reuse the `{7,9}` bitmap. - If split 0 starts with only `P` and RF `{7,9}` arrives before split 1, split 0 uses `digest(P)` while split 1 rebuilds and uses `digest(P, RF{7,9})`; the late RF is never stored under the stale `P` key. If any current conjunct returns digest `0`, condition cache is disabled for that split while runtime-filter pushdown remains enabled. This is the correctness fallback for expressions that cannot represent their complete semantics in a stable digest—for example, an RF probe expression containing a digest-unsupported lambda, or a future/custom stateful RF whose payload is not digestible. Standalone `TableReader` callers with an RF also remain conservative unless they provide an explicit digest for the exact split snapshot. ### Release note Improve Parquet external scan performance for highly selective predicates and allow payload-safe condition cache reuse with ready runtime filters. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh -j 64 --run --filter='ParquetScanTest.*:NewParquetReaderTest.PageIndexFilteredGapFlushesPendingOutputSkipOnce:TableReaderTest.ConditionCache*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload'` - `./run-be-ut.sh -j 64 --run --filter='TableReaderTest.ConditionCache*'` - `build-support/check-format.sh` - clang-tidy attempted but blocked by pre-existing remote toolchain header and NOLINT errors - Behavior changed: Yes. Lazy Parquet output readers defer skips across empty selections, and condition cache supports runtime filters only with an exact split digest. - Does this need documentation: No
1 parent 7f302a1 commit 30a44d6

16 files changed

Lines changed: 427 additions & 31 deletions

‎be/src/exec/scan/file_scanner.cpp‎

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1942,25 +1942,7 @@ bool FileScanner::_should_enable_condition_cache() {
19421942
return false;
19431943
}
19441944

1945-
// Runtime filters are query-local dynamic predicates. Some ready RF implementations can hash
1946-
// their payload into get_digest(), but FileScanner cannot rely on that for all RFs reaching the
1947-
// native reader. In particular, ScanLocalState computes _condition_cache_digest during open(),
1948-
// while FileScanner may append late-arrival RFs in _process_late_arrival_conjuncts()
1949-
// immediately before initializing Parquet/ORC readers.
1950-
//
1951-
// Reading a weaker cache entry would be safe by itself: if a cached bitmap only represented
1952-
// static predicate P, false granules for P are also false for P AND RF. The unsafe part is
1953-
// writing. On cache miss, native readers mark survivor granules using all pushed-down
1954-
// predicates, including late RFs. Without a read-only cache mode, this would insert a bitmap for
1955-
// P AND RF under a digest that only represents P.
1956-
//
1957-
// Example:
1958-
// Q1 static predicate: k = 1, late RF payload: partition_key IN ('2024-02-01')
1959-
// Q2 static predicate: k = 1, late RF payload: partition_key IN ('2024-03-01')
1960-
// If both scans share the same file/range/digest, reusing Q1's bitmap for Q2 can skip row
1961-
// ranges according to the wrong RF payload. Keep RF predicate pushdown enabled for reader-side
1962-
// filtering, but do not persist its result in condition cache.
1963-
return !_contains_runtime_filter(_conjuncts) && !_contains_runtime_filter(_push_down_conjuncts);
1945+
return true;
19641946
}
19651947

19661948
bool FileScanner::_should_enable_condition_cache_for_load() const {
@@ -1990,6 +1972,12 @@ void FileScanner::_init_reader_condition_cache() {
19901972
_condition_cache = nullptr;
19911973
_condition_cache_ctx = nullptr;
19921974

1975+
// _process_late_arrival_conjuncts() runs before the Parquet/ORC reader is initialized. Rebuild
1976+
// the key here so a newly arrived cache-safe RF is represented by both its semantics and its
1977+
// payload. If any current conjunct cannot produce a reliable digest, this becomes zero and the
1978+
// existing safety gate below disables condition cache.
1979+
_condition_cache_digest = _current_condition_cache_digest();
1980+
19931981
if (!_should_enable_condition_cache() || !_cur_reader) {
19941982
return;
19951983
}

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -531,6 +531,7 @@ Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range,
531531
// A metadata COUNT split may span scheduler turns. Do not enter that irreversible
532532
// synthetic-row path while a runtime filter can still arrive between batches.
533533
.all_runtime_filters_applied = _applied_rf_num == _total_rf_num,
534+
.condition_cache_digest = _current_condition_cache_digest(),
534535
.cache = _kv_cache,
535536
.current_range = range,
536537
.current_split_format = current_split_format,

‎be/src/exec/scan/scanner.cpp‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,40 @@ Status Scanner::try_append_late_arrival_runtime_filter() {
270270
return Status::OK();
271271
}
272272

273+
uint64_t Scanner::_current_condition_cache_digest() const {
274+
DORIS_CHECK(_state != nullptr);
275+
DORIS_CHECK(_local_state != nullptr);
276+
if (_local_state->get_condition_cache_digest() == 0) {
277+
return 0;
278+
}
279+
280+
// ScanLocalState computed its digest after collecting the RFs that were ready during open(). A
281+
// scanner may later clone more RF conjuncts between file splits, so rebuild from its current
282+
// snapshot instead of reusing that stale value. For example, split 0 may use P, while split 1
283+
// starts after an IN RF with payload {7, 9} arrives and must use digest(P AND RF{7, 9}). A
284+
// different payload {8, 10} consequently receives a different key. get_digest() returning zero
285+
// is the correctness fallback for an RF whose complete semantics cannot be represented.
286+
return _build_condition_cache_digest(_state->query_options().condition_cache_digest,
287+
_conjuncts);
288+
}
289+
290+
uint64_t Scanner::_build_condition_cache_digest(uint64_t seed, const VExprContextSPtrs& conjuncts) {
291+
for (const auto& conjunct : conjuncts) {
292+
seed = conjunct->get_digest(seed);
293+
if (seed == 0) {
294+
return 0;
295+
}
296+
}
297+
return seed;
298+
}
299+
300+
#ifdef BE_TEST
301+
uint64_t Scanner::TEST_build_condition_cache_digest(uint64_t seed,
302+
const VExprContextSPtrs& conjuncts) {
303+
return _build_condition_cache_digest(seed, conjuncts);
304+
}
305+
#endif
306+
273307
Status Scanner::close(RuntimeState* state) {
274308
#ifndef BE_TEST
275309
COUNTER_UPDATE(_local_state->_scanner_wait_worker_timer, _scanner_wait_worker_timer);

‎be/src/exec/scan/scanner.h‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,19 @@ class Scanner {
9797
// eg, for file scanner, return the current file path.
9898
virtual std::string get_current_scan_range_name() { return "not implemented"; }
9999

100+
#ifdef BE_TEST
101+
static uint64_t TEST_build_condition_cache_digest(uint64_t seed,
102+
const VExprContextSPtrs& conjuncts);
103+
#endif
104+
100105
protected:
106+
// Rebuild the condition-cache digest from the scanner's current conjunct snapshot. The local
107+
// state's digest is used only as a safety gate: zero means condition cache was disabled during
108+
// scan-node open (for example by TopN or an expression without a reliable digest).
109+
uint64_t _current_condition_cache_digest() const;
110+
static uint64_t _build_condition_cache_digest(uint64_t seed,
111+
const VExprContextSPtrs& conjuncts);
112+
101113
virtual Status _prepare_impl() {
102114
_has_prepared = true;
103115
return Status::OK();

‎be/src/format_v2/parquet/parquet_profile.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,8 @@ void ParquetProfile::init(RuntimeProfile* profile) {
7171
parquet_profile, 1);
7272
arrow_read_records_time =
7373
ADD_CHILD_TIMER_WITH_LEVEL(profile, "ArrowReadRecordsTime", parquet_profile, 1);
74+
arrow_skip_records_time =
75+
ADD_CHILD_TIMER_WITH_LEVEL(profile, "ArrowSkipRecordsTime", parquet_profile, 1);
7476
materialization_time =
7577
ADD_CHILD_TIMER_WITH_LEVEL(profile, "MaterializationTime", parquet_profile, 1);
7678
lazy_read_filtered_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "FilteredRowsByLazyRead",
@@ -198,6 +200,7 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const {
198200
.reader_skip_rows = reader_skip_rows,
199201
.reader_select_rows = reader_select_rows,
200202
.arrow_read_records_time = arrow_read_records_time,
203+
.arrow_skip_records_time = arrow_skip_records_time,
201204
.materialization_time = materialization_time,
202205
};
203206
}

‎be/src/format_v2/parquet/parquet_profile.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ struct ParquetColumnReaderProfile {
3535
RuntimeProfile::Counter* reader_skip_rows = nullptr; // rows skipped by skip()
3636
RuntimeProfile::Counter* reader_select_rows = nullptr; // rows selected by select()
3737
RuntimeProfile::Counter* arrow_read_records_time = nullptr; // Arrow RecordReader time (ns)
38+
RuntimeProfile::Counter* arrow_skip_records_time = nullptr; // Arrow SkipRecords time (ns)
3839
RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns)
3940
};
4041

@@ -103,6 +104,7 @@ struct ParquetProfile {
103104
RuntimeProfile::Counter* reader_skip_rows = nullptr;
104105
RuntimeProfile::Counter* reader_select_rows = nullptr;
105106
RuntimeProfile::Counter* arrow_read_records_time = nullptr;
107+
RuntimeProfile::Counter* arrow_skip_records_time = nullptr;
106108
RuntimeProfile::Counter* materialization_time = nullptr;
107109

108110
RuntimeProfile::Counter* lazy_read_filtered_rows = nullptr;

‎be/src/format_v2/parquet/parquet_scan.cpp‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -655,6 +655,12 @@ void ParquetScanScheduler::reset_current_row_group() {
655655
_current_selected_ranges.clear();
656656
_current_range_idx = 0;
657657
_current_range_rows_read = 0;
658+
// Readers are row-group scoped. If every remaining row was filtered, no future output can
659+
// observe the non-predicate readers' position, so dropping them together with their pending lag
660+
// avoids a useless end-of-row-group SkipRecords call. Example: predicate readers advance from 0
661+
// to 10,000 while lazy readers stay at 0; clearing both readers here is sufficient because the
662+
// next row group constructs a new set starting at its own row 0.
663+
_pending_non_predicate_skip_rows = 0;
658664
_current_predicate_prefetched = false;
659665
_current_non_predicate_prefetched = false;
660666
_current_merge_range_active = false;
@@ -787,10 +793,23 @@ Status ParquetScanScheduler::skip_current_row_group_rows(int64_t rows) {
787793
for (const auto& column_reader : _current_predicate_columns | std::views::values) {
788794
RETURN_IF_ERROR(column_reader->skip(rows));
789795
}
796+
// Keep page-index/condition-cache gaps pending for lazy columns as well. For example, after a
797+
// fully filtered [0, 32) batch and a pruned [32, 96) gap, predicate readers are at 96 while lazy
798+
// readers remain at 0; one later skip(96) is cheaper than skip(32) followed by skip(64).
799+
DORIS_CHECK(_pending_non_predicate_skip_rows <= std::numeric_limits<int64_t>::max() - rows);
800+
_pending_non_predicate_skip_rows += rows;
801+
_current_row_group_rows_read += rows;
802+
return Status::OK();
803+
}
804+
805+
Status ParquetScanScheduler::flush_pending_non_predicate_skip_rows() {
806+
if (_pending_non_predicate_skip_rows == 0) {
807+
return Status::OK();
808+
}
790809
for (const auto& column_reader : _current_non_predicate_columns | std::views::values) {
791-
RETURN_IF_ERROR(column_reader->skip(rows));
810+
RETURN_IF_ERROR(column_reader->skip(_pending_non_predicate_skip_rows));
792811
}
793-
_current_row_group_rows_read += rows;
812+
_pending_non_predicate_skip_rows = 0;
794813
return Status::OK();
795814
}
796815

@@ -1395,6 +1414,18 @@ Status ParquetScanScheduler::read_current_row_group_batch(
13951414
.column->filter(output_filter, selected_rows)));
13961415
}
13971416
}
1417+
if (selected_rows == 0) {
1418+
// Predicate readers have consumed this physical batch, but touching every lazy column here
1419+
// turns a long rejected prefix into `empty_batches * lazy_columns` Arrow calls. Record only
1420+
// the positional lag. If [0, 32), [32, 64), and [64, 96) are empty, the first surviving
1421+
// batch performs one skip(96) per lazy column. If the row group ends instead, reset drops the
1422+
// lazy readers without flushing because no value from them can be observed.
1423+
DORIS_CHECK(_pending_non_predicate_skip_rows <=
1424+
std::numeric_limits<int64_t>::max() - batch_rows);
1425+
_pending_non_predicate_skip_rows += batch_rows;
1426+
*rows = 0;
1427+
return Status::OK();
1428+
}
13981429
if (!_current_merge_range_active && selected_rows > 0 &&
13991430
!_current_non_predicate_columns.empty()) {
14001431
// Do not prefetch lazy output columns until at least one row survives filtering. This is
@@ -1406,6 +1437,9 @@ Status ParquetScanScheduler::read_current_row_group_batch(
14061437

14071438
{
14081439
SCOPED_TIMER(_scan_profile.column_read_time);
1440+
// Bring lazy readers to the first row of the current physical batch before interpreting its
1441+
// selection vector. This also merges pending range gaps with fully filtered batches.
1442+
RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows());
14091443
for (const auto& [fid, column_reader] : _current_non_predicate_columns) {
14101444
auto position_it = request.local_positions.find(format::LocalColumnId(fid));
14111445
DORIS_CHECK(position_it != request.local_positions.end());

‎be/src/format_v2/parquet/parquet_scan.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,7 @@ class ParquetScanScheduler {
152152
const format::FileScanRequest& request, bool* has_row_group);
153153

154154
Status skip_current_row_group_rows(int64_t rows);
155+
Status flush_pending_non_predicate_skip_rows();
155156

156157
Status read_filter_columns(int64_t batch_rows, const format::FileScanRequest& request,
157158
Block* file_block, SelectionVector* selection,
@@ -203,6 +204,10 @@ class ParquetScanScheduler {
203204
_current_selected_ranges; // selected ranges for the current row group after page-index pruning
204205
size_t _current_range_idx = 0; // current selected_range index
205206
int64_t _current_range_rows_read = 0; // rows read in the current range
207+
// Predicate readers move immediately because they decide which rows survive. Non-predicate
208+
// readers can lag behind across fully filtered batches and range gaps; the lag is flushed once
209+
// before the next surviving batch is materialized, or discarded with the row group.
210+
int64_t _pending_non_predicate_skip_rows = 0;
206211

207212
bool _current_predicate_prefetched = false;
208213
bool _current_non_predicate_prefetched = false;

‎be/src/format_v2/parquet/reader/scalar_column_reader.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@ Status ScalarColumnReader::skip_records(int64_t rows) {
183183
return Status::OK();
184184
}
185185
int64_t skipped_rows = 0;
186+
SCOPED_TIMER(_profile.arrow_skip_records_time);
186187
try {
187188
_record_reader->Reset();
188189
while (skipped_rows < rows) {

‎be/src/format_v2/table_reader.cpp‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -527,7 +527,8 @@ Status TableReader::init(TableReadOptions&& options) {
527527
_scanner_profile = options.scanner_profile;
528528
_file_slot_descs = options.file_slot_descs;
529529
_push_down_agg_type = options.push_down_agg_type;
530-
_condition_cache_digest = options.condition_cache_digest;
530+
_initial_condition_cache_digest = options.condition_cache_digest;
531+
_condition_cache_digest = _initial_condition_cache_digest;
531532
_projected_columns = std::move(options.projected_columns);
532533
_system_properties = create_system_properties(_scan_params);
533534
_mapper_options.mode = TableColumnMappingMode::BY_NAME;
@@ -631,10 +632,11 @@ bool TableReader::_should_enable_condition_cache(const FileScanRequest& file_req
631632
!file_request.delete_conjuncts.empty()) {
632633
return false;
633634
}
634-
// Runtime filters can arrive late and their payload is not guaranteed to be represented by the
635-
// scan-local digest. Without a read-only mode, a MISS could insert a bitmap for P AND RF under
636-
// the digest for only P. This mirrors the old FileScanner guard.
637-
return !contains_runtime_filter(file_request.conjuncts);
635+
// Only scanner-driven splits provide a digest rebuilt from the exact RF snapshot. Keep the
636+
// conservative behavior for standalone TableReader callers: their initial digest may describe
637+
// only static predicate P and must not store P AND RF under that key.
638+
return _condition_cache_digest_covers_current_split ||
639+
!contains_runtime_filter(file_request.conjuncts);
638640
}
639641

640642
Status TableReader::_init_reader_condition_cache(const FileScanRequest& file_request) {
@@ -828,6 +830,17 @@ Status TableReader::prepare_split(const SplitReadOptions& options) {
828830
SCOPED_TIMER(_profile.prepare_split_timer);
829831
_current_split_pruned = false;
830832
_all_runtime_filters_applied_for_split = options.all_runtime_filters_applied;
833+
_condition_cache_digest_covers_current_split = options.condition_cache_digest.has_value();
834+
if (options.condition_cache_digest.has_value()) {
835+
// The split snapshot may include RFs that arrived after TableReader::init(). Use the digest
836+
// computed from that exact snapshot. Example: an initial P digest must not be used to store
837+
// the bitmap for P AND late RF{7, 9}; the scanner supplies digest(P AND RF{7, 9}) here.
838+
_condition_cache_digest = *options.condition_cache_digest;
839+
} else {
840+
// An explicit scanner digest is split-scoped. Restore the init-time digest when a later
841+
// standalone split omits it instead of leaking the previous split's RF payload into its key.
842+
_condition_cache_digest = _initial_condition_cache_digest;
843+
}
831844
if (options.conjuncts.has_value()) {
832845
_conjuncts = *options.conjuncts;
833846
}

0 commit comments

Comments
 (0)