Skip to content

Commit f391164

Browse files
authored
[fix](scanner)(nereids) Harden FileScannerV2 and fix external COUNT pushdown semantics (#65548)
### What problem does this PR solve? Issue Number: None Related PR: None This PR fixes two correctness areas in external file scans. #### 1. Preserve external `COUNT(*)` / `COUNT(column)` semantics External `COUNT(*)` retained an arbitrary scan slot after column pruning. BE then inferred aggregate semantics from that post-pruning scan shape and could treat the retained nullable placeholder as the argument of `COUNT(column)`. This produced wrong results, including 9015 instead of 10000 rows for special ORC data and 116 instead of 219 rows for Hive basic types. The fix transports semantic COUNT arguments from Nereids through `TPlanNode.push_down_count_slot_ids` to both file-scanner implementations and their table/hybrid readers. The thrift field has an explicit compatibility contract: - absent: an old FE plan with unknown COUNT semantics; use the conservative normal-scan path; - present and empty: explicit `COUNT(*)` / row-count semantics; - present and non-empty: explicit `COUNT(column)` arguments, which may use metadata only when every mapping is proven safe. Nereids no longer applies storage COUNT pushdown to cast arguments whose null/error behavior would be lost. FE split planning also restricts Iceberg/Paimon metadata-count split reduction and Hive/TVF no-split shortcuts to explicit row-count semantics, so a `COUNT(column)` fallback retains all data splits and normal scan parallelism. BE V1, V2, Hudi, Paimon, and table-reader paths preserve the same argument state, reject unsafe metadata shortcuts, and keep adaptive batching enabled for real-row fallbacks. #### 2. Harden FileScannerV2 schema and reader edge cases FileScannerV2 treated valid empty splits as failures, could count stopped EOF or malformed Native input as an empty file, localized evolved nested predicates without fully preserving type/nullability contracts, and rejected whole Parquet schemas when only unprojected leaves used unsupported logical types. Parquet page-cache range metadata also lost warm non-exact reuse while lacking a bound. The fix distinguishes stopped reads, valid empty input, and malformed/truncated input; keeps unsafe nested predicates above `TableReader`; validates only semantically required Parquet projections while preserving strict checks for real predicate and `COUNT(column)` inputs; uses safe row-position/default carriers for placeholder-only paths; and shares bounded per-file page-range indexes without adding a process-wide lock to the `ReadAt` hot path. ### Release note Fix wrong external `COUNT(*)` / `COUNT(column)` results and unsafe metadata pushdown across Nereids, thrift, FE split planning, and BE file readers. Also harden FileScannerV2 handling of empty or interrupted input, evolved nested predicates, unsupported unprojected Parquet logical types, COUNT placeholders, hybrid readers, and bounded cross-reader page-cache range reuse. ### Check List (For Author) - Test: Regression test / Unit Test - BE ASAN unit coverage for V1/V2 COUNT semantics, table-level metadata gates, schema-evolution fallbacks, COUNT placeholders, adaptive batching, and Hudi/Paimon/Iceberg hybrid paths - FE unit coverage for Nereids COUNT argument capture and Hive/Iceberg/Paimon/TVF split-planning behavior - `ColumnMapper*.*` BE ASAN suite (111 tests passed) - Existing external regression cases `test_special_orc_formats` and `test_hive_basic_type` reproduce the fixed COUNT(*) wrong results - `test_file_scanner_v2_review_fixes` regression suite covers empty input, evolved nested types, unsupported Parquet placeholders, and COUNT fallback behavior - Behavior changed: Yes (COUNT semantics are transported explicitly; unsafe or ambiguous metadata COUNT plans fall back to normal scanning; valid empty splits are skipped; stopped or malformed input is not counted as empty; unsupported unprojected Parquet leaves no longer fail unrelated scans) - Does this need documentation: No
1 parent 54e0e97 commit f391164

55 files changed

Lines changed: 2310 additions & 188 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/exec/operator/scan_operator.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1046,6 +1046,12 @@ TPushAggOp::type ScanLocalState<Derived>::get_push_down_agg_type() {
10461046
return _parent->cast<typename Derived::Parent>()._push_down_agg_type;
10471047
}
10481048

1049+
template <typename Derived>
1050+
const std::optional<std::vector<int32_t>>& ScanLocalState<Derived>::get_push_down_count_slot_ids()
1051+
const {
1052+
return _parent->cast<typename Derived::Parent>()._push_down_count_slot_ids;
1053+
}
1054+
10491055
template <typename Derived>
10501056
int64_t ScanLocalState<Derived>::limit_per_scanner() {
10511057
return _parent->cast<typename Derived::Parent>()._limit_per_scanner;
@@ -1231,6 +1237,9 @@ Status ScanOperatorX<LocalStateType>::init(const TPlanNode& tnode, RuntimeState*
12311237
} else {
12321238
_push_down_agg_type = TPushAggOp::type::NONE;
12331239
}
1240+
if (tnode.__isset.push_down_count_slot_ids) {
1241+
_push_down_count_slot_ids = tnode.push_down_count_slot_ids;
1242+
}
12341243

12351244
if (tnode.__isset.topn_filter_source_node_ids) {
12361245
_topn_filter_source_node_ids = tnode.topn_filter_source_node_ids;

‎be/src/exec/operator/scan_operator.h‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#pragma once
1919

2020
#include <cstdint>
21+
#include <optional>
2122
#include <set>
2223
#include <string>
2324

@@ -74,6 +75,7 @@ class ScanLocalStateBase : public PipelineXLocalState<> {
7475
virtual void set_scan_ranges(RuntimeState* state,
7576
const std::vector<TScanRangeParams>& scan_ranges) = 0;
7677
virtual TPushAggOp::type get_push_down_agg_type() = 0;
78+
virtual const std::optional<std::vector<int32_t>>& get_push_down_count_slot_ids() const = 0;
7779

7880
// If scan operator is serial operator(like topn), its real parallelism is 1.
7981
// Otherwise, its real parallelism is query_parallel_instance_num.
@@ -252,6 +254,7 @@ class ScanLocalState : public ScanLocalStateBase {
252254
const std::vector<TScanRangeParams>& scan_ranges) override {}
253255

254256
TPushAggOp::type get_push_down_agg_type() override;
257+
const std::optional<std::vector<int32_t>>& get_push_down_count_slot_ids() const override;
255258

256259
std::vector<Dependency*> execution_dependencies() override {
257260
if (_filter_dependencies.empty()) {
@@ -452,6 +455,16 @@ class ScanOperatorX : public OperatorX<LocalStateType> {
452455

453456
TPushAggOp::type _push_down_agg_type;
454457

458+
// Semantic arguments of a pushed-down COUNT. This is deliberately optional because absence
459+
// and an empty list have different meanings during a BE-first rolling upgrade:
460+
//
461+
// - nullopt: an old FE did not send the field, so the new BE must use the normal scan;
462+
// - empty: the new FE explicitly planned COUNT(*)/COUNT(1);
463+
// - non-empty: the new FE explicitly planned COUNT(col).
464+
//
465+
// Treating nullopt as empty would silently reinterpret an old plan as COUNT(*).
466+
std::optional<std::vector<int32_t>> _push_down_count_slot_ids;
467+
455468
// Record the value of the aggregate function 'count' from doris's be
456469
int64_t _push_down_count = -1;
457470
const int _parallel_tasks = 0;

‎be/src/exec/scan/file_scanner.h‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
#include <stdint.h>
2222

2323
#include <memory>
24+
#include <optional>
2425
#include <string>
2526
#include <unordered_map>
2627
#include <unordered_set>
@@ -80,6 +81,10 @@ class FileScanner : public Scanner {
8081
const VExprContextSPtrs& TEST_runtime_filter_partition_prune_ctxs() const {
8182
return _runtime_filter_partition_prune_ctxs;
8283
}
84+
static TPushAggOp::type TEST_effective_push_down_agg_type(
85+
TPushAggOp::type agg_type, const std::optional<std::vector<int32_t>>& count_slot_ids) {
86+
return _effective_push_down_agg_type(agg_type, count_slot_ids);
87+
}
8388
#endif
8489

8590
FileScanner(RuntimeState* state, FileScanLocalState* parent, int64_t limit,
@@ -334,9 +339,28 @@ class FileScanner : public Scanner {
334339
void _update_adaptive_batch_size_before_truncate(const Block& block);
335340
void _update_adaptive_batch_size_after_truncate(const Block& block);
336341

342+
static TPushAggOp::type _effective_push_down_agg_type(
343+
TPushAggOp::type agg_type, const std::optional<std::vector<int32_t>>& count_slot_ids) {
344+
if (agg_type != TPushAggOp::type::COUNT) {
345+
return agg_type;
346+
}
347+
// V1's CountReader receives only the file's total row count and emits that many synthetic
348+
// rows. This is exact for COUNT(*)/COUNT(1), but it has no column metadata for NULL or CAST
349+
// semantics. For example, a 10,000-row file with 9,015 non-null values must return 10,000
350+
// for COUNT(*) and 9,015 for COUNT(nullable_col); CountReader can produce only the former.
351+
// Therefore a non-empty argument list must use the normal reader. nullopt is an old FE plan
352+
// that predates the argument field; treating it as empty would silently reinterpret unknown
353+
// semantics as COUNT(*).
354+
return count_slot_ids.has_value() && count_slot_ids->empty() ? TPushAggOp::type::COUNT
355+
: TPushAggOp::type::NONE;
356+
}
357+
337358
TPushAggOp::type _get_push_down_agg_type() const {
338-
return _local_state == nullptr ? TPushAggOp::type::NONE
339-
: _local_state->get_push_down_agg_type();
359+
if (_local_state == nullptr) {
360+
return TPushAggOp::type::NONE;
361+
}
362+
return _effective_push_down_agg_type(_local_state->get_push_down_agg_type(),
363+
_local_state->get_push_down_count_slot_ids());
340364
}
341365

342366
// enable the file meta cache only when

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 71 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,10 @@ void FileScannerV2::TEST_report_file_cache_profile(
272272
bool FileScannerV2::TEST_should_skip_not_found(const Status& status, bool ignore_not_found) {
273273
return _should_skip_not_found(status, ignore_not_found);
274274
}
275+
276+
bool FileScannerV2::TEST_should_skip_empty(const Status& status, bool stopped) {
277+
return _should_skip_empty(status, stopped);
278+
}
275279
#endif
276280

277281
bool FileScannerV2::is_supported(const TFileScanRangeParams& params, const TFileRangeDesc& range) {
@@ -322,6 +326,8 @@ Status FileScannerV2::init(RuntimeState* state, const VExprContextSPtrs& conjunc
322326
RETURN_IF_ERROR(Scanner::init(state, conjuncts));
323327
_get_block_timer =
324328
ADD_TIMER_WITH_LEVEL(_local_state->scanner_profile(), "FileScannerV2GetBlockTime", 1);
329+
_empty_file_counter =
330+
ADD_COUNTER_WITH_LEVEL(_local_state->scanner_profile(), "EmptyFileNum", TUnit::UNIT, 1);
325331
_not_found_file_counter = ADD_COUNTER_WITH_LEVEL(_local_state->scanner_profile(),
326332
"NotFoundFileNum", TUnit::UNIT, 1);
327333
_file_counter =
@@ -394,6 +400,20 @@ Status FileScannerV2::_get_block_impl(RuntimeState* state, Block* block, bool* e
394400
*eof = false;
395401
continue;
396402
}
403+
if (_should_skip_empty(status, _should_stop || _io_ctx->should_stop)) {
404+
// END_OF_FILE here means the reader discovered a valid split with no data while
405+
// opening or probing it, not that the Scanner has exhausted all splits. Examples
406+
// are a zero-byte CSV with an explicit schema and a Doris Native file containing
407+
// only its 12-byte header. Treat it like V1's empty-file path: finish this range,
408+
// discard partial reader state, and let the loop fetch the next split.
409+
RETURN_IF_ERROR(_table_reader->abort_split());
410+
COUNTER_UPDATE(_empty_file_counter, 1);
411+
_state->update_num_finished_scan_range(1);
412+
_has_prepared_split = false;
413+
block->clear_column_data(cast_set<int64_t>(_projected_columns.size()));
414+
*eof = false;
415+
continue;
416+
}
397417
RETURN_IF_ERROR(status);
398418
}
399419
if (*eof) {
@@ -423,9 +443,12 @@ Status FileScannerV2::_prepare_next_split(bool* eos) {
423443

424444
const auto format_type = get_range_format_type(*_params, _current_range);
425445
_init_adaptive_batch_size_state(format_type);
426-
if (_should_run_adaptive_batch_size()) {
427-
// JNI readers open eagerly in prepare_split(). Seed the probe size first so readers
428-
// such as Paimon also use it for their first physical read batch.
446+
if (_block_size_predictor != nullptr) {
447+
// JNI readers open eagerly in prepare_split(). Always seed the probe before preparing
448+
// the next split: its metadata-COUNT decision is not available yet, and the state
449+
// exposed by TableReader can still describe the preceding split. Metadata shortcuts
450+
// ignore this batch size, while row-scan fallbacks need it for their first physical
451+
// read batch.
429452
_table_reader->set_batch_size(_predict_reader_batch_rows());
430453
}
431454
std::map<std::string, Field> partition_values;
@@ -438,6 +461,16 @@ Status FileScannerV2::_prepare_next_split(bool* eos) {
438461
_state->update_num_finished_scan_range(1);
439462
continue;
440463
}
464+
if (_should_skip_empty(status, _should_stop || _io_ctx->should_stop)) {
465+
// Schema discovery can reach EOF before a split becomes prepared. A header-only Native
466+
// file follows this path, while a reader that discovers emptiness on its first
467+
// get_block() follows the symmetric branch in _get_block_impl(). Both paths must
468+
// advance exactly one scan range and preserve later files in the same scan.
469+
RETURN_IF_ERROR(_table_reader->abort_split());
470+
COUNTER_UPDATE(_empty_file_counter, 1);
471+
_state->update_num_finished_scan_range(1);
472+
continue;
473+
}
441474
RETURN_IF_ERROR(status);
442475
if (_table_reader->current_split_pruned()) {
443476
_state->update_num_finished_scan_range(1);
@@ -458,6 +491,21 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) {
458491

459492
VExprContextSPtrs table_conjuncts;
460493
RETURN_IF_ERROR(_build_table_conjuncts(&table_conjuncts));
494+
std::optional<std::vector<format::GlobalIndex>> push_down_count_columns;
495+
const auto& push_down_count_slot_ids = _local_state->get_push_down_count_slot_ids();
496+
if (push_down_count_slot_ids.has_value()) {
497+
push_down_count_columns.emplace();
498+
push_down_count_columns->reserve(push_down_count_slot_ids->size());
499+
for (const auto slot_id : *push_down_count_slot_ids) {
500+
const auto global_index_it = _slot_id_to_global_index.find(slot_id);
501+
if (global_index_it == _slot_id_to_global_index.end()) {
502+
return Status::InternalError(
503+
"Pushed-down COUNT argument is not a projected file scan slot, slot_id={}",
504+
slot_id);
505+
}
506+
push_down_count_columns->push_back(global_index_it->second);
507+
}
508+
}
461509
RETURN_IF_ERROR(_table_reader->init({
462510
.projected_columns = _projected_columns,
463511
.conjuncts = std::move(table_conjuncts),
@@ -468,6 +516,7 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) {
468516
.scanner_profile = _local_state->scanner_profile(),
469517
.file_slot_descs = &_file_slot_descs,
470518
.push_down_agg_type = _local_state->get_push_down_agg_type(),
519+
.push_down_count_columns = std::move(push_down_count_columns),
471520
.condition_cache_digest = _local_state->get_condition_cache_digest(),
472521
}));
473522
return Status::OK();
@@ -540,6 +589,14 @@ bool FileScannerV2::_should_skip_not_found(const Status& status, bool ignore_not
540589
return ignore_not_found && status.is<ErrorCode::NOT_FOUND>();
541590
}
542591

592+
bool FileScannerV2::_should_skip_empty(const Status& status, bool stopped) {
593+
// Several readers use END_OF_FILE both for a valid zero-row split and for an interrupted IO.
594+
// For example, DeletionVectorReader returns END_OF_FILE("stop read.") after try_stop() marks
595+
// the shared IOContext. That status must unwind the stopped scanner; counting it as an empty
596+
// file would incorrectly finish the scan range and increment EmptyFileNum.
597+
return !stopped && status.is<ErrorCode::END_OF_FILE>();
598+
}
599+
543600
bool FileScannerV2::_should_enable_file_meta_cache() const {
544601
return ExecEnv::GetInstance()->file_meta_cache()->enabled() &&
545602
_split_source->num_scan_ranges() < config::max_external_file_meta_cache_num / 3;
@@ -809,10 +866,17 @@ bool FileScannerV2::_should_enable_adaptive_batch_size(TFileFormatType::type for
809866
}
810867

811868
bool FileScannerV2::_should_run_adaptive_batch_size() const {
812-
// COUNT pushdown emits synthetic rows from file metadata and does not materialize file columns,
813-
// so there is no useful row-width sample to learn from.
814-
return _block_size_predictor != nullptr &&
815-
_local_state->get_push_down_agg_type() != TPushAggOp::type::COUNT;
869+
DORIS_CHECK(_table_reader != nullptr);
870+
return _should_run_adaptive_batch_size(_block_size_predictor != nullptr,
871+
_table_reader->current_split_uses_metadata_count());
872+
}
873+
874+
bool FileScannerV2::_should_run_adaptive_batch_size(bool predictor_initialized,
875+
bool current_split_uses_metadata_count) {
876+
// Metadata COUNT emits synthetic rows and has no physical row width to learn from. A raw COUNT
877+
// opcode is not sufficient here: unsupported argument counts, mappings, filters, or deletes
878+
// make TableReader fall back to materializing normal rows, which still need adaptive batching.
879+
return predictor_initialized && !current_split_uses_metadata_count;
816880
}
817881

818882
size_t FileScannerV2::_predict_reader_batch_rows() {

‎be/src/exec/scan/file_scanner_v2.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,12 @@ class FileScannerV2 final : public Scanner {
8787
static void TEST_report_file_cache_profile(
8888
RuntimeProfile* profile, const io::FileCacheStatistics& file_cache_statistics);
8989
static bool TEST_should_skip_not_found(const Status& status, bool ignore_not_found);
90+
static bool TEST_should_skip_empty(const Status& status, bool stopped);
91+
static bool TEST_should_run_adaptive_batch_size(bool predictor_initialized,
92+
bool current_split_uses_metadata_count) {
93+
return _should_run_adaptive_batch_size(predictor_initialized,
94+
current_split_uses_metadata_count);
95+
}
9096
#endif
9197

9298
FileScannerV2(RuntimeState* state, FileScanLocalState* parent, int64_t limit,
@@ -121,6 +127,7 @@ class FileScannerV2 final : public Scanner {
121127
Status _prepare_table_reader_split(const TFileRangeDesc& range,
122128
std::map<std::string, Field> partition_values);
123129
static bool _should_skip_not_found(const Status& status, bool ignore_not_found);
130+
static bool _should_skip_empty(const Status& status, bool stopped);
124131
bool _should_enable_file_meta_cache() const;
125132
std::optional<format::GlobalRowIdContext> _create_global_rowid_context(
126133
const TFileRangeDesc& range) const;
@@ -138,6 +145,8 @@ class FileScannerV2 final : public Scanner {
138145
void _init_adaptive_batch_size_state(TFileFormatType::type format_type);
139146
bool _should_enable_adaptive_batch_size(TFileFormatType::type format_type) const;
140147
bool _should_run_adaptive_batch_size() const;
148+
static bool _should_run_adaptive_batch_size(bool predictor_initialized,
149+
bool current_split_uses_metadata_count);
141150
size_t _predict_reader_batch_rows();
142151
void _update_adaptive_batch_size(const Block& block);
143152
static RealtimeCounterDeltas _collect_realtime_counter_deltas(
@@ -181,6 +190,7 @@ class FileScannerV2 final : public Scanner {
181190
ShardedKVCache* _kv_cache = nullptr;
182191

183192
RuntimeProfile::Counter* _get_block_timer = nullptr;
193+
RuntimeProfile::Counter* _empty_file_counter = nullptr;
184194
RuntimeProfile::Counter* _not_found_file_counter = nullptr;
185195
RuntimeProfile::Counter* _file_counter = nullptr;
186196
RuntimeProfile::Counter* _file_read_bytes_counter = nullptr;

0 commit comments

Comments
 (0)