Skip to content

Commit 2a4191c

Browse files
authored
[improvement](be) Use zero NaN counts for Parquet floating-point pruning (#69655)
### What problem does this PR solve? Problem Summary: Parquet V2 treats every FLOAT/DOUBLE column as having an unknown NaN count. Finite min/max bounds omit NaNs, so the existing comparison and NOT IN guards correctly retain groups that might contain a matching NaN. This also prevents pruning when a writer explicitly supplies `nan_count=0`. For example, a column chunk with `min=max=1` and a trustworthy zero NaN count cannot match `x > 1`, `x >= 2`, `x <> 1`, or `x NOT IN (1)`. Previously the reader decoded those rows anyway. This change makes that zero proof available to the existing ZoneMap evaluator. ### Optimization approach - Add standard optional Statistics field 9, `nan_count`, matching the [Apache Parquet format definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift). Compared the touched definition on master and branch-3.1/4.0/4.1/4.2: existing field IDs retain their meanings and field 9 is unused on those Doris branches. Older readers skip the optional field, and existing files retain the conservative fallback. - Retain a zero-count proof in converted FLOAT/DOUBLE statistics only when the count is explicitly zero, the value/null counts are consistent, and the decoded bounds are usable (or the chunk is all NULL). Missing, positive, negative, or excessive counts do not clear the NaN-unknown flag. Invalid/NaN/unordered bounds remain unusable. - Pass the proof alongside the row-group ZoneMap. For PageIndex evaluation, reuse only the proof from the exact owning row group and physical leaf column. It covers every page of that chunk. Both single-slot and compound OR/AND page evaluation use this scoped proof. - Keep the existing comparator and NOT IN semantics. Integer/string columns and legacy floating files avoid the additional footer decode in the page proof path. Writer support and additional column orders are outside this reader change. ### Optimization effect | Per reader query | Absent count (previous fallback) | Explicit zero count | | --- | ---: | ---: | | RawRowsRead | 524,288 | 4,096 (-99.22%) | | RowGroupsFilteredByMinMax | 0 | 127 | | Actual local read bytes | 6,358,496 | 98,444 (-98.45%) | | Median wall time | 6.908 ms | 0.690 ms (~10.0x) | | Median process CPU time | 6.868 ms | 0.687 ms (~10.0x) | Measured using Clang 20 RELEASE, warm local filesystem cache, one pinned CPU. The absent-count samples range from 6.787 to 7.252 ms and zero-count samples from 0.673 to 0.730 ms. All returned IDs and per-query work counters match their expectations across all samples. The opt-in `ParquetScanTest.ZeroNanCountReaderMicrobench` uses the actual V2 reader and corresponds to: ```sql SELECT id FROM parquet_source WHERE d > 1; ``` Fixture: 128 row groups x 4,096 rows; the first 127 groups contain `d=1`, the last contains `d=2`. FLOAT, DOUBLE, and an integer row identifier are stored using uncompressed PLAIN DataPageV2 with dictionary and PageIndex disabled. Two files have identical data pages and differ only in the optional zero-count fields. The absent-count variant exercises the same conservative behavior as the pre-change reader. Each query checks every returned ID and the pruning counters. Seven samples of eight reader queries per variant follow warmup, alternating variant order. The benchmark records wall time, process CPU time, actual local read bytes, RawRowsRead, and RowGroupsFilteredByMinMax. Results describe a local reader microbenchmark; full SQL timings depend on storage, parallelism, and the rest of the query. ### Validation and reproduction - Related BE suites: 372 tests passed; the opt-in benchmark was skipped in that run and then passed separately. - The three initial reader regressions first failed on unmodified master because zero counts did not reduce RawRowsRead or increase pruning. Their result rows were already correct; the missing behavior was pruning. - Generated C++ Thrift code compiled successfully with the optional field. Old files without the field remain covered by the complete reader suite. - clang-format 16 and BE build-hygiene checks passed. Changed-line clang-tidy passed on all four affected C++ files. Coverage includes FLOAT/DOUBLE; zero/positive/absent/negative/excessive counts; GT/GE/NE/NOT IN; invalid null counts and bounds; all NULL/all NaN/mixed NULL-finite-NaN rows; dictionary on/off; DataPageV1/V2; PageIndex on/off; column/row-group proof scope; and compound OR page pruning when one branch lacks a zero proof. Mixed NaN cases are also checked against a statistics-disabled file. Correctness suites: ```bash BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \ --filter='*Parquet*:*ColumnMapper*:*TableReader*' ``` Opt-in benchmark: ```bash DORIS_PARQUET_NAN_COUNT_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \ ./run-be-ut.sh -j8 --run --filter=ParquetScanTest.ZeroNanCountReaderMicrobench ``` ### Release note Improve Parquet V2 floating-point Min-Max pruning when trustworthy zero NaN counts are available. ### Check List (For Author) - Test - [x] Unit Test - [x] Manual test (reader microbenchmark) - Behavior changed: - [x] Yes. Proven NaN-free floating chunks/pages can be skipped by existing comparison and NOT IN predicates. - Does this need documentation? - [x] No. Internal reader optimization using existing file metadata. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent fc1b822 commit 2a4191c

5 files changed

Lines changed: 494 additions & 36 deletions

File tree

‎be/src/format_v2/parquet/parquet_statistics.cpp‎

Lines changed: 75 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -942,10 +942,12 @@ std::shared_ptr<segment_v2::ZoneMap> make_zonemap_from_statistics(
942942
}
943943

944944
void add_slot_zonemap(ZoneMapEvalContext* ctx, int slot_index, const DataTypePtr& data_type,
945-
std::shared_ptr<segment_v2::ZoneMap> zone_map) {
945+
std::shared_ptr<segment_v2::ZoneMap> zone_map, bool has_zero_nan_count) {
946946
DORIS_CHECK(ctx != nullptr);
947947
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
948948
slot_zone_map.set_data_type_from_parquet(data_type);
949+
// A finite min=max=1 alone cannot exclude NaN > 1; a validated zero count can.
950+
slot_zone_map.floating_nan_count_unknown &= !has_zero_nan_count;
949951
slot_zone_map.zone_map = std::move(zone_map);
950952
ctx->slots.emplace(slot_index, std::move(slot_zone_map));
951953
}
@@ -1015,6 +1017,15 @@ ParquetColumnStatistics ParquetStatisticsUtils::TransformColumnStatistics(
10151017
result.has_null_count = false;
10161018
result.has_null = true;
10171019
}
1020+
// Accept only an explicit zero in consistent floating statistics. Missing/positive/invalid
1021+
// counts, impossible null counts, and NaN or unordered bounds cannot establish this proof.
1022+
const auto physical_type = column_schema.type_descriptor.physical_type;
1023+
result.has_zero_nan_count =
1024+
(physical_type == tparquet::Type::FLOAT || physical_type == tparquet::Type::DOUBLE) &&
1025+
statistics->__isset.nan_count && statistics->nan_count == 0 &&
1026+
(!statistics->__isset.null_count ||
1027+
(statistics->null_count >= 0 && statistics->null_count <= column_value_count)) &&
1028+
(result.has_min_max || !result.has_not_null);
10181029
return result;
10191030
}
10201031

@@ -1035,6 +1046,49 @@ bool ParquetStatisticsUtils::NativeBloomFilterExcludes(
10351046

10361047
namespace {
10371048

1049+
ParquetColumnStatistics native_footer_column_statistics(const tparquet::FileMetaData& metadata,
1050+
const tparquet::RowGroup& row_group,
1051+
const ParquetColumnSchema& column_schema,
1052+
const cctz::time_zone* timezone,
1053+
bool null_count_trusted) {
1054+
if (column_schema.leaf_column_id < 0 ||
1055+
std::cmp_greater_equal(column_schema.leaf_column_id, row_group.columns.size())) {
1056+
return {};
1057+
}
1058+
const auto& chunk = row_group.columns[column_schema.leaf_column_id];
1059+
if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics) {
1060+
return {};
1061+
}
1062+
const auto& column = chunk.meta_data;
1063+
const auto statistics = detail::sanitize_native_footer_statistics(
1064+
column_schema.type_descriptor, column.statistics,
1065+
detail::has_supported_type_defined_order(metadata, column_schema.leaf_column_id));
1066+
return ParquetStatisticsUtils::TransformColumnStatistics(
1067+
column_schema, &statistics, column.num_values, timezone, null_count_trusted);
1068+
}
1069+
1070+
bool native_footer_has_zero_nan_count(const tparquet::FileMetaData& metadata,
1071+
const tparquet::RowGroup& row_group,
1072+
const ParquetColumnSchema& column_schema,
1073+
const cctz::time_zone* timezone, bool null_count_trusted) {
1074+
const auto physical = column_schema.type_descriptor.physical_type;
1075+
if ((physical != tparquet::Type::FLOAT && physical != tparquet::Type::DOUBLE) ||
1076+
column_schema.leaf_column_id < 0 ||
1077+
std::cmp_greater_equal(column_schema.leaf_column_id, row_group.columns.size())) {
1078+
return false;
1079+
}
1080+
const auto& chunk = row_group.columns[column_schema.leaf_column_id];
1081+
// Avoid decoding footer bounds again for integer/string columns and legacy floating files.
1082+
if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics ||
1083+
!chunk.meta_data.statistics.__isset.nan_count ||
1084+
chunk.meta_data.statistics.nan_count != 0) {
1085+
return false;
1086+
}
1087+
return native_footer_column_statistics(metadata, row_group, column_schema, timezone,
1088+
null_count_trusted)
1089+
.has_zero_nan_count;
1090+
}
1091+
10381092
void collect_filtered_leaf_ids(const ParquetColumnSchema& column_schema,
10391093
const format::LocalColumnIndex* projection,
10401094
std::set<int>* leaf_column_ids) {
@@ -1092,24 +1146,11 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata,
10921146
column_schema->leaf_column_id >= static_cast<int>(row_group.columns.size())) {
10931147
return;
10941148
}
1095-
const auto& chunk = row_group.columns[column_schema->leaf_column_id];
1096-
std::shared_ptr<segment_v2::ZoneMap> zone_map;
1097-
if (chunk.__isset.meta_data) {
1098-
const auto& column_metadata = chunk.meta_data;
1099-
std::optional<tparquet::Statistics> safe_statistics;
1100-
if (column_metadata.__isset.statistics) {
1101-
safe_statistics = detail::sanitize_native_footer_statistics(
1102-
column_schema->type_descriptor, column_metadata.statistics,
1103-
detail::has_supported_type_defined_order(metadata,
1104-
column_schema->leaf_column_id));
1105-
}
1106-
zone_map = ParquetStatisticsUtils::MakeZoneMap(
1107-
ParquetStatisticsUtils::TransformColumnStatistics(
1108-
*column_schema,
1109-
safe_statistics.has_value() ? &*safe_statistics : nullptr,
1110-
column_metadata.num_values, timezone, null_count_trusted));
1111-
}
1112-
add_slot_zonemap(ctx, slot_index, column_schema->type, std::move(zone_map));
1149+
const auto statistics = native_footer_column_statistics(metadata, row_group, *column_schema,
1150+
timezone, null_count_trusted);
1151+
add_slot_zonemap(ctx, slot_index, column_schema->type,
1152+
ParquetStatisticsUtils::MakeZoneMap(statistics),
1153+
statistics.has_zero_nan_count);
11131154
};
11141155

11151156
VExprContextSPtrs direct_conjuncts;
@@ -1881,13 +1922,14 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t
18811922
class NativePageIndexPredicateEvaluator {
18821923
public:
18831924
NativePageIndexPredicateEvaluator(
1884-
const tparquet::FileMetaData& metadata,
1925+
const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group,
18851926
const std::unordered_map<int, NativeParquetPageIndex>& page_indexes,
18861927
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
18871928
const format::FileScanRequest& request, int64_t row_group_rows,
18881929
ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone,
18891930
bool null_count_trusted)
18901931
: _metadata(metadata),
1932+
_row_group(row_group),
18911933
_page_indexes(page_indexes),
18921934
_file_schema(file_schema),
18931935
_request(request),
@@ -1912,6 +1954,7 @@ class NativePageIndexPredicateEvaluator {
19121954
private:
19131955
struct SlotPageZoneMaps {
19141956
DataTypePtr data_type;
1957+
bool has_zero_nan_count = false;
19151958
std::vector<RowRange> ranges;
19161959
std::vector<std::shared_ptr<segment_v2::ZoneMap>> zone_maps;
19171960
};
@@ -1964,7 +2007,8 @@ class NativePageIndexPredicateEvaluator {
19642007
std::vector<RowRange> ranges;
19652008
for (size_t page_idx = 0; page_idx < pages->ranges.size(); ++page_idx) {
19662009
ZoneMapEvalContext ctx;
1967-
add_slot_zonemap(&ctx, slot_index, pages->data_type, pages->zone_maps[page_idx]);
2010+
add_slot_zonemap(&ctx, slot_index, pages->data_type, pages->zone_maps[page_idx],
2011+
pages->has_zero_nan_count);
19682012
if (expr->evaluate_zonemap_filter(ctx) != ZoneMapFilterResult::kNoMatch) {
19692013
append_row_range(pages->ranges[page_idx], &ranges);
19702014
}
@@ -1999,6 +2043,10 @@ class NativePageIndexPredicateEvaluator {
19992043
const auto& indexes = index_it->second;
20002044
SlotPageZoneMaps pages;
20012045
pages.data_type = column_schema->type;
2046+
// The owning column chunk's zero covers all its pages. A count from another row group or
2047+
// column would be unsafe, e.g. if this column has finite bounds but also contains a NaN.
2048+
pages.has_zero_nan_count = native_footer_has_zero_nan_count(
2049+
_metadata, _row_group, *column_schema, _timezone, _null_count_trusted);
20022050
pages.ranges.reserve(indexes.offset_index.page_locations.size());
20032051
pages.zone_maps.reserve(indexes.offset_index.page_locations.size());
20042052
for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size();
@@ -2020,6 +2068,7 @@ class NativePageIndexPredicateEvaluator {
20202068
}
20212069

20222070
const tparquet::FileMetaData& _metadata;
2071+
const tparquet::RowGroup& _row_group;
20232072
const std::unordered_map<int, NativeParquetPageIndex>& _page_indexes;
20242073
const std::vector<std::unique_ptr<ParquetColumnSchema>>& _file_schema;
20252074
const format::FileScanRequest& _request;
@@ -2088,6 +2137,8 @@ Status select_row_group_ranges_by_native_page_index(
20882137
continue;
20892138
}
20902139
const auto& indexes = index_it->second;
2140+
const bool has_zero_nan_count = native_footer_has_zero_nan_count(
2141+
metadata, row_group, *column_schema, timezone, null_count_trusted);
20912142
std::vector<RowRange> filter_ranges;
20922143
bool usable = true;
20932144
for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size();
@@ -2103,7 +2154,7 @@ Status select_row_group_ranges_by_native_page_index(
21032154
}
21042155
ZoneMapEvalContext ctx;
21052156
add_slot_zonemap(&ctx, slot_index, column_schema->type,
2106-
ParquetStatisticsUtils::MakeZoneMap(statistics));
2157+
ParquetStatisticsUtils::MakeZoneMap(statistics), has_zero_nan_count);
21072158
if (VExprContext::evaluate_zonemap_filter(conjuncts, ctx) !=
21082159
ZoneMapFilterResult::kNoMatch) {
21092160
append_row_range(page_range, &filter_ranges);
@@ -2123,8 +2174,8 @@ Status select_row_group_ranges_by_native_page_index(
21232174
}
21242175
}
21252176

2126-
NativePageIndexPredicateEvaluator evaluator(metadata, page_indexes, file_schema, request,
2127-
row_group_rows, pruning_stats, timezone,
2177+
NativePageIndexPredicateEvaluator evaluator(metadata, row_group, page_indexes, file_schema,
2178+
request, row_group_rows, pruning_stats, timezone,
21282179
null_count_trusted);
21292180
for (const auto& conjunct : multi_slot_conjuncts) {
21302181
auto conjunct_ranges = evaluator.evaluate(conjunct->root());

‎be/src/format_v2/parquet/parquet_statistics.h‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -95,12 +95,13 @@ struct ParquetPruningStats {
9595
};
9696

9797
struct ParquetColumnStatistics {
98-
Field min_value; // column minimum value converted to Doris type
99-
Field max_value; // column maximum value
100-
bool has_null = false; // whether NULL exists
101-
bool has_not_null = false; // whether non-NULL values exist
102-
bool has_null_count = false; // whether null_count is valid
103-
bool has_min_max = false; // whether min/max is valid after conversion
98+
Field min_value; // column minimum value converted to Doris type
99+
Field max_value; // column maximum value
100+
bool has_null = false; // whether NULL exists
101+
bool has_not_null = false; // whether non-NULL values exist
102+
bool has_null_count = false; // whether null_count is valid
103+
bool has_min_max = false; // whether min/max is valid after conversion
104+
bool has_zero_nan_count = false; // validated proof covering this column's statistics
104105

105106
bool has_any_statistics() const { return has_null_count || has_min_max; }
106107
};

0 commit comments

Comments
 (0)