Skip to content

Commit 26b40dd

Browse files
authored
fix: honor Parquet byte-array statistics ordering (apache#24525)
## Why are the changes needed? ### Which issue does this PR close? Part of apache#10586. This fixes the unsigned string/binary byte-array case; it does not claim to resolve every Parquet ordering issue. It is also the correctness prerequisite for apache#24526. ### Rationale for this change A Parquet scan can skip a file, row group, or page when its statistics prove that no row can satisfy the filter. If those statistics use a different comparison order from the query, that proof is invalid: DataFusion can silently discard a row that should be returned. For example, suppose a Parquet row group contains `'aé'`, `'az'`, and `'b'`: ```sql SELECT s FROM t WHERE s = 'az'; -- Expected: one row containing 'az' ``` Parquet's deprecated byte-array `min`/`max` fields use signed comparison, whereas Arrow compares strings using unsigned UTF-8 bytes. The same values therefore have two different orders: | Comparison | Values in ascending order | Minimum / maximum | | --- | --- | --- | | Legacy signed-byte order | `aé < az < b` | `aé` / `b` | | Arrow's unsigned-byte order | `az < aé < b` | `az` / `b` | The first UTF-8 byte of `é` is `0xC3`: it sorts before `z`'s `0x7A` as a signed byte, but after it as an unsigned byte. If DataFusion interprets the legacy interval `['aé', 'b']` using Arrow's ordering, it sees `'az' < 'aé'` and can incorrectly skip the row group. Merely checking that `min <= max` does not help: the two reported endpoints are still in ascending order. The newer `min_value`/`max_value` fields and page-index bounds also need an ordering declaration that the reader understands. A missing or unknown `column_orders` entry is not enough to justify assuming Arrow's ordering. The [Parquet statistics definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) and [logical-type ordering rules](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) describe these distinctions. This bug is independently observable with an ordinary equality filter; it does not require a large `IN` list. ## What changes were proposed in this PR? ### What changes are included in this PR? The change makes a recognized comparison order a prerequisite for using unsigned byte-array bounds. At the point where Parquet metadata becomes Arrow statistics, DataFusion checks whether the column's physical/logical type and footer establish the expected unsigned order. It also rejects row-group bounds taken from the deprecated signed-order fields. When that evidence is missing, min/max is reported as unknown, so pruning keeps potentially matching data instead of guessing. The rule is applied at the granularity where the statistics are used. An unsafe row group prevents DataFusion from claiming a trustworthy bound for the whole file, but it does not make other row groups' valid bounds unusable. Page-index bounds are checked against the footer independently. The same safeguards cover static and runtime row-group pruning, as well as the inverse predicates used to decide whether every row already satisfies a filter. Null counts and unrelated columns' statistics remain available. The row-group pruning API gains a metadata-aware entry point so callers can supply the footer needed for this decision. The existing entry point remains source-compatible and behaves conservatively when that information is unavailable. Signed logical types such as decimals retain their existing behavior. ### Are there any user-facing changes? Queries no longer discard matching data because of these untrustworthy byte-array bounds. Older files, or files with an unrecognized ordering, may require more scanning. Modern files with trustworthy bounds retain min/max pruning. There is no file-format change or breaking public API change. ## How was this PR tested? ### Are these changes tested? The regression uses actual serialized Parquet files containing the example above, with modern, deprecated, missing-order, and unknown-order metadata. It checks that the matching `az` row survives file, row-group, runtime, and page pruning, while valid statistics can still eliminate unrelated data. On unchanged Apache `f1f0449a`, the adapted equality regression fails because the deprecated-order case loses `az`; the modern-statistics control passes. The dedicated Parquet-crate run passed 231 unit tests and four doctests. Its seven focused ordering tests also cover mixed safe/unsafe row groups, null counts, fixed-length binary and UUID, signed decimal, and logical types with undefined ordering. Formatting, all-targets/all-features Clippy with warnings denied, and `./dev/rust_lint.sh` passed. The extended workspace run passed 10,666 Rust tests, with eight ignored, and all 503 SQL-logic files. The existing metadata benchmark was run on Apache `f1f0449a` and this patch using the same valid modern-footer fixture. Across nine full-statistics cases there was no material regression; the largest case, with 256 columns and 128 row groups, measured 1.330 ms before and 1.332 ms after. Both runs used Rust 1.97.0, `release-nonlto`, 20 samples, and separate build directories on an Apple M5 Max. <details> <summary>Validation commands</summary> ```sh cargo test --locked --profile ci -p datafusion-datasource-parquet cargo bench --locked --profile release-nonlto -p datafusion-datasource-parquet \ --bench parquet_metadata_statistics -- \ metadata_full --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot RUST_BACKTRACE=1 cargo test --locked --profile ci \ --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \ --workspace --lib --tests --bins \ --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption ``` </details>
1 parent a62e84a commit 26b40dd

8 files changed

Lines changed: 956 additions & 24 deletions

File tree

‎datafusion/datasource-parquet/benches/parquet_metadata_statistics.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
2828
use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main};
2929
use datafusion_datasource_parquet::metadata::DFParquetMetadata;
3030
use parquet::arrow::ArrowSchemaConverter;
31+
use parquet::basic::ColumnOrder;
3132
use parquet::data_type::ByteArray;
3233
use parquet::file::metadata::{
3334
ColumnChunkMetaData, FileMetaData, ParquetMetaData, RowGroupMetaData,
@@ -169,13 +170,19 @@ fn make_synthetic_metadata(
169170
})
170171
.collect::<Vec<_>>();
171172

173+
// Model a modern writer so byte-array bounds have a trustworthy order.
174+
let column_orders = schema_descr
175+
.columns()
176+
.iter()
177+
.map(|column| ColumnOrder::TYPE_DEFINED_ORDER(column.sort_order()))
178+
.collect();
172179
let file_metadata = FileMetaData::new(
173180
1,
174181
(spec.row_groups * ROWS_PER_GROUP) as i64,
175182
Some("datafusion parquet metadata benchmark".to_string()),
176183
None,
177184
schema_descr,
178-
None,
185+
Some(column_orders),
179186
);
180187

181188
ParquetMetaData::new(file_metadata, row_groups)

‎datafusion/datasource-parquet/src/metadata.rs‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,13 @@ use object_store::{ObjectMeta, ObjectStore};
4343
use parquet::DecodeResult;
4444
use parquet::arrow::arrow_reader::statistics::StatisticsConverter;
4545
use parquet::arrow::{parquet_column, parquet_to_arrow_schema};
46+
use parquet::basic::{ColumnOrder, SortOrder, Type as PhysicalType};
4647
use parquet::file::metadata::{
4748
PageIndexPolicy, ParquetMetaData, ParquetMetaDataPushDecoder, ParquetMetaDataReader,
4849
RowGroupMetaData, SortingColumn,
4950
};
5051
use parquet::file::statistics::Statistics as ParquetStatistics;
51-
use parquet::schema::types::SchemaDescriptor;
52+
use parquet::schema::types::{ColumnDescriptor, SchemaDescriptor};
5253
use std::any::Any;
5354
use std::sync::Arc;
5455

@@ -57,6 +58,60 @@ use std::sync::Arc;
5758
/// would be too unreliable otherwise.
5859
const PARTIAL_NDV_THRESHOLD: f64 = 0.75;
5960

61+
fn requires_unsigned_byte_array_order(column: &ColumnDescriptor) -> bool {
62+
matches!(
63+
column.physical_type(),
64+
PhysicalType::BYTE_ARRAY | PhysicalType::FIXED_LEN_BYTE_ARRAY
65+
) && column.sort_order() != SortOrder::SIGNED
66+
}
67+
68+
/// Whether a column's min/max bounds lack a comparison order matching Arrow's.
69+
///
70+
/// The deprecated Parquet `min`/`max` fields use signed comparison, unlike
71+
/// Arrow's string and binary comparisons. Even the modern bounds cannot be
72+
/// interpreted without the corresponding footer `column_orders` entry.
73+
/// Signed logical types, such as decimals, retain their existing behavior.
74+
/// Columns with undefined sort orders, such as `INT96`, never have usable
75+
/// min/max bounds regardless of their physical type. The `INT96` check is
76+
/// defensive because parquet-rs does not currently expose those bounds.
77+
pub(crate) fn has_untrusted_min_max_order(
78+
parquet_schema: &SchemaDescriptor,
79+
column_orders: Option<&[ColumnOrder]>,
80+
parquet_column_index: usize,
81+
) -> bool {
82+
let column = parquet_schema.column(parquet_column_index);
83+
if column.sort_order() == SortOrder::UNDEFINED {
84+
return true;
85+
}
86+
requires_unsigned_byte_array_order(&column)
87+
&& (column.sort_order() != SortOrder::UNSIGNED
88+
|| column_orders
89+
.and_then(|orders| orders.get(parquet_column_index))
90+
.copied()
91+
!= Some(ColumnOrder::TYPE_DEFINED_ORDER(SortOrder::UNSIGNED)))
92+
}
93+
94+
/// Whether any row group provides byte-array bounds in the deprecated signed
95+
/// order, or the column's logical order is undefined.
96+
pub(crate) fn has_untrusted_byte_array_stats<'a>(
97+
parquet_schema: &SchemaDescriptor,
98+
parquet_column_index: Option<usize>,
99+
row_groups: impl IntoIterator<Item = &'a RowGroupMetaData>,
100+
) -> bool {
101+
parquet_column_index.is_some_and(|index| {
102+
let column = parquet_schema.column(index);
103+
requires_unsigned_byte_array_order(&column)
104+
&& (column.sort_order() != SortOrder::UNSIGNED
105+
|| row_groups.into_iter().any(|group| {
106+
group.column(index).statistics().is_some_and(|stats| {
107+
stats.is_min_max_deprecated()
108+
&& (stats.min_bytes_opt().is_some()
109+
|| stats.max_bytes_opt().is_some())
110+
})
111+
}))
112+
})
113+
}
114+
60115
/// Handles fetching Parquet file schema, metadata and statistics
61116
/// from object store.
62117
///
@@ -496,6 +551,23 @@ impl<'a> DFParquetMetadata<'a> {
496551
file_metadata.schema_descr(),
497552
) {
498553
Ok(stats_converter) => {
554+
let parquet_index = stats_converter.parquet_column_index();
555+
if parquet_index.is_some_and(|index| {
556+
has_untrusted_min_max_order(
557+
file_metadata.schema_descr(),
558+
file_metadata.column_orders().map(Vec::as_slice),
559+
index,
560+
)
561+
}) || has_untrusted_byte_array_stats(
562+
file_metadata.schema_descr(),
563+
parquet_index,
564+
row_groups_metadata,
565+
) {
566+
// The remaining row groups cannot establish bounds
567+
// for the whole file. Keep unrelated statistics.
568+
min_accs[idx] = None;
569+
max_accs[idx] = None;
570+
}
499571
let mut accumulators = StatisticsAccumulators {
500572
min_accs: &mut min_accs,
501573
max_accs: &mut max_accs,

‎datafusion/datasource-parquet/src/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ mod schema_coercion;
4242
mod sink;
4343
mod sort;
4444
pub mod source;
45+
#[cfg(test)]
46+
mod statistics_order_tests;
4547
mod supported_predicates;
4648
#[cfg(test)]
4749
mod test_util;

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1129,10 +1129,9 @@ impl FiltersPreparedParquetOpen {
11291129
// If there is a predicate that can be evaluated against the metadata
11301130
if let Some(predicate) = self.pruning_predicate.as_ref().map(|p| p.as_ref()) {
11311131
if prepared.enable_row_group_stats_pruning {
1132-
row_groups.prune_by_statistics(
1132+
row_groups.prune_by_statistics_with_metadata(
11331133
&prepared.physical_file_schema,
1134-
loaded.reader_metadata.parquet_schema(),
1135-
rg_metadata,
1134+
&file_metadata,
11361135
predicate,
11371136
&prepared.file_metrics,
11381137
);

‎datafusion/datasource-parquet/src/page_filter.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ use std::sync::Arc;
2222

2323
use super::metrics::ParquetFileMetrics;
2424
use crate::ParquetAccessPlan;
25+
use crate::metadata::has_untrusted_min_max_order;
2526

2627
use arrow::array::BooleanArray;
2728
use arrow::{
@@ -490,6 +491,7 @@ struct PagesPruningStatistics<'a> {
490491
column_index: &'a ParquetColumnIndex,
491492
offset_index: &'a ParquetOffsetIndex,
492493
page_offsets: &'a Vec<PageLocation>,
494+
trusted_min_max: bool,
493495
}
494496

495497
impl<'a> PagesPruningStatistics<'a> {
@@ -529,6 +531,12 @@ impl<'a> PagesPruningStatistics<'a> {
529531
return None;
530532
};
531533
let page_offsets = offset_index_metadata.page_locations();
534+
let file_metadata = parquet_metadata.file_metadata();
535+
let trusted_min_max = !has_untrusted_min_max_order(
536+
file_metadata.schema_descr(),
537+
file_metadata.column_orders().map(Vec::as_slice),
538+
parquet_column_index,
539+
);
532540

533541
Some(Self {
534542
row_group_index,
@@ -537,6 +545,7 @@ impl<'a> PagesPruningStatistics<'a> {
537545
column_index,
538546
offset_index,
539547
page_offsets,
548+
trusted_min_max,
540549
})
541550
}
542551

@@ -563,6 +572,9 @@ impl<'a> PagesPruningStatistics<'a> {
563572
}
564573
impl PruningStatistics for PagesPruningStatistics<'_> {
565574
fn min_values(&self, _column: &datafusion_common::Column) -> Option<ArrayRef> {
575+
if !self.trusted_min_max {
576+
return None;
577+
}
566578
match self.converter.data_page_mins(
567579
self.column_index,
568580
self.offset_index,
@@ -577,6 +589,9 @@ impl PruningStatistics for PagesPruningStatistics<'_> {
577589
}
578590

579591
fn max_values(&self, _column: &datafusion_common::Column) -> Option<ArrayRef> {
592+
if !self.trusted_min_max {
593+
return None;
594+
}
580595
match self.converter.data_page_maxes(
581596
self.column_index,
582597
self.offset_index,

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,11 @@ impl RowGroupPruner {
232232
.collect::<Vec<_>>();
233233
let stats = RowGroupPruningStatistics {
234234
parquet_schema: self.parquet_metadata.file_metadata().schema_descr(),
235+
column_orders: self
236+
.parquet_metadata
237+
.file_metadata()
238+
.column_orders()
239+
.map(Vec::as_slice),
235240
row_group_metadatas,
236241
arrow_schema: self.arrow_schema.as_ref(),
237242
// Match the existing static row-group pruning behavior: when a

0 commit comments

Comments
 (0)