Skip to content

Commit 9082d6b

Browse files
fix: discard parquet bounds when row group statistics are missing (#25228)
## Which issue does this PR close? Closes #25226. ## Rationale for this change MIN/MAX queries can return incorrect results when a Parquet file contains a mix of row groups with and without bounds. A missing bound is converted to a NULL in the per-row-group statistics array. Aggregating that array skips the NULL, allowing another group's endpoint to be reported as an exact bound for the entire file. The optimizer can then answer the aggregate from incomplete statistics. For example, a file contains `["a", "b"]` with statistics and two 8192-character strings starting with `z` without statistics. `SELECT LENGTH(MIN(a)), LENGTH(MAX(a))` previously returned `1, 1`; the correct result is `1, 8192`. ## What changes are included in this PR? - Invalidate a file-level bound when any row group has no converted bound and may contain non-NULL values. Return `Precision::Absent` so that incomplete statistics cannot establish the file's extrema. - Handle minimum and maximum independently, preserving a usable bound when only the other endpoint is missing. - Preserve bounds from other groups when the group with no bound is empty or is proven all NULL. The all-NULL check requires both the column value count and the explicitly recorded NULL count to equal the row count. - Express the active-accumulator precondition with `expect`: callers already check for `Some`, and the path that clears the accumulator returns immediately. This removes an unreachable fallback return. ## What is the testing strategy for this PR? `test_min_max_require_complete_row_group_bounds` covers absent statistics, independently missing minimum/maximum values, unknown and partial NULL counts, empty groups, all-NULL groups, and complete bounds. Every case runs in both row-group orders. `parquet_missing_bounds.slt` verifies both the stored string lengths and the aggregate results. Its fixture is generated at runtime using public Rust Parquet writer APIs, enabling chunk statistics for the first row group and disabling them for the second. Disabling statistics is deliberate: parquet-rs truncates long extrema rather than omitting them. No generated Parquet file is committed and no footer is manually edited. Local validation: - All 10 tests in `metadata::tests::statistics_tests` passed. - `parquet_missing_bounds.slt` passed with the fix. Running the same fixture and SQL test against unmodified main reproduced the incorrect `1, 1` result. - LLVM coverage confirmed that the removed fallback return had zero executions and that the replacement precondition assertion was exercised 36 times by the statistics tests. The updated Codecov patch percentage remains subject to the next CI run. ## Are there any user-facing changes? MIN/MAX queries return correct results for Parquet files with incomplete row-group bounds. Valid statistics remain usable for each independently complete endpoint, including when other groups are empty or all NULL. There are no public API changes. --------- Co-authored-by: Jay Zhan <[email protected]>
1 parent b5345a8 commit 9082d6b

3 files changed

Lines changed: 248 additions & 6 deletions

File tree

‎datafusion/datasource-parquet/src/metadata.rs‎

Lines changed: 136 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -739,9 +739,9 @@ fn summarize_column_statistics(
739739
) -> Result<()> {
740740
let parquet_index = stats_converter.parquet_column_index();
741741

742-
if let Some(max_acc) = &mut accumulators.max_accs[logical_schema_index] {
742+
if accumulators.max_accs[logical_schema_index].is_some() {
743743
accumulators.is_max_value_exact[logical_schema_index] = summarize_bound(
744-
max_acc,
744+
&mut accumulators.max_accs[logical_schema_index],
745745
&stats_converter.row_group_maxes(row_groups_metadata)?,
746746
parquet_index,
747747
row_groups_metadata,
@@ -750,9 +750,9 @@ fn summarize_column_statistics(
750750
)?;
751751
}
752752

753-
if let Some(min_acc) = &mut accumulators.min_accs[logical_schema_index] {
753+
if accumulators.min_accs[logical_schema_index].is_some() {
754754
accumulators.is_min_value_exact[logical_schema_index] = summarize_bound(
755-
min_acc,
755+
&mut accumulators.min_accs[logical_schema_index],
756756
&stats_converter.row_group_mins(row_groups_metadata)?,
757757
parquet_index,
758758
row_groups_metadata,
@@ -785,13 +785,41 @@ fn summarize_column_statistics(
785785
/// parquet statistics. `row_group_exactness` rebuilds the exactness as a Boolean
786786
/// array and is only called for the rare case where row groups disagree.
787787
fn summarize_bound<A: Accumulator>(
788-
acc: &mut A,
788+
acc: &mut Option<A>,
789789
values: &ArrayRef,
790790
parquet_index: Option<usize>,
791791
row_groups_metadata: &[RowGroupMetaData],
792792
is_exact: impl Fn(&ParquetStatistics) -> bool,
793793
row_group_exactness: impl FnOnce() -> Result<BooleanArray>,
794794
) -> Result<Option<bool>> {
795+
// A NULL converted bound can mean missing statistics, not just all-NULL
796+
// data. Ignoring it in MIN/MAX would let another row group's exact endpoint
797+
// incorrectly establish an exact bound for the whole file. Drop this bound
798+
// unless the row group is empty or is proven to contain only NULLs.
799+
if values.null_count() > 0
800+
&& parquet_index.is_some_and(|column_index| {
801+
row_groups_metadata
802+
.iter()
803+
.enumerate()
804+
.any(|(index, group)| {
805+
if values.is_valid(index) || group.num_rows() == 0 {
806+
return false;
807+
}
808+
let column = group.column(column_index);
809+
let all_null = column
810+
.statistics()
811+
.and_then(|stats| stats.null_count_opt())
812+
.is_some_and(|nulls| nulls == column.num_values() as u64);
813+
!all_null
814+
})
815+
})
816+
{
817+
*acc = None;
818+
return Ok(None);
819+
}
820+
let Some(acc) = acc.as_mut() else {
821+
return Ok(None);
822+
};
795823
acc.update_batch(&[Arc::clone(values)])?;
796824

797825
Ok(
@@ -1281,7 +1309,7 @@ mod tests {
12811309
);
12821310
}
12831311

1284-
mod ndv_tests {
1312+
mod statistics_tests {
12851313
use super::*;
12861314
use arrow::datatypes::Field;
12871315
use parquet::basic::Type as PhysicalType;
@@ -1678,6 +1706,108 @@ mod tests {
16781706
);
16791707
}
16801708

1709+
#[test]
1710+
fn test_min_max_require_complete_row_group_bounds() {
1711+
let known =
1712+
|| ParquetStatistics::int32(Some(1), Some(3), None, Some(0), false);
1713+
let exact = |v| Precision::Exact(ScalarValue::Int32(Some(v)));
1714+
let cases = [
1715+
(None, 3, Precision::Absent, Precision::Absent),
1716+
(
1717+
Some(ParquetStatistics::int32(
1718+
None,
1719+
Some(6),
1720+
None,
1721+
Some(0),
1722+
false,
1723+
)),
1724+
3,
1725+
Precision::Absent,
1726+
exact(6),
1727+
),
1728+
(
1729+
Some(ParquetStatistics::int32(
1730+
Some(4),
1731+
None,
1732+
None,
1733+
Some(0),
1734+
false,
1735+
)),
1736+
3,
1737+
exact(1),
1738+
Precision::Absent,
1739+
),
1740+
(
1741+
Some(ParquetStatistics::int32(None, None, None, None, false)),
1742+
3,
1743+
Precision::Absent,
1744+
Precision::Absent,
1745+
),
1746+
(
1747+
Some(ParquetStatistics::int32(None, None, None, Some(2), false)),
1748+
3,
1749+
Precision::Absent,
1750+
Precision::Absent,
1751+
),
1752+
// Empty and proven all-NULL row groups do not contribute extrema.
1753+
(None, 0, exact(1), exact(3)),
1754+
(
1755+
Some(ParquetStatistics::int32(None, None, None, Some(3), false)),
1756+
3,
1757+
exact(1),
1758+
exact(3),
1759+
),
1760+
(
1761+
Some(ParquetStatistics::int32(
1762+
Some(4),
1763+
Some(6),
1764+
None,
1765+
Some(0),
1766+
false,
1767+
)),
1768+
3,
1769+
exact(1),
1770+
exact(6),
1771+
),
1772+
];
1773+
for (other, rows, expected_min, expected_max) in cases {
1774+
for reverse in [false, true] {
1775+
let schema_descr = create_schema_descr(1);
1776+
let arrow_schema = create_arrow_schema(1);
1777+
let mut groups = vec![
1778+
create_row_group_with_stats(
1779+
&schema_descr,
1780+
vec![Some(known())],
1781+
3,
1782+
),
1783+
create_row_group_with_stats(
1784+
&schema_descr,
1785+
vec![other.clone()],
1786+
rows,
1787+
),
1788+
];
1789+
if reverse {
1790+
groups.reverse();
1791+
}
1792+
let metadata = create_parquet_metadata(schema_descr, groups);
1793+
let stats = DFParquetMetadata::statistics_from_parquet_metadata(
1794+
&metadata,
1795+
&arrow_schema,
1796+
)
1797+
.unwrap();
1798+
assert_eq!(
1799+
stats.column_statistics[0].min_value, expected_min,
1800+
"other={other:?}, rows={rows}, reverse={reverse}"
1801+
);
1802+
assert_eq!(
1803+
stats.column_statistics[0].max_value, expected_max,
1804+
"other={other:?}, rows={rows}, reverse={reverse}"
1805+
);
1806+
assert_eq!(stats.num_rows, Precision::Exact((3 + rows) as usize));
1807+
}
1808+
}
1809+
}
1810+
16811811
#[test]
16821812
fn test_distinct_count_no_statistics_at_all() {
16831813
// No statistics in row group should return Absent for all stats

‎datafusion/sqllogictest/src/test_context.rs‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,9 @@ impl TestContext {
148148

149149
let file_name = relative_path.file_name().unwrap().to_str().unwrap();
150150
match file_name {
151+
"parquet_missing_bounds.slt" => {
152+
register_parquet_missing_bounds(&mut test_ctx).await;
153+
}
151154
"cte.slt" => {
152155
info!("Registering strict schema provider for CTE tests");
153156
register_strict_schema_provider(test_ctx.session_ctx());
@@ -415,6 +418,77 @@ pub async fn register_partition_table(test_ctx: &mut TestContext) {
415418
.unwrap();
416419
}
417420

421+
/// Write row groups with different statistics settings using the public writer API.
422+
async fn register_parquet_missing_bounds(test_ctx: &mut TestContext) {
423+
use datafusion::parquet::column::writer::ColumnWriterImpl;
424+
use datafusion::parquet::data_type::{ByteArray, ByteArrayType};
425+
use datafusion::parquet::file::properties::{EnabledStatistics, WriterProperties};
426+
use datafusion::parquet::file::writer::{
427+
SerializedFileWriter, SerializedPageWriter, TrackedWrite,
428+
};
429+
use datafusion::parquet::schema::parser::parse_message_type;
430+
431+
test_ctx.enable_testdir();
432+
let path = test_ctx.testdir_path().join("missing_bounds.parquet");
433+
let column_path = test_ctx.testdir_path().join("column.pages");
434+
let schema = Arc::new(
435+
parse_message_type("message schema { REQUIRED BINARY a (UTF8); }").unwrap(),
436+
);
437+
let mut writer = SerializedFileWriter::new(
438+
File::create(&path).unwrap(),
439+
schema,
440+
Arc::new(WriterProperties::default()),
441+
)
442+
.unwrap();
443+
let long_value = "z".repeat(8192);
444+
for (values, statistics) in [
445+
(["a", "b"], EnabledStatistics::Chunk),
446+
(
447+
[long_value.as_str(), long_value.as_str()],
448+
EnabledStatistics::None,
449+
),
450+
] {
451+
// parquet-rs truncates long extrema rather than omitting them. Disable
452+
// statistics for the second chunk to exercise the missing-bound case
453+
// produced naturally by writers such as PyArrow, without editing metadata.
454+
let properties = Arc::new(
455+
WriterProperties::builder()
456+
.set_statistics_enabled(statistics)
457+
.build(),
458+
);
459+
let mut buffer = TrackedWrite::new(File::create(&column_path).unwrap());
460+
let mut column = ColumnWriterImpl::<ByteArrayType>::new(
461+
writer.schema_descr().column(0),
462+
properties,
463+
Box::new(SerializedPageWriter::new(&mut buffer)),
464+
);
465+
let values = values.map(ByteArray::from);
466+
column.write_batch(&values, None, None).unwrap();
467+
let result = column.close().unwrap();
468+
assert_eq!(
469+
result.metadata.statistics().is_some(),
470+
statistics == EnabledStatistics::Chunk
471+
);
472+
buffer.into_inner().unwrap();
473+
let mut group = writer.next_row_group().unwrap();
474+
group
475+
.append_column(&File::open(&column_path).unwrap(), result)
476+
.unwrap();
477+
group.close().unwrap();
478+
}
479+
writer.close().unwrap();
480+
std::fs::remove_file(&column_path).unwrap();
481+
test_ctx
482+
.ctx
483+
.register_parquet(
484+
"missing_bounds",
485+
path.to_str().unwrap(),
486+
ParquetReadOptions::default(),
487+
)
488+
.await
489+
.unwrap();
490+
}
491+
418492
// registers a LOCAL TEMPORARY table.
419493
pub fn register_temp_table(ctx: &SessionContext) {
420494
#[derive(Debug)]
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# The test context writes two row groups using the Rust writer API:
19+
# ["a", "b"] with statistics, and two 8192-character strings without statistics.
20+
# This models the missing bounds produced by writers that omit oversized extrema.
21+
22+
# Verify that the long values are present in the actual data.
23+
query I
24+
SELECT LENGTH(a) FROM missing_bounds ORDER BY a;
25+
----
26+
1
27+
1
28+
8192
29+
8192
30+
31+
# The first group's maximum must not be treated as the entire file's maximum.
32+
query II
33+
SELECT LENGTH(MIN(a)), LENGTH(MAX(a)) FROM missing_bounds;
34+
----
35+
1 8192
36+
37+
statement ok
38+
DROP TABLE missing_bounds;

0 commit comments

Comments
 (0)