Skip to content

Commit 3f41e77

Browse files
committed
fix: prune row groups when statistics collapse the predicate to a constant
Closes apache#24769. `constant_columns_from_stats` substitutes columns that file statistics prove constant — including the all-NULL case, which folds to a NULL literal. When the substituted predicate then simplifies to a bare constant (`NULL`, or `false`), `build_pruning_predicates` returns `None` because there are no column references left, and the opener falls through with no pruning at all. That makes the substitution strictly counterproductive for these files. A predicate like `col = <literal>` over a column whose statistics say `null_count == row_count` is provably unsatisfiable, and before the substitution existed it *was* pruned via the `col_null_count != row_count` conjunct of the pruning predicate. The regression bisects to apache#22969, which enabled `collect_statistics` by default and so began feeding the substitution on paths that previously had no file statistics. Recognise the collapsed-to-constant case explicitly: if the predicate is a `false`/NULL literal and the original predicate referenced a column that `constant_columns_from_stats` proved constant, skip every remaining row group and credit the skip to statistics pruning. Results are unchanged either way — a filter drops NULL and false rows alike — but the scan work is not. The `stats_constants_in_predicate` guard deliberately narrows the skip to collapses that file statistics contributed to. A predicate can also collapse via the missing-column adapter (schema evolution) or partition-value folding; both keep their existing behaviour — partition-driven collapse is already `FilePruner`'s job, and widening the skip to missing columns changes the raw scan output that nine `evolved_schema` / `test_pushdown_with_missing_*` tests assert. The regression test asserts the `row_groups_pruned_statistics` metric rather than the row count, since a row-count assertion cannot tell "pruned" from "scanned, then row-filtered": both yield zero rows.
1 parent 9ef3d47 commit 3f41e77

2 files changed

Lines changed: 160 additions & 3 deletions

File tree

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 149 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,9 @@ use datafusion_common::{
5858
ColumnStatistics, HashSet, Result, ScalarValue, Statistics, exec_err, internal_err,
5959
};
6060
use datafusion_datasource::{PartitionedFile, TableSchema};
61-
use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking};
61+
use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking, Literal};
6262
use datafusion_physical_expr::simplifier::PhysicalExprSimplifier;
63+
use datafusion_physical_expr::utils::collect_columns;
6364
use datafusion_physical_expr_adapter::PhysicalExprAdapterFactory;
6465
use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
6566
use datafusion_physical_expr_common::sort_expr::LexOrdering;
@@ -460,6 +461,11 @@ struct PreparedParquetOpen {
460461
enable_page_index: bool,
461462
enable_bloom_filter: bool,
462463
enable_row_group_stats_pruning: bool,
464+
/// True when the (original) predicate references a column that
465+
/// `constant_columns_from_stats` proved constant for this file, so a
466+
/// predicate that later simplifies to a constant did so on the
467+
/// strength of file statistics — see `prune_row_groups`.
468+
stats_constants_in_predicate: bool,
463469
limit: Option<usize>,
464470
coerce_int96: Option<TimeUnit>,
465471
coerce_int96_tz: Option<Arc<str>>,
@@ -794,10 +800,17 @@ impl ParquetMorselizer {
794800
// Note that if there are statistics for partition columns there will be overlap,
795801
// but since we use a HashMap, we'll just overwrite the partition values with the
796802
// constant values from statistics (which should be the same).
797-
literal_columns.extend(constant_columns_from_stats(
803+
let stats_constants = constant_columns_from_stats(
798804
partitioned_file.statistics.as_deref(),
799805
&logical_file_schema,
800-
));
806+
);
807+
let stats_constants_in_predicate = !stats_constants.is_empty()
808+
&& self.predicate.as_ref().is_some_and(|p| {
809+
collect_columns(p)
810+
.iter()
811+
.any(|c| stats_constants.contains_key(c.name()))
812+
});
813+
literal_columns.extend(stats_constants);
801814

802815
let mut projection = self.projection.clone();
803816
let mut predicate = self.predicate.clone();
@@ -867,6 +880,7 @@ impl ParquetMorselizer {
867880
enable_page_index: self.enable_page_index,
868881
enable_bloom_filter: self.enable_bloom_filter,
869882
enable_row_group_stats_pruning: self.enable_row_group_stats_pruning,
883+
stats_constants_in_predicate,
870884
limit: self.limit,
871885
coerce_int96: self.coerce_int96,
872886
coerce_int96_tz: self.coerce_int96_tz.clone(),
@@ -1128,6 +1142,44 @@ impl FiltersPreparedParquetOpen {
11281142
row_groups.prune_by_range(rg_metadata, range);
11291143
}
11301144

1145+
// If constant-column substitution (`constant_columns_from_stats`)
1146+
// plus simplification collapsed the predicate to a constant that can
1147+
// never be true (`false`, or NULL — filters treat NULL as false),
1148+
// the file's own statistics have proven that no row can match: skip
1149+
// every remaining row group, credited to statistics pruning. Without
1150+
// this, the substitution is strictly counterproductive for such
1151+
// files: it *removes* the column references the pruning predicate
1152+
// would need (`build_pruning_predicates` returns `None` for a bare
1153+
// literal), so a file that was previously pruned via its
1154+
// `null_count` statistics is instead scanned in full.
1155+
//
1156+
// Deliberately narrowed by `stats_constants_in_predicate`: a
1157+
// predicate can also collapse via the missing-column adapter or
1158+
// partition-value folding, and those paths keep their existing
1159+
// behaviour (files pruned by partition values are `FilePruner`'s
1160+
// job). Only a collapse the file's own statistics contributed to
1161+
// is treated as proof here.
1162+
if prepared.enable_row_group_stats_pruning
1163+
&& prepared.stats_constants_in_predicate
1164+
&& prepared
1165+
.predicate
1166+
.as_ref()
1167+
.and_then(|p| p.downcast_ref::<Literal>())
1168+
.is_some_and(|l| {
1169+
l.value().is_null() || l.value() == &ScalarValue::Boolean(Some(false))
1170+
})
1171+
{
1172+
prepared
1173+
.file_metrics
1174+
.row_groups_pruned_statistics
1175+
.add_pruned(row_groups.remaining_row_group_count());
1176+
row_groups.skip_all();
1177+
return Ok(RowGroupsPrunedParquetOpen {
1178+
prepared: self,
1179+
row_groups,
1180+
});
1181+
}
1182+
11311183
// If there is a predicate that can be evaluated against the metadata
11321184
if let Some(predicate) = self.pruning_predicate.as_ref().map(|p| p.as_ref()) {
11331185
if prepared.enable_row_group_stats_pruning {
@@ -3023,6 +3075,100 @@ mod test {
30233075
assert_eq!(num_rows, 0);
30243076
}
30253077

3078+
#[tokio::test]
3079+
async fn test_prune_all_null_column_equality_from_file_statistics() {
3080+
// Regression: a column whose file statistics say every value is
3081+
// NULL cannot satisfy `col = <literal>`, so the file's row groups
3082+
// must be pruned. Constant-column substitution folds the all-NULL
3083+
// column to a NULL literal, collapsing the predicate to a
3084+
// constant; the opener has to recognise that and skip the row
3085+
// groups, rather than lose the pruning because the substituted
3086+
// predicate no longer references any column.
3087+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
3088+
3089+
let batch = record_batch!(
3090+
("a", Int32, vec![Some(1), Some(2), Some(3)]),
3091+
("b", Utf8, vec![None::<&str>, None, None])
3092+
)
3093+
.unwrap();
3094+
let data_size =
3095+
write_parquet(Arc::clone(&store), "file.parquet", batch.clone()).await;
3096+
let file_schema = batch.schema();
3097+
let mut file = PartitionedFile::new(
3098+
"file.parquet".to_string(),
3099+
u64::try_from(data_size).unwrap(),
3100+
);
3101+
// Statistics as a collecting `ListingTable` would supply them: the
3102+
// `b` column's null count equals the row count, i.e. every value is
3103+
// NULL. (`new_unknown` pre-fills one entry per field, so overwrite
3104+
// rather than append.)
3105+
let mut statistics = Statistics::new_unknown(&file_schema);
3106+
statistics.num_rows = Precision::Exact(3);
3107+
statistics.column_statistics[1].null_count = Precision::Exact(3);
3108+
file.statistics = Some(Arc::new(statistics));
3109+
let table_schema = Arc::new(Schema::new(vec![
3110+
Field::new("a", DataType::Int32, false),
3111+
Field::new("b", DataType::Utf8, true),
3112+
]));
3113+
let table_schema_for_opener = TableSchemaBuilder::from(&file_schema).build();
3114+
// The row count alone cannot tell "pruned" from "scanned, then
3115+
// row-filtered" — both yield zero rows here — so assert on the
3116+
// pruning metric, which is the behaviour under test.
3117+
use datafusion_physical_plan::metrics::MetricValue;
3118+
let pruned_row_groups = |metrics: &ExecutionPlanMetricsSet| {
3119+
metrics
3120+
.clone_inner()
3121+
.iter()
3122+
.find_map(|m| match m.value() {
3123+
MetricValue::PruningMetrics {
3124+
name,
3125+
pruning_metrics,
3126+
} if name == "row_groups_pruned_statistics" => {
3127+
Some(pruning_metrics.pruned())
3128+
}
3129+
_ => None,
3130+
})
3131+
.expect("row_groups_pruned_statistics metric is emitted")
3132+
};
3133+
let make_opener = |predicate, metrics: ExecutionPlanMetricsSet| {
3134+
ParquetMorselizerBuilder::new()
3135+
.with_store(Arc::clone(&store))
3136+
.with_table_schema(table_schema_for_opener.clone())
3137+
.with_projection_indices(&[0])
3138+
.with_predicate(predicate)
3139+
.with_row_group_stats_pruning(true)
3140+
.with_metrics(metrics)
3141+
.build()
3142+
};
3143+
3144+
// `b = 'x'` cannot match: every `b` is NULL, so the file's one row
3145+
// group is pruned from statistics rather than read and filtered.
3146+
let metrics = ExecutionPlanMetricsSet::new();
3147+
let expr = col("b").eq(lit("x"));
3148+
let predicate = logical2physical(&expr, &table_schema);
3149+
let opener = make_opener(predicate, metrics.clone());
3150+
let stream = open_file(&opener, file.clone()).await.unwrap();
3151+
let (_, num_rows) = count_batches_and_rows(stream).await;
3152+
assert_eq!(num_rows, 0);
3153+
assert_eq!(
3154+
pruned_row_groups(&metrics),
3155+
1,
3156+
"an all-NULL column cannot satisfy equality: the row group must be pruned"
3157+
);
3158+
3159+
// A predicate the statistics cannot disprove still reads the file,
3160+
// so the skip above is not simply "prune everything".
3161+
let metrics = ExecutionPlanMetricsSet::new();
3162+
let expr = col("a").eq(lit(2));
3163+
let predicate = logical2physical(&expr, &table_schema);
3164+
let opener = make_opener(predicate, metrics.clone());
3165+
let stream = open_file(&opener, file).await.unwrap();
3166+
let (num_batches, num_rows) = count_batches_and_rows(stream).await;
3167+
assert_eq!(num_batches, 1);
3168+
assert_eq!(num_rows, 3);
3169+
assert_eq!(pruned_row_groups(&metrics), 0);
3170+
}
3171+
30263172
#[tokio::test]
30273173
async fn test_prune_on_partition_value_and_data_value() {
30283174
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;

‎datafusion/datasource-parquet/src/row_group_filter.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,17 @@ impl RowGroupAccessPlanFilter {
7272
Self { access_plan }
7373
}
7474

75+
/// Skip every remaining row group: used when the predicate has been
76+
/// proven unsatisfiable for the whole file (for example, when
77+
/// constant-column substitution from file statistics collapsed it to a
78+
/// constant `false`/NULL), so no row group can contain a matching row.
79+
pub fn skip_all(&mut self) {
80+
let indexes: Vec<usize> = self.access_plan.row_group_indexes();
81+
for idx in indexes {
82+
self.access_plan.skip(idx);
83+
}
84+
}
85+
7586
/// Return true if there are no row groups
7687
pub fn is_empty(&self) -> bool {
7788
self.access_plan.is_empty()

0 commit comments

Comments
 (0)