Repository navigation
Commit 887948a
fix: prune row groups when file statistics collapse the predicate to a constant (apache#24770)
> [!NOTE]
> This fix was investigated and written with AI assistance (Claude
Code), posted with the account owner's review and consent.
## Which issue does this PR close?
- Closes apache#24769.
## Rationale for this change
A query like `SELECT ... WHERE col = 'x'` over a Parquet file whose
statistics say `col` is entirely NULL cannot match any row, and
DataFusion 54 pruned such row groups from the footer without reading
them. On 55 the same query reads them.
The cause is an interaction rather than a bug in pruning itself.
`constant_columns_from_stats` substitutes columns that file statistics
prove constant, and its all-NULL branch folds such a column to a NULL
literal. Once substituted, the predicate simplifies to a bare constant —
`NULL` here — at which point `build_pruning_predicates` returns `None`
(there are no column references left to build a pruning predicate over),
and `prune_row_groups` falls through with no pruning at all.
So for exactly the files where the statistics carry the *most*
information, the substitution is strictly counterproductive: before it,
the pruning predicate's own `col_null_count != row_count` conjunct
proved the row group empty and skipped it.
This surfaced as a regression when bisecting a real workload (a Parquet
log store where a `body` column is NULL for the vast majority of rows)
from 54 → 55. The bisect lands on apache#22969, which removed
`ListingOptions::collect_stat` in favour of the session's
`execution.collect_statistics` — default `true`. That change is correct
in itself; it simply began feeding per-file statistics to the
substitution on paths that previously had none, exposing the gap.
Setting `execution.collect_statistics = false` restores pruning on 55,
which is a useful confirmation but obviously not a fix.
Results were never wrong — a filter drops NULL and false rows alike —
but the scan work is real: row groups that used to be skipped from the
footer are now decoded in full.
## What changes are included in this PR?
`prune_row_groups` now recognises the collapsed-to-constant case: if the
(post-substitution, post-simplification) predicate is a `false` or NULL
literal, every remaining row group is skipped and the skip is credited
to `row_groups_pruned_statistics`. A small
`RowGroupAccessPlanFilter::skip_all` helper is added alongside the
existing `prune_by_*` methods.
The check is deliberately narrow — a downcast to `Literal` plus a
NULL/`false` value test — so it cannot affect predicates that still
reference columns; those take the existing path unchanged.
## Are these changes tested?
Yes: `test_prune_all_null_column_equality_from_file_statistics` in
`opener/mod.rs`, modelled on the neighbouring
`test_prune_on_partition_values_and_file_statistics`. It fails on
current `main` (3 rows scanned, 0 pruned) and passes with this change.
One note on how it asserts, in case it saves a reviewer time: it checks
the `row_groups_pruned_statistics` metric rather than the returned row
count. A row-count assertion cannot distinguish "pruned" from "scanned,
then row-filtered" — both give zero rows — and an earlier draft of this
test passed against the unfixed code for exactly that reason.
Verified `cargo test -p datafusion-datasource-parquet --lib opener::` —
47 passed. The 16 pre-existing `bloom_filter::tests` failures on this
crate are unrelated and reproduce identically on an untouched checkout.
## Are there any user-facing changes?
No API or result changes. Queries that were already correct stay
correct; affected scans read less data. Plans may show more row groups
pruned, and `row_groups_pruned_statistics` increases correspondingly.
---------
Co-authored-by: Jens Holdgaard Pedersen <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent d5eb246 commit 887948a
2 files changed
Lines changed: 485 additions & 3 deletions
0 commit comments