You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3f41e77
Browse filesBrowse the repository at this point in the historyBrowse files
fix: prune row groups when statistics collapse the predicate to a constant
Closesapache#24769.
`constant_columns_from_stats` substitutes columns that file statistics
prove constant — including the all-NULL case, which folds to a NULL
literal. When the substituted predicate then simplifies to a bare
constant (`NULL`, or `false`), `build_pruning_predicates` returns
`None` because there are no column references left, and the opener
falls through with no pruning at all.
That makes the substitution strictly counterproductive for these
files. A predicate like `col = <literal>` over a column whose
statistics say `null_count == row_count` is provably unsatisfiable,
and before the substitution existed it *was* pruned via the
`col_null_count != row_count` conjunct of the pruning predicate. The
regression bisects to apache#22969, which enabled `collect_statistics` by
default and so began feeding the substitution on paths that previously
had no file statistics.
Recognise the collapsed-to-constant case explicitly: if the predicate
is a `false`/NULL literal and the original predicate referenced a
column that `constant_columns_from_stats` proved constant, skip every
remaining row group and credit the skip to statistics pruning.
Results are unchanged either way — a filter drops NULL and false rows
alike — but the scan work is not.
The `stats_constants_in_predicate` guard deliberately narrows the
skip to collapses that file statistics contributed to. A predicate
can also collapse via the missing-column adapter (schema evolution)
or partition-value folding; both keep their existing behaviour —
partition-driven collapse is already `FilePruner`'s job, and widening
the skip to missing columns changes the raw scan output that nine
`evolved_schema` / `test_pushdown_with_missing_*` tests assert.
The regression test asserts the `row_groups_pruned_statistics` metric
rather than the row count, since a row-count assertion cannot tell
"pruned" from "scanned, then row-filtered": both yield zero rows.
0 commit comments