Skip to content

Commit 2ce4952

Browse files
committed
feat(parquet): decline pushdown when projection has few non-filter columns
RowFilter has a fixed per-row machinery overhead that only pays for itself when the wide-column decode it lets us skip is meaningful. When the projection is narrow and the filter columns already cover most of it (typical for GROUP BY col-style queries), the overhead dominates and pushdown regresses. Add a plan-time heuristic in ParquetSource::try_pushdown_filters that declines pushdown when the projection contains fewer than 3 columns not referenced by the filter. When declined, the filter stays above the scan in a FilterExec (correctness preserved) and the predicate is still injected into ParquetSource for stats / bloom / page-index pruning. Kept intentionally simple — a plan-time column-count check with no tuning knob. Complexity budget is reserved for the runtime adaptive-placement work tracked in apache#22883. Part of apache#3463.
1 parent 1e77af8 commit 2ce4952

1 file changed

Lines changed: 38 additions & 1 deletion

File tree

‎datafusion/datasource-parquet/src/source.rs‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ use datafusion_datasource::file_scan_config::FileScanConfig;
4646
use datafusion_functions::core::file_row_index::FileRowIndexFunc;
4747
use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking};
4848
use datafusion_physical_expr::projection::ProjectionExprs;
49+
use datafusion_physical_expr::utils::collect_columns;
4950
use datafusion_physical_expr::{EquivalenceProperties, conjunction};
5051
use datafusion_physical_expr_adapter::DefaultPhysicalExprAdapterFactory;
5152
use datafusion_physical_expr_adapter::rewrite::{
@@ -827,7 +828,43 @@ impl FileSource for ParquetSource {
827828
// because even if scan pushdown is disabled we can still use the filters for stats pruning.
828829
let config_pushdown_enabled = config.execution.parquet.pushdown_filters;
829830
let table_pushdown_enabled = self.pushdown_filters();
830-
let pushdown_filters = table_pushdown_enabled || config_pushdown_enabled;
831+
let mut pushdown_filters = table_pushdown_enabled || config_pushdown_enabled;
832+
833+
// Simple heuristic (issue #3463, discussion on PR #23369):
834+
// RowFilter has a fixed per-row machinery overhead that only pays
835+
// for itself when the wide-column decode it lets us skip is
836+
// meaningful. When the projection is narrow and the filter columns
837+
// already cover most of it (typical for `GROUP BY col`-style
838+
// queries such as ClickBench Q10/Q11/Q40), the overhead dominates
839+
// and pushdown regresses. Decline pushdown for those scans; keep it
840+
// for wider projections where the fast-path pays.
841+
//
842+
// Kept intentionally simple: a plan-time column-count check with
843+
// no tuning knob. A future adaptive-placement pass (#22883) can
844+
// supersede this with a runtime cost model.
845+
//
846+
// When declined: the filter stays above the scan in a FilterExec
847+
// (correctness preserved) and the predicate is still injected
848+
// into `ParquetSource` for stats / bloom / page-index pruning.
849+
const PUSHDOWN_MIN_NON_FILTER_COLS: usize = 3;
850+
if pushdown_filters {
851+
let filter_col_indices: std::collections::HashSet<usize> = filters
852+
.iter()
853+
.flat_map(|f| collect_columns(f).into_iter().map(|c| c.index()))
854+
.collect();
855+
let non_filter_projected = self
856+
.projection
857+
.as_ref()
858+
.iter()
859+
.flat_map(|pe| collect_columns(&pe.expr))
860+
.map(|c| c.index())
861+
.filter(|idx| !filter_col_indices.contains(idx))
862+
.collect::<std::collections::HashSet<_>>()
863+
.len();
864+
if non_filter_projected < PUSHDOWN_MIN_NON_FILTER_COLS {
865+
pushdown_filters = false;
866+
}
867+
}
831868

832869
let mut source = self.clone();
833870
let filters: Vec<PushedDownPredicate> = filters

0 commit comments

Comments
 (0)