Skip to content

Commit 5a8ca02

Browse files
committed
perf(parquet-scan): skip dynamic filters in pushdown=false post-scan filter
The pushdown=false path in the parquet opener split the whole predicate into 'post_scan_conjuncts' — a per-batch FilterExec-equivalent — which included any dynamic filter conjuncts (HashJoin bounds, TopK threshold, aggregate dynamic filter). For join-heavy TPC-H / TPC-DS this dominates cost: HashJoin's Partitioned- mode dynamic filter is a 'CASE hash(col) % N WHEN pid THEN bounds ELSE lit(false) END' — per-row hash + modulo + CASE branch — and it prunes almost nothing on high-match-rate joins where the downstream hash lookup would eliminate the same rows anyway. Local TPC-H SF1 Q9 profile showed 1.1% self-time in 'expressions::case::PartialResultIndex::merge_n' and 1.3% in 'arrow_select::filter::filter_native' on the PR, both at 0% on main — driving Q9 from 40ms → 80ms (2.09x on CI, 1.79x locally). This commit filters DynamicFilterPhysicalExpr-containing conjuncts out of 'post_scan_conjuncts'. Effects: - RowGroupPruner (added by #22450) still sees the full predicate via prepared.predicate, so RG-level dynamic pruning continues to fire on bounds/threshold updates. - pushdown_filters=true path unchanged — dynamic filters still go through the arrow-rs RowFilter. - Downstream operator does the exact equivalent: HashJoin's hash lookup filters rows the bounds would have filtered; TopK's sort heap filters rows the threshold would have filtered. No wrong results. Local TPC-H SF1 (release-nonlto, 3 iters): - baseline (HEAD~2, pre-#22384) avg: 27.65 ms - PR + this fix avg: 26.24 ms (net 5% ahead of baseline) - Q9 individually: 34.54 → 34.33 ms (matches baseline, was 80.74 before) Also regenerates push_down_filter_parquet.slt for the membership-off default (from the earlier 'split membership from bounds' commit).
1 parent 82ea18c commit 5a8ca02

2 files changed

Lines changed: 25 additions & 14 deletions

File tree

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1407,10 +1407,20 @@ impl RowGroupsPrunedParquetOpen {
14071407
// through to post-scan.
14081408
row_filter_generator.rejected_conjuncts().to_vec()
14091409
} else if let Some(predicate) = prepared.predicate.as_ref() {
1410-
// Pushdown disabled: the whole predicate runs post-scan (in-scan
1411-
// equivalent of a FilterExec).
1410+
// Pushdown disabled: static conjuncts run post-scan (in-scan
1411+
// equivalent of a FilterExec). Dynamic filters are *dropped* here
1412+
// — they stay in `prepared.predicate` for the RowGroupPruner
1413+
// below (#22450 dynamic RG-level pruning), so their contribution
1414+
// survives. Applying them per-batch here is redundant with the
1415+
// downstream operator that already exact-matches on the same
1416+
// key (HashJoin hash lookup / TopK sort heap) and, for
1417+
// Partitioned HashJoin's `CASE hash(col) % N WHEN pid THEN
1418+
// bounds ELSE lit(false) END`, expensive.
14121419
datafusion_physical_expr::split_conjunction(predicate)
14131420
.into_iter()
1421+
.filter(|conjunct| {
1422+
!DynamicFilterTracking::classify(conjunct).contains_dynamic_filter()
1423+
})
14141424
.cloned()
14151425
.collect()
14161426
} else {

0 commit comments

Comments
 (0)