Skip to content

Commit cae5fe6

Browse files
adriangbclaude
andcommitted
hash join: wrap pushed-down dynamic filter in OptionalFilterPhysicalExpr
The adaptive parquet scheduler in `selectivity::demote_or_drop` only drops a filter (`FilterState::Dropped`) when its expression is wrapped in `OptionalFilterPhysicalExpr`; otherwise the filter can only ever be demoted to post-scan, where it then runs on every batch forever. Hash-join dynamic filters had no wrapper, so for join-heavy queries with low-selectivity build sides (most of TPC-H) they cycled between row-level and post-scan and never got dropped. On SF1 with `pushdown_filters=true` this dominated runtime — Q9 was 121.5 ms vs 68.5 ms on main (1.77x), Q17 134 ms → 134 ms, etc. Disabling join dynamic filter pushdown entirely brought Q9 to 49 ms, isolating the cost to "running ineffective dynamic filters that never get dropped". Wrap the `Arc<DynamicFilterPhysicalExpr>` in `OptionalFilterPhysicalExpr` at the `gather_filters_for_pushdown` handoff to mark it as drop-eligible, and unwrap it in `handle_child_pushdown_result` before `Arc::downcast::<DynamicFilterPhysicalExpr>` so the join keeps the bare filter for `.update()` / `.mark_complete()`. Join correctness does not depend on this filter — it's a probe-side scan-time hint — so dropping it is always safe. TPC-H SF1, pushdown=true (mean of 8 iters, drop iter 0), branch / main: Q3 0.74x Q5 0.73x Q7 0.74x Q8 0.88x Q9 0.80x Q14 0.69x Q17 0.55x Q20 0.89x All 963 hash-join unit tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent ada0472 commit cae5fe6

1 file changed

Lines changed: 23 additions & 4 deletions

File tree

  • datafusion/physical-plan/src/joins/hash_join

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,9 @@ use datafusion_physical_expr::projection::{ProjectionRef, combine_projections};
9090
use datafusion_physical_expr::{PhysicalExpr, PhysicalExprRef};
9191

9292
use datafusion_common::hash_utils::RandomState;
93-
use datafusion_physical_expr_common::physical_expr::fmt_sql;
93+
use datafusion_physical_expr_common::physical_expr::{
94+
OptionalFilterPhysicalExpr, fmt_sql,
95+
};
9496
use datafusion_physical_expr_common::utils::evaluate_expressions_to_arrays;
9597
use futures::TryStreamExt;
9698
use parking_lot::Mutex;
@@ -1644,9 +1646,16 @@ impl ExecutionPlan for HashJoinExec {
16441646
if phase == FilterPushdownPhase::Post
16451647
&& self.allow_join_dynamic_filter_pushdown(config)
16461648
{
1647-
// Add actual dynamic filter to right side (probe side)
1649+
// Add actual dynamic filter to right side (probe side). Wrap
1650+
// it in `OptionalFilterPhysicalExpr` so the parquet adaptive
1651+
// scheduler can drop it entirely when measured selectivity is
1652+
// too low to justify the per-batch evaluation cost. Join
1653+
// correctness does not depend on this filter — it's purely a
1654+
// scan-time pruning optimization — so dropping it is safe.
16481655
let dynamic_filter = Self::create_dynamic_filter(&self.on);
1649-
right_child = right_child.with_self_filter(dynamic_filter);
1656+
let pushdown_filter: Arc<dyn PhysicalExpr> =
1657+
Arc::new(OptionalFilterPhysicalExpr::new(dynamic_filter));
1658+
right_child = right_child.with_self_filter(pushdown_filter);
16501659
}
16511660

16521661
Ok(FilterDescription::new()
@@ -1667,7 +1676,17 @@ impl ExecutionPlan for HashJoinExec {
16671676
if let Some(filter) = right_child_self_filters.first() {
16681677
// Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
16691678
// "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
1670-
let predicate = Arc::clone(&filter.predicate);
1679+
// We wrap the dynamic filter in `OptionalFilterPhysicalExpr` at
1680+
// `gather_filters_for_pushdown`; unwrap it here so we can recover
1681+
// the inner `Arc<DynamicFilterPhysicalExpr>` the join holds onto
1682+
// for `.update()` / `.mark_complete()`.
1683+
let predicate: Arc<dyn PhysicalExpr> = if let Some(opt) =
1684+
filter.predicate.downcast_ref::<OptionalFilterPhysicalExpr>()
1685+
{
1686+
opt.inner()
1687+
} else {
1688+
Arc::clone(&filter.predicate)
1689+
};
16711690
if let Ok(dynamic_filter) =
16721691
Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
16731692
{

0 commit comments

Comments
 (0)