Skip to content

Commit 4885533

Browse files
mdashtikosiew
authored andcommitted
fix: re-enable null-equal join dynamic filters with an IS NULL predicate (apache#23106)
## Which issue does this close? Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the proper null-equal semantics. ## Rationale for this change apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti joins; this re-enables the null-equal case the same way. ## What changes are included in this PR? - Revert the null-equal `return false` in `allow_join_dynamic_filter_pushdown`. - Generalize the shared probe-NULL helper to cover both null-aware (single-key) and null-equal (multi-key) joins: OR `key IS NULL` for every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity. ## Are these changes tested? Yes. apache#22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and `preserve_probe_nulls` unit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening). ## Are there any user-facing changes? Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results.
1 parent b6e9efc commit 4885533

4 files changed

Lines changed: 426 additions & 52 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -863,14 +863,6 @@ impl HashJoinExec {
863863
return false;
864864
}
865865

866-
// Bounds and membership filters derived from the build side do not
867-
// account for null-equal matching: a probe-side NULL key evaluates
868-
// such predicates to NULL and would be pruned, even though it can
869-
// match a build-side NULL when nulls compare equal.
870-
if self.null_equality == NullEquality::NullEqualsNull {
871-
return false;
872-
}
873-
874866
// A null-aware anti join emits a build-side NULL only when the probe
875867
// is truly empty. The pushed filter can empty the probe by pruning
876868
// every row, which would surface that NULL wrongly. A NOT NULL build
@@ -1424,6 +1416,7 @@ impl ExecutionPlan for HashJoinExec {
14241416
filter,
14251417
on_right,
14261418
repartition_random_state,
1419+
self.null_equality,
14271420
self.null_aware,
14281421
))
14291422
})))
@@ -6956,7 +6949,7 @@ mod tests {
69566949
}
69576950

69586951
#[test]
6959-
fn test_dynamic_filter_pushdown_rejects_null_equal_join() -> Result<()> {
6952+
fn test_dynamic_filter_pushdown_allowed_for_null_equal_join() -> Result<()> {
69606953
let (_, _, on) = build_schema_and_on()?;
69616954
let left = build_table(("a1", &vec![1]), ("b1", &vec![1]), ("c1", &vec![1]));
69626955
let right = build_table(("a2", &vec![1]), ("b1", &vec![1]), ("c2", &vec![1]));
@@ -6979,7 +6972,9 @@ mod tests {
69796972
false,
69806973
)?;
69816974

6982-
assert!(!join.allow_join_dynamic_filter_pushdown(session_config.options()));
6975+
// Null-equal joins keep dynamic filter pushdown: the pushed predicate carries an
6976+
// `IS NULL` disjunct so a probe-side NULL still reaches the join.
6977+
assert!(join.allow_join_dynamic_filter_pushdown(session_config.options()));
69836978

69846979
Ok(())
69856980
}

0 commit comments

Comments
 (0)