Skip to content

Commit 5f50da7

Browse files
mdashtilukekim
authored andcommitted
fix: re-enable null-equal join dynamic filters with an IS NULL predicate (apache#23106)
## Which issue does this close? Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the proper null-equal semantics. ## Rationale for this change apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti joins; this re-enables the null-equal case the same way. ## What changes are included in this PR? - Revert the null-equal `return false` in `allow_join_dynamic_filter_pushdown`. - Generalize the shared probe-NULL helper to cover both null-aware (single-key) and null-equal (multi-key) joins: OR `key IS NULL` for every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity. ## Are these changes tested? Yes. apache#22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and `preserve_probe_nulls` unit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening). ## Are there any user-facing changes? Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results. (cherry picked from commit 0d5f9b1) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/exec.rs: renamed the null-equal test as upstream does; its upstream neighbour (test_swap_inputs_rejects_dynamic_filter) is not on 54.1. - datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: test scaffolding only. Skipped the upstream-only helpers (including the reported() helper this commit edits); added null_equality to the 54.1 test helper; imported Column in the test module, which upstream's module already does and the new tests need. - datafusion/physical-plan/src/joins/hash_join/stream.rs: upstream's test module there is built on a BuildReportHandle API that 54.1 does not have, so its one-line change is not applied. 54.1 reports a dropped partition through report_canceled_partition (CanceledUnknown), which this change already treats as possibly holding a NULL key. - datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: the five new EXPLAIN / EXPLAIN ANALYZE expectations follow the 54.1 display: no dynamic_rg_pruning field, and the scan_efficiency_ratio byte counts of the files 54.1's parquet writer produces. Predicates, row counts and pruning metrics are as upstream wrote them.
1 parent 3919f28 commit 5f50da7

4 files changed

Lines changed: 421 additions & 51 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -905,14 +905,6 @@ impl<A: CollectLeftAccumulator + 'static> HashJoinExec<A> {
905905
return false;
906906
}
907907

908-
// Bounds and membership filters derived from the build side do not
909-
// account for null-equal matching: a probe-side NULL key evaluates
910-
// such predicates to NULL and would be pruned, even though it can
911-
// match a build-side NULL when nulls compare equal.
912-
if self.null_equality == NullEquality::NullEqualsNull {
913-
return false;
914-
}
915-
916908
// A null-aware anti join emits a build-side NULL only when the probe
917909
// is truly empty. The pushed filter can empty the probe by pruning
918910
// every row, which would surface that NULL wrongly. A NOT NULL build
@@ -1414,6 +1406,7 @@ impl<A: CollectLeftAccumulator + 'static> ExecutionPlan for HashJoinExec<A> {
14141406
filter,
14151407
on_right,
14161408
repartition_random_state,
1409+
self.null_equality,
14171410
self.null_aware,
14181411
))
14191412
})))
@@ -6465,7 +6458,7 @@ mod tests {
64656458
}
64666459

64676460
#[test]
6468-
fn test_dynamic_filter_pushdown_rejects_null_equal_join() -> Result<()> {
6461+
fn test_dynamic_filter_pushdown_allowed_for_null_equal_join() -> Result<()> {
64696462
let (_, _, on) = build_schema_and_on()?;
64706463
let left = build_table(("a1", &vec![1]), ("b1", &vec![1]), ("c1", &vec![1]));
64716464
let right = build_table(("a2", &vec![1]), ("b1", &vec![1]), ("c2", &vec![1]));
@@ -6488,7 +6481,9 @@ mod tests {
64886481
false,
64896482
)?;
64906483

6491-
assert!(!join.allow_join_dynamic_filter_pushdown(session_config.options()));
6484+
// Null-equal joins keep dynamic filter pushdown: the pushed predicate carries an
6485+
// `IS NULL` disjunct so a probe-side NULL still reaches the join.
6486+
assert!(join.allow_join_dynamic_filter_pushdown(session_config.options()));
64926487

64936488
Ok(())
64946489
}

0 commit comments

Comments
 (0)