Repository navigation
Commit 5f50da7
fix: re-enable null-equal join dynamic filters with an IS NULL predicate (apache#23106)
## Which issue does this close?
Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the
proper null-equal semantics.
## Rationale for this change
apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins:
the build-side bounds and membership predicates evaluate to NULL for a
probe-side NULL key, so they prune rows that should null-match a
build-side NULL. Its description already named the better fix, "generate
a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti
joins; this re-enables the null-equal case the same way.
## What changes are included in this PR?
- Revert the null-equal `return false` in
`allow_join_dynamic_filter_pushdown`.
- Generalize the shared probe-NULL helper to cover both null-aware
(single-key) and null-equal (multi-key) joins: OR `key IS NULL` for
every nullable probe key. A NOT NULL key never widens the filter, so an
all-NOT-NULL join keeps full selectivity.
## Are these changes tested?
Yes. apache#22965's SLT now asserts the filter is back on the probe with the
result unchanged, plus a multi-key null-equal case. The reject unit test
flips to assert pushdown is allowed, and `preserve_probe_nulls` unit
tests cover both the mixed nullable/NOT NULL case (only the nullable key
widens) and the all-NOT-NULL case (no widening).
## Are there any user-facing changes?
Null-equal joins regain dynamic filter pushdown, so they prune the probe
scan again while returning correct results.
(cherry picked from commit 0d5f9b1)
Conflicts resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/physical-plan/src/joins/hash_join/exec.rs: renamed the null-equal
test as upstream does; its upstream neighbour
(test_swap_inputs_rejects_dynamic_filter) is not on 54.1.
- datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: test
scaffolding only. Skipped the upstream-only helpers (including the
reported() helper this commit edits); added null_equality to the 54.1
test helper; imported Column in the test module, which upstream's module
already does and the new tests need.
- datafusion/physical-plan/src/joins/hash_join/stream.rs: upstream's test
module there is built on a BuildReportHandle API that 54.1 does not have,
so its one-line change is not applied. 54.1 reports a dropped partition
through report_canceled_partition (CanceledUnknown), which this change
already treats as possibly holding a NULL key.
- datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: the five
new EXPLAIN / EXPLAIN ANALYZE expectations follow the 54.1 display: no
dynamic_rg_pruning field, and the scan_efficiency_ratio byte counts of the
files 54.1's parquet writer produces. Predicates, row counts and pruning
metrics are as upstream wrote them.1 parent 3919f28 commit 5f50da7
4 files changed
Lines changed: 421 additions & 51 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
905 | 905 | | |
906 | 906 | | |
907 | 907 | | |
908 | | - | |
909 | | - | |
910 | | - | |
911 | | - | |
912 | | - | |
913 | | - | |
914 | | - | |
915 | | - | |
916 | 908 | | |
917 | 909 | | |
918 | 910 | | |
| |||
1414 | 1406 | | |
1415 | 1407 | | |
1416 | 1408 | | |
| 1409 | + | |
1417 | 1410 | | |
1418 | 1411 | | |
1419 | 1412 | | |
| |||
6465 | 6458 | | |
6466 | 6459 | | |
6467 | 6460 | | |
6468 | | - | |
| 6461 | + | |
6469 | 6462 | | |
6470 | 6463 | | |
6471 | 6464 | | |
| |||
6488 | 6481 | | |
6489 | 6482 | | |
6490 | 6483 | | |
6491 | | - | |
| 6484 | + | |
| 6485 | + | |
| 6486 | + | |
6492 | 6487 | | |
6493 | 6488 | | |
6494 | 6489 | | |
| |||
0 commit comments