Repository navigation
Commit 0d5f9b1
authored
fix: re-enable null-equal join dynamic filters with an IS NULL predicate (apache#23106)
## Which issue does this close?
Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the
proper null-equal semantics.
## Rationale for this change
apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins:
the build-side bounds and membership predicates evaluate to NULL for a
probe-side NULL key, so they prune rows that should null-match a
build-side NULL. Its description already named the better fix, "generate
a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti
joins; this re-enables the null-equal case the same way.
## What changes are included in this PR?
- Revert the null-equal `return false` in
`allow_join_dynamic_filter_pushdown`.
- Generalize the shared probe-NULL helper to cover both null-aware
(single-key) and null-equal (multi-key) joins: OR `key IS NULL` for
every nullable probe key. A NOT NULL key never widens the filter, so an
all-NOT-NULL join keeps full selectivity.
## Are these changes tested?
Yes. apache#22965's SLT now asserts the filter is back on the probe with the
result unchanged, plus a multi-key null-equal case. The reject unit test
flips to assert pushdown is allowed, and `preserve_probe_nulls` unit
tests cover both the mixed nullable/NOT NULL case (only the nullable key
widens) and the all-NOT-NULL case (no widening).
## Are there any user-facing changes?
Null-equal joins regain dynamic filter pushdown, so they prune the probe
scan again while returning correct results.1 parent 0062b4b commit 0d5f9b1
4 files changed
Lines changed: 426 additions & 52 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
863 | 863 | | |
864 | 864 | | |
865 | 865 | | |
866 | | - | |
867 | | - | |
868 | | - | |
869 | | - | |
870 | | - | |
871 | | - | |
872 | | - | |
873 | | - | |
874 | 866 | | |
875 | 867 | | |
876 | 868 | | |
| |||
1424 | 1416 | | |
1425 | 1417 | | |
1426 | 1418 | | |
| 1419 | + | |
1427 | 1420 | | |
1428 | 1421 | | |
1429 | 1422 | | |
| |||
6956 | 6949 | | |
6957 | 6950 | | |
6958 | 6951 | | |
6959 | | - | |
| 6952 | + | |
6960 | 6953 | | |
6961 | 6954 | | |
6962 | 6955 | | |
| |||
6979 | 6972 | | |
6980 | 6973 | | |
6981 | 6974 | | |
6982 | | - | |
| 6975 | + | |
| 6976 | + | |
| 6977 | + | |
6983 | 6978 | | |
6984 | 6979 | | |
6985 | 6980 | | |
| |||
0 commit comments