Repository navigation
Commit 3919f28
fix: Disable join dynamic filters for null-equal joins (apache#22965)
## Which issue does this PR close?
- Closes #apache#22964
## Rationale for this change
We presently allow dynamic filter pushdown to be applied to null-equal
hash joins. This might result in pushing a predicate down into the
probe-side plan, where the predicate will not be evaluated with the
null-equal semantics that are required.
Longer-term, we might consider supporting this case with the correct
semantics (e.g., generate a predicate with `OR IS NULL ...`), but for
now disabling pushdown for null-equal joins seems much more practical.
## What changes are included in this PR?
* Disable hash join dynamic filter pushdown for null-equal joins
* Add SLT test with end-to-end repro
* Add unit test
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
(cherry picked from commit baa497d)
Conflicts resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/physical-plan/src/joins/hash_join/exec.rs: this branch already
carries 5975a2d (apache#23173), which upstream merged later and which adds its
own gate and tests at the same places. Placed this commit's gate and test
ahead of those, the order they have upstream.
- datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: same; the
null-equal block goes ahead of the apache#23173 block, as upstream has it.1 parent 3e58817 commit 3919f28
2 files changed
Lines changed: 79 additions & 0 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
905 | 905 | | |
906 | 906 | | |
907 | 907 | | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
908 | 916 | | |
909 | 917 | | |
910 | 918 | | |
| |||
6456 | 6464 | | |
6457 | 6465 | | |
6458 | 6466 | | |
| 6467 | + | |
| 6468 | + | |
| 6469 | + | |
| 6470 | + | |
| 6471 | + | |
| 6472 | + | |
| 6473 | + | |
| 6474 | + | |
| 6475 | + | |
| 6476 | + | |
| 6477 | + | |
| 6478 | + | |
| 6479 | + | |
| 6480 | + | |
| 6481 | + | |
| 6482 | + | |
| 6483 | + | |
| 6484 | + | |
| 6485 | + | |
| 6486 | + | |
| 6487 | + | |
| 6488 | + | |
| 6489 | + | |
| 6490 | + | |
| 6491 | + | |
| 6492 | + | |
| 6493 | + | |
| 6494 | + | |
| 6495 | + | |
6459 | 6496 | | |
6460 | 6497 | | |
6461 | 6498 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1023 | 1023 | | |
1024 | 1024 | | |
1025 | 1025 | | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
1026 | 1068 | | |
1027 | 1069 | | |
1028 | 1070 | | |
| |||
0 commit comments