Repository navigation
Commit 8e22f85
feat: enable null-aware anti-join support via DataFusion 53
DataFusion 53.0.0 introduced null-aware anti-join support in HashJoinExec
via a new boolean parameter to try_new(). This enables Comet to offload
Spark's NOT IN subquery pattern (BroadcastHashJoinExec with
isNullAwareAntiJoin=true) to the native layer.
Changes:
- Add null_aware_anti_join field (tag 6) to HashJoin proto message
- Pass join.null_aware_anti_join to HashJoinExec::try_new() in planner.rs
- Allow BuildRight+LeftAnti joins through when isNullAwareAntiJoin=true
on BroadcastHashJoinExec; regular BuildRight+LeftAnti on
ShuffledHashJoinExec remains blocked (issue apache#457)
- Set NullAwareAntiJoin field in the proto builder from operators.scala
Null-aware semantics: if the build side contains ANY null, all left rows
are suppressed regardless of match status (SQL NOT IN with nullable column).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>1 parent 964e578 commit 8e22f85
3 files changed
Lines changed: 16 additions & 8 deletions
File tree
- native
- core/src/execution
- proto/src/proto
- spark/src/main/scala/org/apache/spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1699 | 1699 | | |
1700 | 1700 | | |
1701 | 1701 | | |
1702 | | - | |
1703 | | - | |
1704 | | - | |
1705 | | - | |
1706 | | - | |
1707 | | - | |
| 1702 | + | |
1708 | 1703 | | |
1709 | 1704 | | |
1710 | 1705 | | |
| |||
4038 | 4033 | | |
4039 | 4034 | | |
4040 | 4035 | | |
| 4036 | + | |
4041 | 4037 | | |
4042 | 4038 | | |
4043 | 4039 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
341 | 341 | | |
342 | 342 | | |
343 | 343 | | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
344 | 347 | | |
345 | 348 | | |
346 | 349 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1710 | 1710 | | |
1711 | 1711 | | |
1712 | 1712 | | |
1713 | | - | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
1714 | 1722 | | |
1715 | | - | |
| 1723 | + | |
1716 | 1724 | | |
1717 | 1725 | | |
1718 | 1726 | | |
| |||
1754 | 1762 | | |
1755 | 1763 | | |
1756 | 1764 | | |
| 1765 | + | |
1757 | 1766 | | |
1758 | 1767 | | |
1759 | 1768 | | |
| |||
0 commit comments