Repository navigation
Commit ff7e1e3
committed
fix: fall back to Spark for NOT IN over an ordered streamed side
Comet runs a null-aware anti join unswapped in CollectLeft mode, so
DataFusion builds on Spark's streamed side. On its hash map path the
build batches are concatenated in reverse, and the unmatched rows come
out in reverse batch order. Spark has already dropped any sort above
the join because the join reports the streamed side's ordering, so
sortWithinPartitions returns unsorted partitions and a sort aggregate
above the join splits groups.
Sorting the native output instead needed about four times the memory
pool under fair_unified, so fall back to Spark when a null-aware anti
join's streamed side reports an ordering, until the DataFusion fix
(apache/datafusion#26226) reaches Comet. This drops the HashJoin
output_ordering field and the planner sort from this branch.1 parent 0946593 commit ff7e1e3
6 files changed
Lines changed: 180 additions & 829 deletions
File tree
- docs/source/user-guide/latest
- compatibility
- native
- core/src/execution
- proto/src/proto
- spark/src
- main/scala/org/apache/spark/sql/comet
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
99 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
102 | | - | |
| 101 | + | |
| 102 | + | |
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| |||
0 commit comments