Skip to content

Commit ff7e1e3

Browse files
committed
fix: fall back to Spark for NOT IN over an ordered streamed side
Comet runs a null-aware anti join unswapped in CollectLeft mode, so DataFusion builds on Spark's streamed side. On its hash map path the build batches are concatenated in reverse, and the unmatched rows come out in reverse batch order. Spark has already dropped any sort above the join because the join reports the streamed side's ordering, so sortWithinPartitions returns unsorted partitions and a sort aggregate above the join splits groups. Sorting the native output instead needed about four times the memory pool under fair_unified, so fall back to Spark when a null-aware anti join's streamed side reports an ordering, until the DataFusion fix (apache/datafusion#26226) reaches Comet. This drops the HashJoin output_ordering field and the planner sort from this branch.
1 parent 0946593 commit ff7e1e3

6 files changed

Lines changed: 180 additions & 829 deletions

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,11 @@ Both results are valid under Spark's semantics for these functions.
9494

9595
## Hash Join
9696

97-
When Spark reports a hash join's output as ordered by its streamed side, Comet keeps that order.
98-
When DataFusion does not keep it, for example when it comes from a cached table or the join is a
99-
`NOT IN`, Comet sorts the native join output to restore it.
97+
Spark plans `NOT IN` as a null-aware anti join, and the native hash join builds it on Spark's
98+
streamed side without keeping that side's order. When Spark reports the streamed side as ordered,
99+
for example the output of a window, a sort-merge join, or a cached table sorted within its
100+
partitions, Comet falls back to Spark for the join, so that a sort Spark removed above it on the
101+
strength of that order is not lost.
100102

101103
## Window Functions
102104

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,8 +98,8 @@ omitted from the tables below and may be reconsidered based on demand:
9898

9999
| Operator | Status | Notes |
100100
| ----------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
101-
| `BroadcastHashJoinExec` | ✅ | Falls back to Spark when an outer or `NOT IN` join reports an output ordering whose type or expression Comet cannot sort natively. |
102-
| `ShuffledHashJoinExec` | ✅ | Falls back to Spark when an outer join reports an output ordering whose type or expression Comet cannot sort natively. |
101+
| `BroadcastHashJoinExec` | ✅ | Falls back to Spark for a `NOT IN` (null-aware anti join) whose streamed side reports an ordering. |
102+
| `ShuffledHashJoinExec` | ✅ | |
103103
| `SortMergeJoinExec` | ✅ | Supports `BINARY` join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark. |
104104
| `BroadcastNestedLoopJoinExec` | ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
105105

0 commit comments

Comments
 (0)