Skip to content

Commit 939a05a

Browse files
committed
docs: qualify where native hash joins keep the streamed order
The operator compatibility guide now says the order is kept where Comet can show that it holds, and points to #6922 for the caches it still leaves undeclared. The contributor guide says what a leaf that reports an ordering has to do natively.
1 parent 204440a commit 939a05a

2 files changed

Lines changed: 11 additions & 4 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_operator.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,6 +374,12 @@ declares the order through an operator only when the operator says it keeps it.
374374
cannot keep the order natively must not report it, or has to be added to `mayReportFalseOrder` in
375375
`operators.scala`, which makes the hash join serde send no order over it.
376376

377+
A leaf that reports an ordering is where the order starts, so the planner has to know that the
378+
native operator produces its rows in that order. It knows that of an input from the JVM and of a
379+
scan of files, which reads the files of each partition one after the other, and the native range
380+
declares its order itself through `SortedInputExec`. A new leaf that reports an ordering must
381+
declare it natively the same way.
382+
377383
#### Plan Identity and Exchange Reuse
378384

379385
Spark's `ReuseExchangeAndSubquery` identifies equivalent plans through canonicalization and may

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,9 @@ Both results are valid under Spark's semantics for these functions.
9494
## Hash Join Output Order
9595

9696
Spark reports the sort order of a hash join's streamed side as the join's own, and drops a sort
97-
above the join that this order satisfies. Comet's native hash joins keep that order, including a
98-
left or right outer join over a streamed side that was sorted outside the native plan, such as a
99-
sorted relation read from the [in-memory cache](#in-memory-cache).
97+
above the join that this order satisfies. Comet's native hash joins keep that order where Comet can
98+
show that it holds. That includes a left or right outer join over a streamed side that was sorted
99+
outside the native plan, such as a sorted relation read from the [in-memory cache](#in-memory-cache).
100100

101101
Comet passes the order to the native join as sort keys, and an outer join needs at least the first
102102
key to be one that Comet can express natively. Declaring any leading part of the order keeps the
@@ -109,7 +109,8 @@ Where the order Spark reports might not hold, Comet declares nothing and an oute
109109
its unmatched rows out of order. The output of a `NOT IN` subquery or of a nested loop join
110110
([#6898](https://github.com/apache/datafusion-comet/issues/6898)) is one such case, and so is a
111111
cache or checkpoint that one of them wrote. Comet also declares nothing when it cannot follow the
112-
order back to its source, as with an RDD checkpoint.
112+
order back to its source, as with an RDD checkpoint, and over some caches written by an outer hash
113+
join whose order does hold ([#6922](https://github.com/apache/datafusion-comet/issues/6922)).
113114

114115
## Window Functions
115116

0 commit comments

Comments
 (0)