You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 939a05a
Browse filesBrowse the repository at this point in the historyBrowse files
docs: qualify where native hash joins keep the streamed order
The operator compatibility guide now says the order is kept where Comet
can show that it holds, and points to #6922 for the caches it still
leaves undeclared. The contributor guide says what a leaf that reports
an ordering has to do natively.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/operators.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -94,9 +94,9 @@ Both results are valid under Spark's semantics for these functions.
94
94
## Hash Join Output Order
95
95
96
96
Spark reports the sort order of a hash join's streamed side as the join's own, and drops a sort
97
-
above the join that this order satisfies. Comet's native hash joins keep that order, including a
98
-
left or right outer join over a streamed side that was sorted outside the native plan, such as a
99
-
sorted relation read from the [in-memory cache](#in-memory-cache).
97
+
above the join that this order satisfies. Comet's native hash joins keep that order where Comet can
98
+
show that it holds. That includes a left or right outer join over a streamed side that was sorted
99
+
outside the native plan, such as a sorted relation read from the [in-memory cache](#in-memory-cache).
100
100
101
101
Comet passes the order to the native join as sort keys, and an outer join needs at least the first
102
102
key to be one that Comet can express natively. Declaring any leading part of the order keeps the
@@ -109,7 +109,8 @@ Where the order Spark reports might not hold, Comet declares nothing and an oute
109
109
its unmatched rows out of order. The output of a `NOT IN` subquery or of a nested loop join
110
110
([#6898](https://github.com/apache/datafusion-comet/issues/6898)) is one such case, and so is a
111
111
cache or checkpoint that one of them wrote. Comet also declares nothing when it cannot follow the
112
-
order back to its source, as with an RDD checkpoint.
112
+
order back to its source, as with an RDD checkpoint, and over some caches written by an outer hash
113
+
join whose order does hold ([#6922](https://github.com/apache/datafusion-comet/issues/6922)).
0 commit comments