You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3ff0939
Browse filesBrowse the repository at this point in the historyBrowse files
fix: sort the input of a Spark window over a doubted order
Spark plans no sort below a window whose child reports the order it
needs, and its window trusts that order. Over a Comet operator that may
not keep it, such as a native nested loop join (#6898), a window that
stays in Spark grouped rows out of order. CometExecRule now sorts that
input again. With that, a window with a collated key falls back in every
case, and so does a window with ORDER BY but no PARTITION BY over a
doubted order, since the native window checks no order there. The
compatibility guide lists collated keys among the window's fallbacks.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/operators.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -219,6 +219,9 @@ incorrect result. When any single window expression in a `WindowExec` falls back
219
219
-`lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)).
220
220
- A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric.
221
221
- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize.
222
+
- Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation (e.g.
223
+
`UTF8_LCASE`). The native window finds partitions and peers by comparing keys byte by byte, which splits values
224
+
that the collation ties.
222
225
223
226
A `RANGE` frame with an explicit offset on a `DATE``ORDER BY` runs natively, but over a date outside Spark's
224
227
supported range of 0001-01-01 to 9999-12-31, such as one produced by `date_add` with a very large day count, the
0 commit comments