You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 545735c
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/native_shuffle.md
+4-7Lines changed: 4 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,12 +83,9 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
83
83
Spark's `mapsort` normalization makes physical entry order irrelevant. A collated string at any
84
84
depth still disqualifies the key. The config defaults to `false` pending measurement of the
85
85
nested hashing paths, so by default a complex hash key falls back to JVM shuffle.
86
-
- A hash key that is or contains a decimal wider than 18 digits stays on JVM shuffle when the
87
-
shuffle's stage starts at a typed `Dataset` conversion
88
-
(`spark.comet.convert.typedDataset.enabled`) and the shuffle has more than one partition.
89
-
Native shuffle hashes such decimals differently from Spark
90
-
([#5994](https://github.com/apache/datafusion-comet/issues/5994)). Without the conversion
91
-
this shuffle would have used JVM shuffle, and a join partner may still use it.
86
+
Decimal leaves with precision greater than 18 also disqualify a hash key when there is more
87
+
than one output partition: native decimal hashing differs from Spark's. A one-partition hash
88
+
exchange stays native because the writer serializes it as `SinglePartition` without hashing.
0 commit comments