You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1ec0e66
Browse filesBrowse the repository at this point in the historyBrowse files
fix: keep wide-decimal shuffles above a typed Dataset conversion on JVM shuffle
A typed Dataset conversion moves the shuffle above it from Comet's columnar
shuffle, which partitions with Spark's hash, to native shuffle. Native
shuffle hashes decimals wider than 18 digits differently from Spark
(#5994), so a join on such keys with an input that stays on columnar
shuffle, for example one with an array<int> column the conversion
declines, put matching keys in different partitions and returned 9 rows
instead of 100. Such a shuffle now stays on columnar shuffle unless it has
one partition, the same rule #6005 proposes for every native shuffle.
Also declare the benchmark's row count as a Long, which the strict
Spark 3.5 compile requires.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/native_shuffle.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -71,6 +71,12 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
71
71
Spark's `mapsort` normalization makes physical entry order irrelevant. A collated string at any
72
72
depth still disqualifies the key. The config defaults to `false` pending measurement of the
73
73
nested hashing paths, so by default a complex hash key falls back to JVM shuffle.
74
+
- A hash key that is or contains a decimal wider than 18 digits stays on JVM shuffle when the
75
+
shuffle's stage starts at a typed `Dataset` conversion
76
+
(`spark.comet.convert.typedDataset.enabled`) and the shuffle has more than one partition.
77
+
Native shuffle hashes such decimals differently from Spark
78
+
([#5994](https://github.com/apache/datafusion-comet/issues/5994)). Without the conversion
79
+
this shuffle would have used JVM shuffle, and a join partner may still use it.
0 commit comments