Repository navigation
Commit a20a4a5
committed
fix: keep a shuffle whose key is computed from a string on the JVM columnar shuffle
The string guard looked only at the type of each hash key, so a key that
is a number computed from a string, such as hash(s), passed it. Native
shuffle evaluates such a key after the import into native has replaced
invalid UTF-8 in the string, so it can still send a row to a different
partition than Spark's partitioner does, and a join whose other input
stays on the JVM columnar shuffle loses the matches. The guard now looks
for a string anywhere in the key expression. A wide decimal still matters
only as the type of the key itself, because Comet's hash of a wide
decimal already falls back to Spark.
The string join test also joins on the hash of the strings, and uses two
keys instead of four. Both shuffles write the strings with invalid UTF-8
replaced, so the keys stay apart only while each key expression sends
them to different partitions, and two of the four hashes share one.1 parent d597480 commit a20a4a5
4 files changed
Lines changed: 28 additions & 20 deletions
File tree
- docs/source/user-guide/latest/tuning
- spark/src
- main/scala/org/apache
- comet
- spark/sql/comet/execution/shuffle
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
74 | | - | |
75 | | - | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
247 | 247 | | |
248 | 248 | | |
249 | 249 | | |
250 | | - | |
251 | | - | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
252 | 253 | | |
253 | 254 | | |
254 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
584 | 584 | | |
585 | 585 | | |
586 | 586 | | |
587 | | - | |
588 | | - | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
589 | 590 | | |
590 | 591 | | |
591 | 592 | | |
592 | 593 | | |
593 | 594 | | |
594 | 595 | | |
595 | | - | |
596 | | - | |
597 | | - | |
598 | | - | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
599 | 600 | | |
600 | 601 | | |
601 | 602 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
224 | | - | |
225 | | - | |
226 | | - | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
227 | 229 | | |
228 | 230 | | |
229 | 231 | | |
230 | 232 | | |
231 | 233 | | |
232 | | - | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
233 | 237 | | |
234 | 238 | | |
235 | 239 | | |
236 | 240 | | |
237 | 241 | | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
242 | 248 | | |
243 | 249 | | |
244 | 250 | | |
| |||
0 commit comments