Repository navigation
Commit d8d7ce4
committed
fix: leave calendar intervals out of the shuffle input conversion
Arrow holds the time part of an interval in nanoseconds, so
IntervalMonthDayNanoWriter overflows on a calendar interval with more
microseconds than that can hold, which Spark accepts. The JVM columnar
shuffle leaves calendar intervals to Spark's shuffle, and the conversion
now does too.1 parent 16a1f73 commit d8d7ce4
4 files changed
Lines changed: 26 additions & 9 deletions
File tree
- docs/source/user-guide/latest/tuning
- spark/src
- main/scala/org/apache
- comet
- spark/sql/comet/execution/shuffle
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
213 | | - | |
214 | | - | |
215 | | - | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
569 | 569 | | |
570 | 570 | | |
571 | 571 | | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
572 | 576 | | |
573 | 577 | | |
574 | 578 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
260 | 261 | | |
261 | 262 | | |
262 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
263 | 276 | | |
264 | 277 | | |
265 | 278 | | |
| |||
0 commit comments