Repository navigation
Commit 66c63bd
authored
fix: keep the values of sliced batches when coalescing a broadcast (#6900)
* fix: keep the values of sliced batches when coalescing a broadcast
A slice of a native string, binary, list or map array reaches the JVM with offsets that start
past zero, with the rows before the slice still at the front of its data. Arrow Java writes
those bytes into the IPC stream, and the VectorSchemaRootAppender that coalesces broadcast
batches assumes offsets start at zero, so it prepended them to the first value of each slice.
serializeBatches now writes such a batch through a slice of its root, which rebases the
offsets to zero and leaves the earlier bytes out.
* Pass Long arguments explicitly in the UtilsSuite slice helper
The strict-warnings build rejects the implicit Int to Long widening in the `ArrowFieldNode`
and `ArrowBuf.slice` calls, so convert the arguments explicitly.
* Document that sliced variable-width arrays reach the JVM with offsets past zero
`zero_offsets` handles `ArrayData::offset`, but a sliced string, binary, list or map array still
reaches the JVM with offsets that start past zero and the rows before the slice at the front of its
data. Say so in the "Array Offsets" section, and that `serializeBatches` rebases broadcast batches
because `VectorSchemaRootAppender` needs offsets that start at zero.
* Close the rebased slice when writing a broadcast batch fails
`splitAndTransfer` takes its own references on the buffers the slice shares with the batch, so a
write that failed, as it does when a cancelled broadcast interrupts the task, left them pinned after
the batch closed. Close the slice in a `finally`, and test that the slice's allocator drains to zero
after an interrupted write.1 parent a230ae5 commit 66c63bd
4 files changed
Lines changed: 181 additions & 6 deletions
File tree
- docs/source/contributor-guide
- spark/src
- main/scala/org/apache/spark/sql/comet/util
- test/scala/org/apache
- comet/exec
- spark/sql/comet/util
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
241 | 241 | | |
242 | 242 | | |
243 | 243 | | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
244 | 254 | | |
245 | 255 | | |
246 | 256 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
275 | 286 | | |
276 | 287 | | |
277 | 288 | | |
| |||
281 | 292 | | |
282 | 293 | | |
283 | 294 | | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
284 | 312 | | |
285 | 313 | | |
286 | 314 | | |
| |||
380 | 408 | | |
381 | 409 | | |
382 | 410 | | |
| 411 | + | |
383 | 412 | | |
384 | 413 | | |
385 | 414 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
1716 | 1718 | | |
1717 | 1719 | | |
1718 | 1720 | | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
1719 | 1749 | | |
1720 | 1750 | | |
1721 | 1751 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
22 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
26 | 34 | | |
| 35 | + | |
27 | 36 | | |
28 | 37 | | |
29 | 38 | | |
| |||
58 | 67 | | |
59 | 68 | | |
60 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
61 | 128 | | |
62 | 129 | | |
63 | 130 | | |
| |||
197 | 264 | | |
198 | 265 | | |
199 | 266 | | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
200 | 306 | | |
0 commit comments