Repository navigation
Commit 3194f1a
committed
refactor: decide offsets as lengths once per shuffle block writer
ShuffleBlockWriter::try_new_with_offsets_as_lengths replaces the builder. The writer now decides
when it is built: it checks once, on an empty batch, that arrow-ipc writes the buffers the
mapped roles describe, keeps plain offsets if not, and sets the codec-tag flag in its header.
That removes the per-block flag plumbing through every codec arm and the header rewrite.
The writer and reader share one buffer-range helper and one rewrite loop. The reader rebuilds
offsets before freezing the body. The two Scala tests share a cometShuffleMetric helper.1 parent c317d54 commit 3194f1a
8 files changed
Lines changed: 298 additions & 269 deletions
File tree
- native/shuffle/src
- spark_unsafe
- writers
- spark/src/test/scala/org/apache
- comet/exec
- spark/sql
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
323 | 325 | | |
324 | 326 | | |
325 | 327 | | |
326 | | - | |
| 328 | + | |
327 | 329 | | |
328 | 330 | | |
329 | 331 | | |
| |||
347 | 349 | | |
348 | 350 | | |
349 | 351 | | |
350 | | - | |
| 352 | + | |
351 | 353 | | |
| 354 | + | |
352 | 355 | | |
353 | 356 | | |
354 | 357 | | |
| |||
378 | 381 | | |
379 | 382 | | |
380 | 383 | | |
381 | | - | |
382 | | - | |
| 384 | + | |
383 | 385 | | |
384 | | - | |
| 386 | + | |
385 | 387 | | |
386 | 388 | | |
387 | | - | |
| 389 | + | |
388 | 390 | | |
389 | 391 | | |
390 | 392 | | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
| 393 | + | |
409 | 394 | | |
410 | 395 | | |
411 | 396 | | |
412 | | - | |
| 397 | + | |
| 398 | + | |
413 | 399 | | |
414 | 400 | | |
415 | 401 | | |
| |||
421 | 407 | | |
422 | 408 | | |
423 | 409 | | |
424 | | - | |
| 410 | + | |
425 | 411 | | |
426 | 412 | | |
427 | 413 | | |
| |||
485 | 471 | | |
486 | 472 | | |
487 | 473 | | |
488 | | - | |
| 474 | + | |
489 | 475 | | |
490 | 476 | | |
491 | | - | |
| 477 | + | |
492 | 478 | | |
493 | 479 | | |
494 | 480 | | |
| |||
541 | 527 | | |
542 | 528 | | |
543 | 529 | | |
544 | | - | |
| 530 | + | |
545 | 531 | | |
546 | | - | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
547 | 536 | | |
548 | 537 | | |
549 | 538 | | |
| |||
578 | 567 | | |
579 | 568 | | |
580 | 569 | | |
581 | | - | |
582 | | - | |
| 570 | + | |
| 571 | + | |
583 | 572 | | |
584 | 573 | | |
585 | | - | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
586 | 577 | | |
587 | 578 | | |
588 | 579 | | |
| |||
650 | 641 | | |
651 | 642 | | |
652 | 643 | | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
653 | 661 | | |
654 | 662 | | |
655 | 663 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
109 | 124 | | |
110 | 125 | | |
111 | 126 | | |
112 | 127 | | |
113 | 128 | | |
114 | 129 | | |
115 | 130 | | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
122 | 138 | | |
123 | 139 | | |
124 | 140 | | |
| |||
127 | 143 | | |
128 | 144 | | |
129 | 145 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
136 | 153 | | |
137 | 154 | | |
138 | | - | |
| 155 | + | |
139 | 156 | | |
140 | 157 | | |
141 | 158 | | |
142 | | - | |
| 159 | + | |
| 160 | + | |
143 | 161 | | |
144 | 162 | | |
145 | 163 | | |
| |||
148 | 166 | | |
149 | 167 | | |
150 | 168 | | |
151 | | - | |
| 169 | + | |
152 | 170 | | |
153 | | - | |
154 | | - | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
155 | 175 | | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | 176 | | |
167 | 177 | | |
168 | 178 | | |
169 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
170 | 194 | | |
171 | 195 | | |
172 | 196 | | |
173 | 197 | | |
174 | 198 | | |
175 | 199 | | |
176 | | - | |
| 200 | + | |
177 | 201 | | |
178 | | - | |
| 202 | + | |
| 203 | + | |
179 | 204 | | |
180 | 205 | | |
181 | 206 | | |
182 | | - | |
183 | | - | |
| 207 | + | |
| 208 | + | |
184 | 209 | | |
| 210 | + | |
185 | 211 | | |
186 | 212 | | |
187 | 213 | | |
| |||
195 | 221 | | |
196 | 222 | | |
197 | 223 | | |
198 | | - | |
| 224 | + | |
| 225 | + | |
199 | 226 | | |
200 | 227 | | |
201 | 228 | | |
202 | | - | |
| 229 | + | |
203 | 230 | | |
204 | 231 | | |
205 | 232 | | |
| |||
209 | 236 | | |
210 | 237 | | |
211 | 238 | | |
212 | | - | |
| 239 | + | |
213 | 240 | | |
214 | 241 | | |
215 | 242 | | |
| |||
218 | 245 | | |
219 | 246 | | |
220 | 247 | | |
221 | | - | |
222 | | - | |
| 248 | + | |
| 249 | + | |
223 | 250 | | |
224 | 251 | | |
225 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
131 | | - | |
132 | | - | |
| 131 | + | |
133 | 132 | | |
134 | 133 | | |
135 | 134 | | |
| |||
202 | 201 | | |
203 | 202 | | |
204 | 203 | | |
205 | | - | |
206 | | - | |
| 204 | + | |
| 205 | + | |
207 | 206 | | |
208 | 207 | | |
209 | 208 | | |
| |||
336 | 335 | | |
337 | 336 | | |
338 | 337 | | |
339 | | - | |
340 | | - | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
341 | 343 | | |
342 | 344 | | |
343 | 345 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1375 | 1375 | | |
1376 | 1376 | | |
1377 | 1377 | | |
1378 | | - | |
1379 | 1378 | | |
1380 | 1379 | | |
1381 | 1380 | | |
| |||
1452 | 1451 | | |
1453 | 1452 | | |
1454 | 1453 | | |
1455 | | - | |
1456 | | - | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
1457 | 1459 | | |
1458 | 1460 | | |
1459 | 1461 | | |
| |||
0 commit comments