Repository navigation
Commit cb07053
committed
fix: address review findings on the cache scan and serializer
Five issues from review of 86a91ea.
Emit zero-column batches for an empty-output cache scan. The scan
widened an empty projection to a placeholder column so the native plan
would have a non-empty scan schema, but that claim was untrue: a
zero-column scan plans and runs. Meanwhile the emitted batches
disagreed with the declared output, so a join over a cached relation
read the wrong column and silently returned a wrong sum rather than
failing. Removing the widening fixes the join and makes count(*) decode
nothing at all.
Serialize each column with the dictionary provider it was decoded with.
Columns decoded from separate streams have independent dictionary ID
namespaces, so re-encoding a decoded batch with the first column's
provider could not resolve the rest.
Release readers opened before a later column fails to decode. The
holder is published to the task-completion listener only after its
constructor returns, so a partial failure leaked off-heap for the life
of the executor.
Build the cached RDD in doExecuteColumnar rather than at planning time.
CachedRDDBuilder.cachedColumnBuffers executes the cached plan, so
planning a query over an adaptively cached relation ran a job and
finalized that plan during EXPLAIN.
Report large-offset Arrow vectors as not directly writable.
isArrowBacked accepted any CometVector, including one wrapping
LargeVarCharVector or LargeVarBinaryVector, which getFieldVector
rejects; such batches now take the conversion path instead of failing
at cache materialization.1 parent 86a91ea commit cb07053
5 files changed
Lines changed: 272 additions & 83 deletions
File tree
- spark/src
- main/scala/org/apache/spark/sql/comet
- execution/arrow
- util
- test/scala/org/apache
- comet/exec
- spark/sql/comet/util
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
29 | 29 | | |
30 | | - | |
31 | 30 | | |
32 | 31 | | |
33 | 32 | | |
| |||
49 | 48 | | |
50 | 49 | | |
51 | 50 | | |
52 | | - | |
| 51 | + | |
53 | 52 | | |
54 | 53 | | |
55 | 54 | | |
| |||
58 | 57 | | |
59 | 58 | | |
60 | 59 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
83 | 90 | | |
84 | 91 | | |
85 | 92 | | |
| |||
107 | 114 | | |
108 | 115 | | |
109 | 116 | | |
110 | | - | |
| 117 | + | |
111 | 118 | | |
112 | 119 | | |
113 | 120 | | |
| |||
127 | 134 | | |
128 | 135 | | |
129 | 136 | | |
130 | | - | |
| 137 | + | |
131 | 138 | | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
| 139 | + | |
156 | 140 | | |
157 | 141 | | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | 142 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
366 | 367 | | |
367 | 368 | | |
368 | 369 | | |
369 | | - | |
370 | | - | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
371 | 398 | | |
372 | 399 | | |
373 | 400 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
280 | 280 | | |
281 | 281 | | |
282 | 282 | | |
283 | | - | |
284 | | - | |
285 | 283 | | |
286 | 284 | | |
287 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
288 | 289 | | |
289 | 290 | | |
290 | 291 | | |
| |||
440 | 441 | | |
441 | 442 | | |
442 | 443 | | |
443 | | - | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
444 | 455 | | |
445 | 456 | | |
446 | 457 | | |
447 | | - | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
448 | 473 | | |
449 | | - | |
| 474 | + | |
450 | 475 | | |
451 | 476 | | |
452 | 477 | | |
453 | | - | |
454 | | - | |
455 | | - | |
456 | | - | |
457 | | - | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
458 | 481 | | |
459 | | - | |
| 482 | + | |
460 | 483 | | |
461 | 484 | | |
462 | 485 | | |
463 | 486 | | |
464 | 487 | | |
465 | | - | |
| 488 | + | |
466 | 489 | | |
467 | 490 | | |
468 | 491 | | |
469 | 492 | | |
470 | 493 | | |
471 | 494 | | |
| 495 | + | |
472 | 496 | | |
473 | 497 | | |
474 | 498 | | |
| |||
482 | 506 | | |
483 | 507 | | |
484 | 508 | | |
485 | | - | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
486 | 520 | | |
487 | 521 | | |
488 | 522 | | |
489 | | - | |
490 | | - | |
491 | | - | |
492 | | - | |
493 | | - | |
494 | | - | |
495 | | - | |
496 | | - | |
497 | | - | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
498 | 527 | | |
499 | 528 | | |
500 | 529 | | |
0 commit comments