Repository navigation
Commit a83ecb3
feat(delta): keep input_file_name() native via one-task-per-partition
Closes the input_file_name() / input_file_block_* fallback (apache#75 design A).
Previously CometScanRule declined the entire Delta scan whenever the
plan referenced these expressions, falling back to vanilla Spark+Delta.
CometExecRDD.setInputFileForDeltaScan already sets InputFileBlockHolder,
but only with the FIRST task's path -- correct only when 1 task per
partition. We now:
- Detect input_file_name() / input_file_block_* in the plan and tag
the relation's options with comet.delta.needsInputFileName=true.
- In CometDeltaNativeScan.convert: skip byte-range splitting in
splitTasks when the tag is set.
- In CometDeltaNativeScanExec.packTasks: emit one task per group when
the constructor's oneTaskPerPartition flag is set.
With 1 file per partition, setInputFileForDeltaScan attributes every
row correctly. input_file_name() stays a JVM-evaluated expression (no
native serde) in any Project above the Comet exec frontier.
The propagation goes through relation.options rather than a ThreadLocal
because CometScanRule (sets the flag) and CometExecRule (calls convert
/ createExec) run as separate plan-transform passes; a ThreadLocal
would be cleared before convert/createExec ran.
Verified:
- DeletionVectorsSuite 293/293 pass (was passing via fallback; now
via native one-task-per-partition path).
- DeleteSQLSuite + UpdateSQLSuite 133/133 pass (these are the
canonical input_file_name() consumers via findTouchedFiles).
- Today's run produced zero "Native Delta scan is not compatible
with input_file_name" fallback messages.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 8588afe commit a83ecb3
3 files changed
Lines changed: 70 additions & 41 deletions
File tree
- spark/src/main/scala/org/apache
- comet
- rules
- serde/operator
- spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
349 | 349 | | |
350 | 350 | | |
351 | 351 | | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
383 | 378 | | |
384 | 379 | | |
385 | 380 | | |
| |||
1478 | 1473 | | |
1479 | 1474 | | |
1480 | 1475 | | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
1481 | 1484 | | |
1482 | 1485 | | |
1483 | 1486 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
64 | 78 | | |
65 | 79 | | |
66 | 80 | | |
| |||
638 | 652 | | |
639 | 653 | | |
640 | 654 | | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
645 | 661 | | |
646 | 662 | | |
647 | 663 | | |
| |||
778 | 794 | | |
779 | 795 | | |
780 | 796 | | |
| 797 | + | |
781 | 798 | | |
782 | 799 | | |
783 | 800 | | |
| |||
791 | 808 | | |
792 | 809 | | |
793 | 810 | | |
794 | | - | |
| 811 | + | |
| 812 | + | |
795 | 813 | | |
796 | 814 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
64 | 71 | | |
65 | 72 | | |
66 | 73 | | |
| |||
141 | 148 | | |
142 | 149 | | |
143 | 150 | | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
148 | 155 | | |
149 | 156 | | |
| 157 | + | |
150 | 158 | | |
151 | 159 | | |
152 | 160 | | |
| |||
0 commit comments