Repository navigation
Commit 16780c0
feat(delta): DML rewrites synthesize via kernel-enumerate + path-filter (apache#76 Stage B option a)
MERGE / UPDATE / DELETE rewrites (TahoeBatchFileIndex) now get kernel's per-file
transforms -- so they synthesize in-worker like regular reads, no DeltaSyntheticColumnsExec.
The driver enumerates the pinned snapshot via kernel (with the synthesize read schema:
row_index/row_id metadata columns) and FILTERS the result to the touched AddFile path set
(touched files are a subset of the snapshot). Matching is exact string equality on the
resolved file_path -- both sides resolve identically (table_root + rel, or pass-through for
scheme'd paths).
Safety: if ANY touched file isn't found in kernel's enumeration -- or enumeration throws --
decline to the legacy buildTaskListFromAddFiles + DeltaSyntheticColumnsExec path (never wrong
files). Validated: the only declines in the full suite are 2 OptimizeJob-internal edge cases
(CM-id materialised row_commit_version, which kernel genuinely can't express; and a file-not-
found race during concurrent OPTIMIZE) -- both correctly fall back.
- DeltaReflection.isDmlRewriteFileIndex: TahoeBatchFileIndex only (CDC family excluded -- it
references files outside the snapshot, kept on the legacy path).
- Empty predicate for the DML enumeration: the AddFile set is the authoritative selection;
kernel stats pruning could drop a touched file and force a needless decline.
Full contrib Delta suite: 148 passed, 0 failed (synthesize on for regular + DML reads).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>1 parent ffdda0f commit 16780c0
2 files changed
Lines changed: 91 additions & 13 deletions
File tree
- contrib/delta/src/main/scala/org/apache/comet/contrib/delta
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
645 | 645 | | |
646 | 646 | | |
647 | 647 | | |
648 | | - | |
649 | | - | |
650 | | - | |
651 | | - | |
652 | | - | |
653 | | - | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
654 | 656 | | |
655 | 657 | | |
656 | 658 | | |
| |||
719 | 721 | | |
720 | 722 | | |
721 | 723 | | |
722 | | - | |
723 | | - | |
724 | | - | |
725 | | - | |
726 | | - | |
727 | | - | |
728 | | - | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
729 | 796 | | |
730 | 797 | | |
731 | 798 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
492 | 492 | | |
493 | 493 | | |
494 | 494 | | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
495 | 506 | | |
496 | 507 | | |
497 | 508 | | |
| |||
0 commit comments