Repository navigation
Commit bc6429f
committed
fix: strip the Iceberg write's input transition before transition cancellation
Review feedback: the write-boundary strip ran as an arm inside the rule's
`transformUp`, so the generic `ColumnarToRowExec(CometSparkToColumnarExec)`
cancellation reached the write's child first and destroyed the Arrow bridge the
write's FFI input needs. Over a row source that arm drops the
`CometSparkToColumnarExec` outright, leaving the write with a row child; over a
Spark-columnar source it keeps a transition but hands the write Spark
`ColumnarVector`s where the FFI adapter requires `CometVector`s. Both shapes are
reachable through `spark.comet.sparkToColumnar.enabled`, whose
`CometScanWrapper` passes `requiresNativeChildren` and is then unwrapped by
`CometExecRule`, leaving the bridge directly beneath the write.
Move the strip into its own pass ahead of `transformUp` so the write's input
transition is removed before anything else can consume it.
Tests:
- Two plan-level tests in `CometIcebergWriteDetectionSuite` pin the boundary for
both source representations. Both fail without this change.
- `captureWrite` now runs `assertColumnarContract` on every captured plan rather
than only the one test written to exercise it.
- AQE-off CoW UPDATE and MERGE tests cover the two remaining rewrite shapes from
#5689. The partitioned MERGE engages natively (the rebalance exchange makes the
write's child Comet-native even though `MergeRowsExec` is not), so its subtree
mixes a Spark-columnar scan, a row-based operator and Comet operators.1 parent 585ffe8 commit bc6429f
3 files changed
Lines changed: 185 additions & 22 deletions
File tree
- spark/src
- main/scala/org/apache/comet/rules
- test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | 94 | | |
107 | 95 | | |
108 | 96 | | |
| |||
181 | 169 | | |
182 | 170 | | |
183 | 171 | | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
184 | 205 | | |
185 | 206 | | |
186 | 207 | | |
187 | 208 | | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
195 | 215 | | |
196 | 216 | | |
197 | 217 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
743 | 743 | | |
744 | 744 | | |
745 | 745 | | |
746 | | - | |
747 | 746 | | |
748 | 747 | | |
749 | 748 | | |
| |||
805 | 804 | | |
806 | 805 | | |
807 | 806 | | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
808 | 874 | | |
809 | 875 | | |
810 | 876 | | |
| |||
1686 | 1752 | | |
1687 | 1753 | | |
1688 | 1754 | | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
1689 | 1759 | | |
1690 | 1760 | | |
1691 | 1761 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | | - | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
34 | 40 | | |
35 | 41 | | |
36 | 42 | | |
| 43 | + | |
37 | 44 | | |
| 45 | + | |
38 | 46 | | |
39 | 47 | | |
40 | 48 | | |
| |||
881 | 889 | | |
882 | 890 | | |
883 | 891 | | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
884 | 957 | | |
885 | 958 | | |
886 | 959 | | |
| |||
0 commit comments