Repository navigation
Commit 300a490
contrib(delta) fix: DPP CSAB crash on MERGE target read (standard waitForSubqueries path)
The full-regression re-run surfaced the MergeInto "...isPartitioned: true"
tests still crashing with "CometSubqueryAdaptiveBroadcastExec ... does not
support the execute() code path" -- even though F1/apache#198 fixed the SELECT-join
case.
Root cause: the optimizer rule's DPP rewrite is installed in-place via a
transient `dppFiltersOverride` var (NOT a constructor field), so it is LOST
whenever the plan is COPIED after the rule runs -- which MERGE does (it
re-plans the target read internally). The executing scan then reverts to the
placeholder-bearing `dppFilters`. F1 only guarded the fused-block resolution
path (`ensureSubqueriesResolved`), but a MERGE target read executes the scan
as a native-block ROOT via the STANDARD lifecycle
(`CometNativeColumnarToRowExec` -> child.executeColumnar() ->
SparkPlan.waitForSubqueries()), which executed the placeholder and crashed.
Fix: override `waitForSubqueries` on CometDeltaNativeScanExec too, sharing
`resolveExecutableDppSubqueries()` with `ensureSubqueriesResolved` -- resolve
only executable DPP subqueries, skip adaptive-broadcast placeholders. The
native scan has no non-DPP subqueries, so not delegating to super is safe.
Result: crash-safe in both resolution paths; pruning still applies when the
in-place rewrite survives (SELECT joins), and a MERGE/re-planned scan reads
all partitions (correct -- the MERGE join filters -- just unpruned).
Added a MERGE-into-partitioned guard to CometDeltaDppReproSuite; existing
DPP/row-tracking/feature repros still pass. docs/08-known-limitations A1
updated with the transient-override / waitForSubqueries detail. Verification
of the real MergeIntoSuite tests is via the in-progress full regression
re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 683b74b commit 300a490
3 files changed
Lines changed: 81 additions & 10 deletions
File tree
- contrib/delta
- docs
- src
- main/scala/org/apache/spark/sql/comet
- test/scala/org/apache/comet/contrib/delta
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
76 | 84 | | |
77 | 85 | | |
78 | 86 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
141 | 142 | | |
142 | 143 | | |
143 | 144 | | |
| |||
146 | 147 | | |
147 | 148 | | |
148 | 149 | | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
149 | 168 | | |
150 | 169 | | |
151 | 170 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
96 | 140 | | |
0 commit comments