Skip to content

Commit 300a490

Browse files
schenksjclaude
andcommitted
contrib(delta) fix: DPP CSAB crash on MERGE target read (standard waitForSubqueries path)
The full-regression re-run surfaced the MergeInto "...isPartitioned: true" tests still crashing with "CometSubqueryAdaptiveBroadcastExec ... does not support the execute() code path" -- even though F1/apache#198 fixed the SELECT-join case. Root cause: the optimizer rule's DPP rewrite is installed in-place via a transient `dppFiltersOverride` var (NOT a constructor field), so it is LOST whenever the plan is COPIED after the rule runs -- which MERGE does (it re-plans the target read internally). The executing scan then reverts to the placeholder-bearing `dppFilters`. F1 only guarded the fused-block resolution path (`ensureSubqueriesResolved`), but a MERGE target read executes the scan as a native-block ROOT via the STANDARD lifecycle (`CometNativeColumnarToRowExec` -> child.executeColumnar() -> SparkPlan.waitForSubqueries()), which executed the placeholder and crashed. Fix: override `waitForSubqueries` on CometDeltaNativeScanExec too, sharing `resolveExecutableDppSubqueries()` with `ensureSubqueriesResolved` -- resolve only executable DPP subqueries, skip adaptive-broadcast placeholders. The native scan has no non-DPP subqueries, so not delegating to super is safe. Result: crash-safe in both resolution paths; pruning still applies when the in-place rewrite survives (SELECT joins), and a MERGE/re-planned scan reads all partitions (correct -- the MERGE join filters -- just unpruned). Added a MERGE-into-partitioned guard to CometDeltaDppReproSuite; existing DPP/row-tracking/feature repros still pass. docs/08-known-limitations A1 updated with the transient-override / waitForSubqueries detail. Verification of the real MergeIntoSuite tests is via the in-progress full regression re-run. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 683b74b commit 300a490

3 files changed

Lines changed: 81 additions & 10 deletions

File tree

‎contrib/delta/docs/08-known-limitations.md‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,18 @@ The Delta own-suite regression is run via
6969
recomputed (not memoized) so a parent block's `findAllPlanData` sees the
7070
pruned task lists after the broadcast is materialized.
7171
- **Guard:** `CometDeltaDppReproSuite` asserts native engagement, correct
72-
results, AND that the fact scan reads ~120 of 2000 rows (real pruning).
73-
- **Residual:** if the rule somehow doesn't run, `ensureSubqueriesResolved` /
74-
`applyDppFilters` skip the unexecutable placeholder and read all partitions
75-
(correct, unpruned) instead of crashing.
72+
results, AND that the fact scan reads ~120 of 2000 rows (real pruning) for a
73+
SELECT broadcast join; plus a MERGE-into-partitioned guard.
74+
- **Residual (MERGE / re-planned plans):** the in-place rewrite lives in a
75+
transient `dppFiltersOverride` (not a constructor field), so it is LOST
76+
whenever the plan is copied after the optimizer rule runs (e.g. MERGE
77+
re-plans internally). In that case `effectiveDppFilters` reverts to the
78+
placeholder and the scan reads ALL partitions (correct, unpruned) rather
79+
than pruning. To stay crash-safe, BOTH subquery-resolution paths skip the
80+
unexecutable placeholder: the fused-block path (`ensureSubqueriesResolved`,
81+
via `findAllPlanData`) and the standard-lifecycle path (`waitForSubqueries`,
82+
used when the scan is a native-block root, e.g. a MERGE target read under
83+
`CometNativeColumnarToRowExec`). `applyDppFilters` enforces the same skip.
7684
- **Commits:** `64cd878a` (no-crash) → in-place rewrite + stable-group pruning.
7785

7886
### A2. Cloud credential plumbing gaps

‎contrib/delta/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala‎

Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -132,12 +132,13 @@ case class CometDeltaNativeScanExec(
132132
super.doPrepare()
133133
}
134134

135-
// Comet's native-scan subquery lifecycle (see CometLeafExec). The default
136-
// `prepare(); waitForSubqueries()` would execute EVERY collected subquery; if
137-
// an unrewritten placeholder slipped through it would crash. Resolve only the
138-
// executable DPP subqueries; skip placeholders (read-all fallback).
139-
override def ensureSubqueriesResolved(): Unit = {
140-
prepare()
135+
// Resolve only the DPP subqueries we can execute; skip adaptive-broadcast
136+
// PLACEHOLDERS (CometSubqueryAdaptiveBroadcastExec / SubqueryAdaptiveBroadcastExec),
137+
// which throw from doExecute(). When the optimizer rule's in-place rewrite reached
138+
// this instance, `effectiveDppFilters` holds the executable form and pruning applies;
139+
// otherwise the placeholder is skipped and the scan reads all partitions (correct, the
140+
// surrounding Filter/join still prunes). `applyDppFilters` enforces the same skip.
141+
private def resolveExecutableDppSubqueries(): Unit = {
141142
effectiveDppFilters.foreach {
142143
case DynamicPruningExpression(inSub: InSubqueryExec)
143144
if !isUnexecutableDpp(inSub.plan) && inSub.values().isEmpty =>
@@ -146,6 +147,24 @@ case class CometDeltaNativeScanExec(
146147
}
147148
}
148149

150+
// Comet's native-scan subquery lifecycle (see CometLeafExec): used when this scan is
151+
// fused inside a parent native block (findAllPlanData path).
152+
override def ensureSubqueriesResolved(): Unit = {
153+
prepare()
154+
resolveExecutableDppSubqueries()
155+
}
156+
157+
// Standard Spark lifecycle path (executeColumnar -> executeQuery -> waitForSubqueries),
158+
// used when this scan is a native-block ROOT executed directly (e.g. the child of a
159+
// CometNativeColumnarToRowExec, as in a MERGE target read). The default would execute
160+
// EVERY collected subquery -- including an unconverted CometSubqueryAdaptiveBroadcastExec
161+
// (the in-place DPP rewrite is lost whenever the plan is copied after the rule runs,
162+
// since `dppFiltersOverride` is not a constructor field) -- and crash. Override to
163+
// resolve only the executable ones, mirroring `ensureSubqueriesResolved`. The native
164+
// scan has no subqueries other than its DPP partition filters, so not delegating to
165+
// `super` is safe.
166+
override def waitForSubqueries(): Unit = resolveExecutableDppSubqueries()
167+
149168
@transient private lazy val commonBytes: Array[Byte] = {
150169
// The typed DeltaScan variant of OpStruct carries the common block directly.
151170
nativeOp.getDeltaScan.getCommon.toByteArray

‎contrib/delta/src/test/scala/org/apache/comet/contrib/delta/CometDeltaDppReproSuite.scala‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
package org.apache.comet.contrib.delta
2121

22+
import org.apache.spark.sql.Row
2223
import org.apache.spark.sql.comet.CometDeltaNativeScanExec
2324

2425
// Repro for regression family F1 (DPP):
@@ -93,4 +94,47 @@ class CometDeltaDppReproSuite extends CometDeltaTestBase {
9394
}
9495
}
9596
}
97+
98+
// Mirrors MergeIntoSuiteBase "basic case - local predicates - ... isPartitioned:
99+
// true" -- the actual failing Delta own-suite test. MERGE into a table partitioned
100+
// by the join key, with cross-join enabled (Delta's local-predicate path) so AQE+DPP
101+
// inserts a dynamic-partition-pruning InSubquery over the target read. The subquery
102+
// resolution here goes through the standard `child.executeColumnar() ->
103+
// waitForSubqueries()` lifecycle on the native block (not the scan's own
104+
// `ensureSubqueriesResolved`), so the CometSubqueryAdaptiveBroadcastExec must be
105+
// CONVERTED, not merely skipped.
106+
test("MERGE into partitioned table with local predicate (DPP)") {
107+
assume(deltaSparkAvailable, "delta-spark not on the test classpath; skipping")
108+
withSQLConf(
109+
"spark.sql.optimizer.dynamicPartitionPruning.enabled" -> "true",
110+
"spark.sql.optimizer.dynamicPartitionPruning.useStats" -> "false",
111+
"spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly" -> "true",
112+
"spark.sql.exchange.reuse" -> "true",
113+
"spark.sql.crossJoin.enabled" -> "true",
114+
"spark.sql.autoBroadcastJoinThreshold" -> "10485760") {
115+
withDeltaTable("merge_dpp") { tablePath =>
116+
val ss = spark
117+
import ss.implicits._
118+
Seq(("1", "2", "noop"), ("1", "4", "noop"), ("3", "2", "noop"), ("4", "4", "noop"))
119+
.toDF("key2", "value", "op")
120+
.repartition(2)
121+
.write.format("delta").partitionBy("key2").save(tablePath)
122+
Seq(("1", "8"), ("0", "3")).toDF("key1", "value")
123+
.createOrReplaceTempView("merge_dpp_src")
124+
spark.sql(
125+
s"""MERGE INTO delta.`$tablePath` trg
126+
|USING merge_dpp_src src
127+
|ON src.key1 = trg.key2 AND trg.key2 < '3'
128+
|WHEN MATCHED THEN UPDATE SET
129+
| key2 = src.key1, value = src.value, op = 'update'
130+
|WHEN NOT MATCHED THEN INSERT
131+
| (key2, value, op) VALUES (src.key1, src.value, 'insert')""".stripMargin)
132+
val rows = spark.read.format("delta").load(tablePath).collect().toSet
133+
val expected = Set(
134+
Row("3", "2", "noop"), Row("4", "4", "noop"),
135+
Row("1", "8", "update"), Row("1", "8", "update"), Row("0", "3", "insert"))
136+
assert(rows == expected, s"MERGE result mismatch\n got=$rows\n want=$expected")
137+
}
138+
}
139+
}
96140
}

0 commit comments

Comments
 (0)