Skip to content

Commit bc6429f

Browse files
committed
fix: strip the Iceberg write's input transition before transition cancellation
Review feedback: the write-boundary strip ran as an arm inside the rule's `transformUp`, so the generic `ColumnarToRowExec(CometSparkToColumnarExec)` cancellation reached the write's child first and destroyed the Arrow bridge the write's FFI input needs. Over a row source that arm drops the `CometSparkToColumnarExec` outright, leaving the write with a row child; over a Spark-columnar source it keeps a transition but hands the write Spark `ColumnarVector`s where the FFI adapter requires `CometVector`s. Both shapes are reachable through `spark.comet.sparkToColumnar.enabled`, whose `CometScanWrapper` passes `requiresNativeChildren` and is then unwrapped by `CometExecRule`, leaving the bridge directly beneath the write. Move the strip into its own pass ahead of `transformUp` so the write's input transition is removed before anything else can consume it. Tests: - Two plan-level tests in `CometIcebergWriteDetectionSuite` pin the boundary for both source representations. Both fail without this change. - `captureWrite` now runs `assertColumnarContract` on every captured plan rather than only the one test written to exercise it. - AQE-off CoW UPDATE and MERGE tests cover the two remaining rewrite shapes from #5689. The partitioned MERGE engages natively (the rebalance exchange makes the write's child Comet-native even though `MergeRowsExec` is not), so its subtree mixes a Spark-columnar scan, a row-based operator and Comet operators.
1 parent 585ffe8 commit bc6429f

3 files changed

Lines changed: 185 additions & 22 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala‎

Lines changed: 40 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ case class EliminateRedundantTransitions(session: SparkSession)
7474
}
7575

7676
private def _apply(plan: SparkPlan): SparkPlan = {
77-
val eliminatedPlan = plan transformUp {
77+
val eliminatedPlan = stripIcebergWriteInputTransition(plan) transformUp {
7878
case ColumnarToRowExec(shuffleExchangeExec: CometShuffleExchangeExec)
7979
if plan.conf.adaptiveExecutionEnabled =>
8080
shuffleExchangeExec
@@ -91,18 +91,6 @@ case class EliminateRedundantTransitions(session: SparkSession)
9191
// Write should be final operation in the plan
9292
case ColumnarToRowExec(nativeWrite: CometNativeWriteExec) =>
9393
nativeWrite
94-
// `CometIcebergWriteExec` is row-based (it emits the serialised Iceberg commit message) but
95-
// consumes Arrow batches from its child over FFI, so Spark inserts a columnar-to-row
96-
// transition *underneath* it. Strip it so `doExecuteColumnar` sees the columnar child
97-
// directly; `CometIcebergNativeWrite.requiresNativeChildren` already guarantees that child
98-
// was Comet-native when the write was converted.
99-
//
100-
// The write deliberately does not tag itself as a `ColumnarToRowTransition` to suppress the
101-
// insertion: Spark leaves such a node untouched, so the whole subtree below the write is
102-
// never visited and the transitions the rest of that subtree needs are never inserted
103-
// (https://github.com/apache/datafusion-comet/issues/5689).
104-
case w: CometIcebergWriteExec =>
105-
stripColumnarToRow(w.child).map(child => w.withNewChildren(Seq(child))).getOrElse(w)
10694
case c @ ColumnarToRowExec(child) if hasCometNativeChild(child) =>
10795
val op = createColumnarToRowExec(child)
10896
if (c.logicalLink.isEmpty) {
@@ -181,17 +169,49 @@ case class EliminateRedundantTransitions(session: SparkSession)
181169
}
182170
}
183171

172+
/**
173+
* `CometIcebergWriteExec` is row-based (it emits the serialised Iceberg commit message) but
174+
* consumes Arrow batches from its child over FFI, so Spark's
175+
* `ApplyColumnarRulesAndInsertTransitions` inserts a columnar-to-row transition *underneath*
176+
* it. Strip that transition so `doExecuteColumnar` sees the columnar child directly;
177+
* `CometIcebergNativeWrite.requiresNativeChildren` already guarantees the child was
178+
* Comet-native when the write was converted.
179+
*
180+
* The write deliberately does not tag itself as a `ColumnarToRowTransition` to suppress the
181+
* insertion: Spark leaves such a node untouched, so the whole subtree below the write is never
182+
* visited and the transitions the rest of that subtree needs are never inserted
183+
* (https://github.com/apache/datafusion-comet/issues/5689).
184+
*
185+
* This runs as a separate pass *before* the main `transformUp`, not as an arm inside it,
186+
* because the write's input transition has to be removed before the generic transition
187+
* cancellation can consume it. `transformUp` visits children first, so the
188+
* `ColumnarToRowExec(CometSparkToColumnarExec)` arm would otherwise rewrite the write's child
189+
* before the write itself is visited, and that arm is destructive at this boundary:
190+
* - over a row source it drops the `CometSparkToColumnarExec` as well, leaving the write with
191+
* a row child and no Arrow producer at all;
192+
* - over a Spark-columnar source it keeps a `ColumnarToRowExec` but drops the Arrow bridge,
193+
* so the write would be handed Spark `ColumnarVector`s where the FFI adapter requires
194+
* `CometVector`s. That shape is reachable whenever `spark.comet.sparkToColumnar.enabled`
195+
* admits the write's source: `CometSparkToColumnarExec.createExec` wraps it in a
196+
* `CometScanWrapper` (a `CometNativeExec`, so `requiresNativeChildren` accepts it), and
197+
* `CometExecRule` then unwraps the placeholder, leaving the bridge directly beneath the
198+
* write.
199+
*/
200+
private def stripIcebergWriteInputTransition(plan: SparkPlan): SparkPlan = plan.transform {
201+
case w: CometIcebergWriteExec =>
202+
stripColumnarToRow(w.child).map(child => w.withNewChildren(Seq(child))).getOrElse(w)
203+
}
204+
184205
/**
185206
* Unwraps a columnar-to-row transition, returning the columnar child underneath it, or `None`
186207
* when `plan` is not such a transition.
187208
*
188-
* All three forms have to be handled. `transformUp` visits children first, so the
189-
* `hasCometNativeChild` arm above has usually already rewritten a `ColumnarToRowExec` over a
190-
* Comet source into one of the Comet variants by the time a parent arm looks at it. It has not
191-
* when the source is an `AQEShuffleReadExec` over a `ShuffleQueryStageExec`: `QueryStageExec`
192-
* is a `LeafExecNode`, so the `op.exists(...)` walk cannot see the Comet exchange inside it and
193-
* the arm misses. That is the shape the Iceberg write gets under AQE, so the plain node reaches
194-
* here.
209+
* The plain `ColumnarToRowExec` is what Spark's insertion pass produces and is the only form
210+
* observed in the suites. The two Comet variants are handled as well because this rule is part
211+
* of `postColumnarTransitions` and AQE applies those rules once per materialised stage plus
212+
* once for the final plan, so it can be handed a plan whose transitions an earlier pass already
213+
* rewrote. Missing a variant would not be caught at planning time -- the write would fail at
214+
* runtime with `requires a columnar (Comet native) child`.
195215
*/
196216
private def stripColumnarToRow(plan: SparkPlan): Option[SparkPlan] = plan match {
197217
case CometNativeColumnarToRowExec(child) => Some(child)

‎spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -743,7 +743,6 @@ class CometIcebergWriteActionSuite
743743
nativeExecs.forall(_.child.supportsColumnar),
744744
"the native write must sit directly on its columnar child, got " +
745745
nativeExecs.map(_.child.nodeName).mkString(", "))
746-
snapshot.plans.foreach(assertColumnarContract)
747746
}
748747
assertRows("native_cow_delete_no_aqe", Seq(1, 3, 4))
749748
}
@@ -805,6 +804,73 @@ class CometIcebergWriteActionSuite
805804
}
806805
}
807806

807+
// The remaining two CoW shapes from issue #5689, with AQE off so the plan gets exactly one
808+
// transition-insertion pass. `captureWrite` runs `assertColumnarContract` on every captured
809+
// plan, so these pin the same invariant as the DELETE case over the UPDATE and MERGE rewrite
810+
// shapes, which put different operators between the columnar CoW scan and the write.
811+
test("native acceleration: ReplaceData (CoW UPDATE) with AQE disabled") {
812+
assumeNativeAcceleration()
813+
withIcebergCatalog { warehouseDir =>
814+
createTable(
815+
warehouseDir,
816+
"native_cow_update_no_aqe",
817+
partitionSpec = "PARTITIONED BY (region)",
818+
properties = Some("'write.update.mode'='copy-on-write'"))
819+
withSQLConf(CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> "false") {
820+
coalesceInsert(
821+
"native_cow_update_no_aqe",
822+
Seq((1, "us-east", 10.0), (2, "us-west", 20.0), (3, "eu", 30.0)))
823+
}
824+
// As in the DELETE case, the IN-subquery is what puts a row-based join between the
825+
// Spark-columnar CoW scan and the write.
826+
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
827+
assertNativeWriteEngages("native_cow_update_no_aqe", Seq(1, 2, 3)) {
828+
spark.sql(
829+
"UPDATE cat.db.native_cow_update_no_aqe SET amount = amount * 2 " +
830+
"WHERE id IN (SELECT col1 FROM VALUES (2) AS t(col1))")
831+
}
832+
}
833+
val r = spark
834+
.sql("SELECT amount FROM cat.db.native_cow_update_no_aqe WHERE id = 2")
835+
.collect()
836+
assert(r.length == 1 && r(0).getDouble(0) == 40.0, s"got ${r.toSeq}")
837+
}
838+
}
839+
840+
test("native acceleration: ReplaceData (CoW MERGE) with AQE disabled") {
841+
// Unlike the unpartitioned MERGE above, this one *does* engage natively: partitioning the
842+
// table puts a `CometColumnarExchange` (REBALANCE_PARTITIONS_BY_COL) between the JVM
843+
// `MergeRowsExec` and the write, so the write's own child is Comet-native and
844+
// `requiresNativeChildren` is satisfied even though `MergeRowsExec` itself is not.
845+
//
846+
// That makes this the strongest of the three CoW shapes for issue #5689: the subtree below
847+
// the write mixes a Spark-columnar `BatchScan`, a row-based `MergeRowsExec` and Comet
848+
// operators, so it needs transitions inserted in three different places.
849+
assumeNativeAcceleration()
850+
withIcebergCatalog { warehouseDir =>
851+
createTable(
852+
warehouseDir,
853+
"native_cow_merge_no_aqe",
854+
partitionSpec = "PARTITIONED BY (region)",
855+
properties = Some("'write.merge.mode'='copy-on-write'"))
856+
withSQLConf(CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> "false") {
857+
coalesceInsert("native_cow_merge_no_aqe", Seq((1, "us-east", 10.0), (2, "us-west", 20.0)))
858+
}
859+
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
860+
assertNativeWriteEngages("native_cow_merge_no_aqe", Seq(1, 2, 3)) {
861+
spark.sql("""
862+
|MERGE INTO cat.db.native_cow_merge_no_aqe t
863+
|USING (SELECT 2 AS id, 'us-west' AS region, 200.0 AS amount UNION ALL
864+
| SELECT 3 AS id, 'eu' AS region, 30.0 AS amount) s
865+
|ON t.id = s.id
866+
|WHEN MATCHED THEN UPDATE SET t.amount = s.amount
867+
|WHEN NOT MATCHED THEN INSERT (id, region, amount) VALUES (s.id, s.region, s.amount)
868+
|""".stripMargin)
869+
}
870+
}
871+
}
872+
}
873+
808874
test("native acceleration: complex types (struct, array, map) round-trip with field IDs") {
809875
assumeNativeAcceleration()
810876
withIcebergCatalog { _ =>
@@ -1686,6 +1752,10 @@ class CometIcebergWriteActionSuite
16861752
private def captureWrite(tableName: String)(action: => Unit): WriteSnapshot = {
16871753
val before = countSnapshots(tableName)
16881754
val plans = capturePlans(spark)(action)
1755+
// Every write in this suite gets the columnar contract checked, not just the ones written to
1756+
// exercise it: a Comet rewrite that hides part of a subtree from Spark's transition-insertion
1757+
// pass is a whole class of bug (issue #5689) and the check is free once the plans are here.
1758+
plans.foreach(assertColumnarContract)
16891759
WriteSnapshot(countSnapshots(tableName) - before, plans)
16901760
}
16911761

‎spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala‎

Lines changed: 74 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,20 @@ import org.apache.iceberg.hadoop.{HadoopConfigurable, HadoopFileIO}
2929
import org.apache.iceberg.io.{FileIO, InputFile, OutputFile}
3030
import org.apache.iceberg.util.SerializableSupplier
3131
import org.apache.spark.SparkConf
32+
import org.apache.spark.rdd.RDD
3233
import org.apache.spark.sql.CometTestBase
33-
import org.apache.spark.sql.comet.IcebergWriteExec
34+
import org.apache.spark.sql.catalyst.InternalRow
35+
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
36+
import org.apache.spark.sql.comet.{CometIcebergWriteExec, CometSparkToColumnarExec, IcebergWriteExec}
37+
import org.apache.spark.sql.execution.{ApplyColumnarRulesAndInsertTransitions, ColumnarToRowExec, LeafExecNode, SparkPlan}
38+
import org.apache.spark.sql.types.IntegerType
39+
import org.apache.spark.sql.vectorized.ColumnarBatch
3440

3541
import org.apache.comet.CometSparkSessionExtensions.isSpark35Plus
3642
import org.apache.comet.iceberg.IcebergReflection
43+
import org.apache.comet.rules.EliminateRedundantTransitions
3744
import org.apache.comet.serde.{Compatible, SupportLevel, Unsupported}
45+
import org.apache.comet.serde.OperatorOuterClass.Operator
3846
import org.apache.comet.serde.operator.CometIcebergNativeWrite
3947

4048
class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTestBase {
@@ -881,6 +889,71 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes
881889
fail(s"expected Unsupported for $tableName, got $other")
882890
}
883891
}
892+
893+
/**
894+
* Runs Spark's transition insertion followed by [[EliminateRedundantTransitions]] over a
895+
* hand-built `CometIcebergWriteExec -> CometSparkToColumnarExec -> source` plan and returns the
896+
* write's final child.
897+
*
898+
* Hand-built rather than driven through SQL because the shape depends on
899+
* `spark.comet.sparkToColumnar.enabled` admitting the write's source operator, and the set of
900+
* admitted operators is itself configurable. What matters is the rule's behaviour at that
901+
* boundary, which this pins directly.
902+
*/
903+
private def writeChildAfterTransitionRules(source: SparkPlan): SparkPlan = {
904+
val write = CometIcebergWriteExec(
905+
Operator.newBuilder().build(),
906+
CometSparkToColumnarExec(source),
907+
batchWrite = null,
908+
table = null,
909+
partitionSpecId = 0)
910+
val withTransitions = ApplyColumnarRulesAndInsertTransitions(Seq.empty, false).apply(write)
911+
// Spark must insert a columnar-to-row transition below the row-based write; if it stops doing
912+
// so the rest of the assertion is vacuous.
913+
assert(
914+
withTransitions.asInstanceOf[CometIcebergWriteExec].child.isInstanceOf[ColumnarToRowExec],
915+
s"expected an inserted ColumnarToRowExec below the write, got:\n$withTransitions")
916+
EliminateRedundantTransitions(spark)
917+
.apply(withTransitions)
918+
.asInstanceOf[CometIcebergWriteExec]
919+
.child
920+
}
921+
922+
// https://github.com/apache/datafusion-comet/issues/5689: the write's input transition has to
923+
// be stripped before the generic `ColumnarToRowExec(CometSparkToColumnarExec)` cancellation
924+
// consumes it, otherwise that arm removes the Arrow bridge the write's FFI input depends on.
925+
// Both source representations are covered because the cancellation treats them differently:
926+
// over a row source it drops the bridge outright, over a Spark-columnar source it keeps a
927+
// transition but leaves the write reading Spark `ColumnarVector`s instead of `CometVector`s.
928+
test("row source keeps its Arrow bridge under the native Iceberg write") {
929+
val source = TransitionProbeLeaf(columnar = false)
930+
val child = writeChildAfterTransitionRules(source)
931+
assert(
932+
child == CometSparkToColumnarExec(source),
933+
s"expected the write to sit directly on CometSparkToColumnarExec, got:\n$child")
934+
}
935+
936+
test("Spark-columnar source keeps its Arrow bridge under the native Iceberg write") {
937+
val source = TransitionProbeLeaf(columnar = true)
938+
val child = writeChildAfterTransitionRules(source)
939+
assert(
940+
child == CometSparkToColumnarExec(source),
941+
s"expected the write to sit directly on CometSparkToColumnarExec, got:\n$child")
942+
}
943+
}
944+
945+
/**
946+
* Planning-only leaf used by the transition-boundary tests: `columnar` selects between the two
947+
* source representations that can sit under a `CometSparkToColumnarExec`. Never executed.
948+
*/
949+
case class TransitionProbeLeaf(columnar: Boolean) extends LeafExecNode {
950+
override def output: Seq[Attribute] = Seq(
951+
AttributeReference("id", IntegerType, nullable = false)())
952+
override def supportsColumnar: Boolean = columnar
953+
override protected def doExecute(): RDD[InternalRow] =
954+
throw new UnsupportedOperationException("planning-only node")
955+
override protected def doExecuteColumnar(): RDD[ColumnarBatch] =
956+
throw new UnsupportedOperationException("planning-only node")
884957
}
885958

886959
/**

0 commit comments

Comments
 (0)