Skip to content

Commit 4c3f048

Browse files
authored
feat: disable native columnar-to-row conversion by default (#5114)
* feat: disable native columnar-to-row conversion by default Change the default of spark.comet.exec.columnarToRow.native.enabled from true to false. Isolated benchmarking in #5112 shows the native converter is 3.7x slower per row than the JVM implementation at the default batch size and up to 15.7x slower for small batches due to a fixed JNI and FFI cost of roughly 10us per batch, while end-to-end benchmarks and TPC-DS runs show no benefit over the JVM operator. Also update the operator scaladoc, which claimed zero-copy strings and reduced GC pressure: rows have been deep-copied to the JVM heap since the fix for #3308, and #4440 found no measurable GC benefit. Part of #5112 * fix: strip CometColumnarToRowExec in DPP subquery broadcast rewrite The non-AQE DPP rewrite in CometExecRule only recognized CometNativeColumnarToRowExec under the SubqueryBroadcastExec's BroadcastExchangeExec. With the native conversion disabled by default, the JVM CometColumnarToRowExec is used instead, and because it is CodegenSupport the prepared subquery plan wraps it as WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(cometPlan))). The rewrite did not match this shape, so SubqueryBroadcastExec was never converted to CometSubqueryBroadcastExec and DPP broadcast exchange reuse stopped working. Unwrap the codegen and InputAdapter layers so both columnar-to-row variants are handled. * test: regenerate plan stability golden files for JVM columnar-to-row default Regenerated with dev/regenerate-golden-files.sh for all Spark versions. The only change is the operator rename from CometNativeColumnarToRow to CometColumnarToRow at plan stage roots; CometSubqueryBroadcast and exchange reuse are unchanged thanks to the DPP rewrite fix. * test: accept CometColumnarToRowExec in SubquerySuite partition pruning test * test: refresh golden files and plan docs for JVM columnar-to-row default * test: accept codegen-wrapped CometColumnarToRow in CachedBatchSerializerNoUnwrapSuite The Comet patch for `CachedBatchSerializerNoUnwrapSuite` accepted either the Spark shape (`WholeStageCodegenExec(ColumnarToRowExec)`) or a cached plan whose top node is in `org.apache.spark.sql.comet`. The latter only held for `CometNativeColumnarToRowExec`, which does not support codegen. Now that the JVM `CometColumnarToRowExec` is the default, the cached plan is `WholeStageCodegenExec(CometColumnarToRow(...))`, so both checks failed: isSparkShape was false, and isCometShape was false unexpected cached plan: *(1) CometColumnarToRow +- CometNativeScan parquet [count#822213] ... Look through a `WholeStageCodegenExec` before classifying the transition node so both Comet operators are accepted. * fix: do not report decimal avg overflow when there is nothing to average Closes #5148. In ANSI mode, `avg` on a decimal column raised `[ARITHMETIC_OVERFLOW] Overflow in sum of decimals` for an ungrouped aggregate whose input was empty or entirely null, where Spark returns NULL. `AvgDecimalAccumulator::merge_batch` derives `is_empty` from the nullness of the merged counts, but a global partial aggregate over zero rows still emits one row with `sum = NULL, count = 0` and that count is not null, so `is_empty` becomes false. `evaluate` then treated the missing sum as an overflow. Guard that check with `count > 0`, matching the neighbouring `is_not_null` check and the `GroupsAccumulator` path (which is why only aggregates without GROUP BY were affected). This is what makes `LogicalPlanTagInSparkPlanSuite` q24a/q24b/q24-v2.7 fail in the Spark 4.1 SQL lane once the JVM columnar-to-row operator is the default: `CometColumnarToRowExec` supports codegen, so `checkGeneratedCode` calls `WholeStageCodegenExec.doCodeGen`, which prepares the plan and eagerly executes q24's `SELECT 0.05 * avg(netpaid) FROM ssales` scalar subquery against the row-less TPC-DS tables that suite creates.
1 parent 4b091ac commit 4c3f048

169 files changed

Lines changed: 411 additions & 320 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎dev/diffs/3.4.3.diff‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,14 +1124,14 @@ index 18123a4d6ec..0fe185baa33 100644
11241124

11251125
test("non-matching optional group") {
11261126
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
1127-
index 75eabcb96f2..f8141c28f60 100644
1127+
index 75eabcb96f2..c68f4827bf1 100644
11281128
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
11291129
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
11301130
@@ -21,10 +21,11 @@ import scala.collection.mutable.ArrayBuffer
11311131

11321132
import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
11331133
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join, LogicalPlan, Project, Sort, Union}
1134-
+import org.apache.spark.sql.comet.{CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
1134+
+import org.apache.spark.sql.comet.{CometColumnarToRowExec, CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
11351135
import org.apache.spark.sql.execution._
11361136
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecution}
11371137
import org.apache.spark.sql.execution.datasources.FileScanRDD
@@ -1140,7 +1140,7 @@ index 75eabcb96f2..f8141c28f60 100644
11401140
import org.apache.spark.sql.execution.joins.{BaseJoinExec, BroadcastHashJoinExec, BroadcastNestedLoopJoinExec}
11411141
import org.apache.spark.sql.internal.SQLConf
11421142
import org.apache.spark.sql.test.SharedSparkSession
1143-
@@ -1543,6 +1544,17 @@ class SubquerySuite extends QueryTest
1143+
@@ -1543,6 +1544,29 @@ class SubquerySuite extends QueryTest
11441144
fs.inputRDDs().forall(
11451145
_.asInstanceOf[FileScanRDD].filePartitions.forall(
11461146
_.files.forall(_.urlEncodedPath.contains("p=0"))))
@@ -1150,6 +1150,18 @@ index 75eabcb96f2..f8141c28f60 100644
11501150
+ fs.inputRDDs().forall(
11511151
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
11521152
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1153+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1154+
+ fs @ CometScanExec(_, _, _, partitionFilters, _, _, _, _, _, _)))) =>
1155+
+ partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1156+
+ fs.inputRDDs().forall(
1157+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1158+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1159+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1160+
+ fs: CometNativeScanExec))) =>
1161+
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1162+
+ fs.inputRDDs().forall(
1163+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1164+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
11531165
+ case CometNativeColumnarToRowExec(fs: CometNativeScanExec) =>
11541166
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
11551167
+ fs.inputRDDs().forall(
@@ -1158,7 +1170,7 @@ index 75eabcb96f2..f8141c28f60 100644
11581170
case _ => false
11591171
})
11601172
}
1161-
@@ -2108,7 +2120,7 @@ class SubquerySuite extends QueryTest
1173+
@@ -2108,7 +2132,7 @@ class SubquerySuite extends QueryTest
11621174

11631175
df.collect()
11641176
val exchanges = collect(df.queryExecution.executedPlan) {

‎dev/diffs/3.5.8.diff‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1065,14 +1065,14 @@ index 71af1fd69c3..81a04c93c9c 100644
10651065
}
10661066
}
10671067
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
1068-
index 04702201f82..c5ab5443ff7 100644
1068+
index 04702201f82..4d38d8d6e51 100644
10691069
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
10701070
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
10711071
@@ -22,10 +22,11 @@ import scala.collection.mutable.ArrayBuffer
10721072
import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
10731073
import org.apache.spark.sql.catalyst.plans.{LeftAnti, LeftSemi}
10741074
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join, LogicalPlan, Project, Sort, Union}
1075-
+import org.apache.spark.sql.comet.{CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
1075+
+import org.apache.spark.sql.comet.{CometColumnarToRowExec, CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
10761076
import org.apache.spark.sql.execution._
10771077
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecution}
10781078
import org.apache.spark.sql.execution.datasources.FileScanRDD
@@ -1081,7 +1081,7 @@ index 04702201f82..c5ab5443ff7 100644
10811081
import org.apache.spark.sql.execution.joins.{BaseJoinExec, BroadcastHashJoinExec, BroadcastNestedLoopJoinExec}
10821082
import org.apache.spark.sql.internal.SQLConf
10831083
import org.apache.spark.sql.test.SharedSparkSession
1084-
@@ -1599,6 +1600,17 @@ class SubquerySuite extends QueryTest
1084+
@@ -1599,6 +1600,29 @@ class SubquerySuite extends QueryTest
10851085
fs.inputRDDs().forall(
10861086
_.asInstanceOf[FileScanRDD].filePartitions.forall(
10871087
_.files.forall(_.urlEncodedPath.contains("p=0"))))
@@ -1091,6 +1091,18 @@ index 04702201f82..c5ab5443ff7 100644
10911091
+ fs.inputRDDs().forall(
10921092
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
10931093
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1094+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1095+
+ fs @ CometScanExec(_, _, _, partitionFilters, _, _, _, _, _, _)))) =>
1096+
+ partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1097+
+ fs.inputRDDs().forall(
1098+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1099+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1100+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1101+
+ fs: CometNativeScanExec))) =>
1102+
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1103+
+ fs.inputRDDs().forall(
1104+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1105+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
10941106
+ case CometNativeColumnarToRowExec(fs: CometNativeScanExec) =>
10951107
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
10961108
+ fs.inputRDDs().forall(
@@ -1099,7 +1111,7 @@ index 04702201f82..c5ab5443ff7 100644
10991111
case _ => false
11001112
})
11011113
}
1102-
@@ -2164,7 +2176,7 @@ class SubquerySuite extends QueryTest
1114+
@@ -2164,7 +2188,7 @@ class SubquerySuite extends QueryTest
11031115

11041116
df.collect()
11051117
val exchanges = collect(df.queryExecution.executedPlan) {

‎dev/diffs/4.0.2.diff‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1298,14 +1298,14 @@ index 0df7f806272..9cdfe8b8f46 100644
12981298
df.collect()
12991299
}
13001300
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
1301-
index 2e33f6505ab..be967565303 100644
1301+
index 2e33f6505ab..f0e64fd58ca 100644
13021302
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
13031303
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
13041304
@@ -23,10 +23,11 @@ import org.apache.spark.SparkRuntimeException
13051305
import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
13061306
import org.apache.spark.sql.catalyst.plans.{LeftAnti, LeftSemi}
13071307
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Join, LogicalPlan, Project, Sort, Union}
1308-
+import org.apache.spark.sql.comet.{CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
1308+
+import org.apache.spark.sql.comet.{CometColumnarToRowExec, CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
13091309
import org.apache.spark.sql.execution._
13101310
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecution}
13111311
import org.apache.spark.sql.execution.datasources.FileScanRDD
@@ -1314,7 +1314,7 @@ index 2e33f6505ab..be967565303 100644
13141314
import org.apache.spark.sql.execution.joins.{BaseJoinExec, BroadcastHashJoinExec, BroadcastNestedLoopJoinExec}
13151315
import org.apache.spark.sql.internal.SQLConf
13161316
import org.apache.spark.sql.test.SharedSparkSession
1317-
@@ -1529,6 +1530,18 @@ class SubquerySuite extends QueryTest
1317+
@@ -1529,6 +1530,30 @@ class SubquerySuite extends QueryTest
13181318
fs.inputRDDs().forall(
13191319
_.asInstanceOf[FileScanRDD].filePartitions.forall(
13201320
_.files.forall(_.urlEncodedPath.contains("p=0"))))
@@ -1324,6 +1324,18 @@ index 2e33f6505ab..be967565303 100644
13241324
+ fs.inputRDDs().forall(
13251325
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
13261326
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1327+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1328+
+ fs @ CometScanExec(_, _, _, partitionFilters, _, _, _, _, _, _)))) =>
1329+
+ partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1330+
+ fs.inputRDDs().forall(
1331+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1332+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1333+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1334+
+ fs: CometNativeScanExec))) =>
1335+
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1336+
+ fs.inputRDDs().forall(
1337+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1338+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
13271339
+ case CometNativeColumnarToRowExec(
13281340
+ fs: CometNativeScanExec) =>
13291341
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
@@ -1333,7 +1345,7 @@ index 2e33f6505ab..be967565303 100644
13331345
case _ => false
13341346
})
13351347
}
1336-
@@ -2094,7 +2107,7 @@ class SubquerySuite extends QueryTest
1348+
@@ -2094,7 +2119,7 @@ class SubquerySuite extends QueryTest
13371349

13381350
df.collect()
13391351
val exchanges = collect(df.queryExecution.executedPlan) {
@@ -1342,7 +1354,7 @@ index 2e33f6505ab..be967565303 100644
13421354
}
13431355
assert(exchanges.size === 1)
13441356
}
1345-
@@ -2678,18 +2691,29 @@ class SubquerySuite extends QueryTest
1357+
@@ -2678,18 +2703,29 @@ class SubquerySuite extends QueryTest
13461358
def checkFileSourceScan(query: String, answer: Seq[Row]): Unit = {
13471359
val df = sql(query)
13481360
checkAnswer(df, answer)

‎dev/diffs/4.1.2.diff‎

Lines changed: 29 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1397,14 +1397,14 @@ index 7bfc8cf4fa6..4bd387801db 100644
13971397
df.collect()
13981398
}
13991399
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
1400-
index 3ba48da0e32..0313aaa9ec6 100644
1400+
index 3ba48da0e32..a7f9513b9c8 100644
14011401
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
14021402
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
14031403
@@ -23,10 +23,11 @@ import org.apache.spark.SparkRuntimeException
14041404
import org.apache.spark.sql.catalyst.expressions.{EqualTo, NamedExpression, OuterReference, SubqueryExpression}
14051405
import org.apache.spark.sql.catalyst.plans.{LeftAnti, LeftSemi}
14061406
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Join, LogicalPlan, Project, Sort, Union}
1407-
+import org.apache.spark.sql.comet.{CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
1407+
+import org.apache.spark.sql.comet.{CometColumnarToRowExec, CometNativeColumnarToRowExec, CometNativeScanExec, CometScanExec}
14081408
import org.apache.spark.sql.execution._
14091409
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecution}
14101410
import org.apache.spark.sql.execution.datasources.FileScanRDD
@@ -1413,7 +1413,7 @@ index 3ba48da0e32..0313aaa9ec6 100644
14131413
import org.apache.spark.sql.execution.joins.{BaseJoinExec, BroadcastHashJoinExec, BroadcastNestedLoopJoinExec}
14141414
import org.apache.spark.sql.internal.SQLConf
14151415
import org.apache.spark.sql.test.SharedSparkSession
1416-
@@ -1529,6 +1530,18 @@ class SubquerySuite extends QueryTest
1416+
@@ -1529,6 +1530,30 @@ class SubquerySuite extends QueryTest
14171417
fs.inputRDDs().forall(
14181418
_.asInstanceOf[FileScanRDD].filePartitions.forall(
14191419
_.files.forall(_.urlEncodedPath.contains("p=0"))))
@@ -1423,6 +1423,18 @@ index 3ba48da0e32..0313aaa9ec6 100644
14231423
+ fs.inputRDDs().forall(
14241424
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
14251425
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1426+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1427+
+ fs @ CometScanExec(_, _, _, partitionFilters, _, _, _, _, _, _)))) =>
1428+
+ partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1429+
+ fs.inputRDDs().forall(
1430+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1431+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
1432+
+ case WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(
1433+
+ fs: CometNativeScanExec))) =>
1434+
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
1435+
+ fs.inputRDDs().forall(
1436+
+ _.asInstanceOf[FileScanRDD].filePartitions.forall(
1437+
+ _.files.forall(_.urlEncodedPath.contains("p=0"))))
14261438
+ case CometNativeColumnarToRowExec(
14271439
+ fs: CometNativeScanExec) =>
14281440
+ fs.partitionFilters.exists(ExecSubqueryExpression.hasSubquery) &&
@@ -1432,7 +1444,7 @@ index 3ba48da0e32..0313aaa9ec6 100644
14321444
case _ => false
14331445
})
14341446
}
1435-
@@ -2094,7 +2107,7 @@ class SubquerySuite extends QueryTest
1447+
@@ -2094,7 +2119,7 @@ class SubquerySuite extends QueryTest
14361448

14371449
df.collect()
14381450
val exchanges = collect(df.queryExecution.executedPlan) {
@@ -1441,7 +1453,7 @@ index 3ba48da0e32..0313aaa9ec6 100644
14411453
}
14421454
assert(exchanges.size === 1)
14431455
}
1444-
@@ -2678,18 +2691,29 @@ class SubquerySuite extends QueryTest
1456+
@@ -2678,18 +2703,29 @@ class SubquerySuite extends QueryTest
14451457
def checkFileSourceScan(query: String, answer: Seq[Row]): Unit = {
14461458
val df = sql(query)
14471459
checkAnswer(df, answer)
@@ -2724,10 +2736,10 @@ index 3e7d26f74bd..79232dc3664 100644
27242736
assert(o1.semanticEquals(o2), "Different output column order after AQE optimization")
27252737
}
27262738
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/CachedBatchSerializerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/CachedBatchSerializerSuite.scala
2727-
index 47b935a2880..3e9b87f5c32 100644
2739+
index 47b935a2880..3fdeab3113c 100644
27282740
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/CachedBatchSerializerSuite.scala
27292741
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/CachedBatchSerializerSuite.scala
2730-
@@ -230,9 +230,16 @@ class CachedBatchSerializerNoUnwrapSuite extends QueryTest
2742+
@@ -230,9 +230,21 @@ class CachedBatchSerializerNoUnwrapSuite extends QueryTest
27312743
assert(cachedPlans.length == 2)
27322744
cachedPlans.foreach {
27332745
cachedPlan =>
@@ -2738,11 +2750,16 @@ index 47b935a2880..3e9b87f5c32 100644
27382750
+ // (CometNativeColumnarToRow / CometColumnarToRow). Accept either the
27392751
+ // Spark shape or the equivalent Comet shape, since the important property
27402752
+ // under this serializer is that the row conversion is not unwrapped.
2741-
+ val isSparkShape =
2742-
+ cachedPlan.isInstanceOf[WholeStageCodegenExec] &&
2743-
+ cachedPlan.asInstanceOf[WholeStageCodegenExec]
2744-
+ .child.isInstanceOf[ColumnarToRowExec]
2745-
+ val isCometShape = cachedPlan.getClass.getName.startsWith("org.apache.spark.sql.comet.")
2753+
+ // CometColumnarToRowExec supports codegen, so it appears as the child of a
2754+
+ // WholeStageCodegenExec, while CometNativeColumnarToRowExec does not and is
2755+
+ // the cached plan itself.
2756+
+ val transition = cachedPlan match {
2757+
+ case w: WholeStageCodegenExec => w.child
2758+
+ case other => other
2759+
+ }
2760+
+ val isSparkShape = cachedPlan.isInstanceOf[WholeStageCodegenExec] &&
2761+
+ transition.isInstanceOf[ColumnarToRowExec]
2762+
+ val isCometShape = transition.getClass.getName.startsWith("org.apache.spark.sql.comet.")
27462763
+ assert(isSparkShape || isCometShape, s"unexpected cached plan:\n$cachedPlan")
27472764
}
27482765
}

‎docs/source/user-guide/latest/installation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ Comet will log output similar to:
161161

162162
```shell
163163
== Physical Plan ==
164-
CometNativeColumnarToRow
164+
CometColumnarToRow
165165
+- CometFilter [a#6], (isnotnull(a#6) AND (a#6 > 5))
166166
+- CometNativeScan parquet [a#6] Batched: true, DataFilters: [isnotnull(a#6), (a#6 > 5)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [], PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct<a:int>
167167
```

0 commit comments

Comments
 (0)