Skip to content

Commit bdbc460

Browse files
authored
feat: give each default Spark-to-Arrow conversion its own spark.comet.convert config (#6602)
Add spark.comet.convert.range.enabled, spark.comet.convert.inMemoryCache.enabled, spark.comet.convert.rdd.enabled and spark.comet.convert.oneRowRelation.enabled for the four operators that spark.comet.sparkToColumnar.supportedOperatorList named by default. All are off by default. The list now defaults to empty and stays the way to convert other leaf operators. Using spark.comet.sparkToColumnar.enabled and the list for the four operators above is deprecated: it keeps working as before and logs a one-time warning naming the new config. The OneRowRelation config matches both the RDDScanExec Spark plans before 4.1 and the OneRowRelationExec it plans from 4.1. CometTestBase and the suites that set the old keys move to the new ones.
1 parent 55c7070 commit bdbc460

22 files changed

Lines changed: 448 additions & 133 deletions

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,25 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
4949
Comet does not provide a Rust-based JSON scan, but when `spark.comet.convert.json.enabled` is enabled, data is immediately
5050
converted into Arrow format, allowing the Comet pipeline to take over after that.
5151

52+
### Other Spark inputs
53+
54+
Comet can also convert the output of these Spark inputs to Arrow format, so that the operators
55+
above them can run in Comet. Each conversion is off by default.
56+
57+
- `spark.comet.convert.range.enabled`: `spark.range` and SQL `range()`, for ranges that Comet does
58+
not generate natively with `spark.comet.exec.range.enabled`.
59+
- `spark.comet.convert.inMemoryCache.enabled`: in-memory cached tables that Comet's native cache
60+
scan does not read, such as tables cached in Spark's default format.
61+
- `spark.comet.convert.rdd.enabled`: a DataFrame created from an RDD of rows, for example with
62+
`spark.createDataFrame(rdd, schema)`.
63+
- `spark.comet.convert.oneRowRelation.enabled`: the single row that a query without a `FROM`
64+
clause, such as `SELECT 1`, reads.
65+
66+
To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
67+
format other than Parquet, JSON and CSV, set `spark.comet.sparkToColumnar.enabled=true` and name the
68+
operator in `spark.comet.sparkToColumnar.supportedOperatorList` by its Spark class name without the
69+
`Exec` suffix, such as `BatchScan` or `FileSourceScan`.
70+
5271
### Spark-to-Comet conversion types
5372

5473
Spark-to-Comet conversion supports `ARRAY<STRING>` and `MAP<STRING,STRING>` with binary
@@ -58,10 +77,8 @@ This applies to Spark row and columnar inputs when conversion is enabled for the
5877
Other array element types, other map key/value types, nested collections, and non-binary
5978
string collations remain unsupported at this conversion boundary. Source defaults are unchanged.
6079

61-
This includes row-backed `ExistingRDD` inputs when
62-
`spark.comet.sparkToColumnar.enabled=true` and
63-
`spark.comet.sparkToColumnar.supportedOperatorList` includes `RDDScan`. Spark still produces
64-
the RDD rows; conversion lets eligible downstream operators execute in Comet.
80+
This includes row-backed `ExistingRDD` inputs when `spark.comet.convert.rdd.enabled=true`. Spark
81+
still produces the RDD rows; conversion lets eligible downstream operators execute in Comet.
6582

6683
## Data Catalogs
6784

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,29 @@ Treat setting one of these keys as a temporary measure. If you find you cannot s
5757
legacy behavior, please open an issue describing your use case so it can be considered before the
5858
key is removed.
5959

60+
## Upgrading to Comet 1.2.0
61+
62+
### Deprecated and Removed Settings
63+
64+
Using `spark.comet.sparkToColumnar.enabled` and `spark.comet.sparkToColumnar.supportedOperatorList`
65+
to convert ranges, in-memory cached tables, RDDs and queries without a `FROM` clause to Arrow is
66+
deprecated, and will stop working in a future major release. Each of these now has a config of its
67+
own:
68+
69+
| Operator in the list | Config |
70+
| -------------------- | -------------------------------------------- |
71+
| `Range` | `spark.comet.convert.range.enabled` |
72+
| `InMemoryTableScan` | `spark.comet.convert.inMemoryCache.enabled` |
73+
| `RDDScan` | `spark.comet.convert.rdd.enabled` |
74+
| `OneRowRelation` | `spark.comet.convert.oneRowRelation.enabled` |
75+
76+
`spark.comet.sparkToColumnar.supportedOperatorList` now defaults to an empty list. When it is not
77+
set, `spark.comet.sparkToColumnar.enabled=true` still converts these four, and a list that names one
78+
of them still converts it, as before. In both cases the driver logs a warning that names the config
79+
to use instead. Before Spark 4.1, Spark plans a query without a `FROM` clause as an `RDDScan`, so on
80+
those versions `RDDScan` in the list also converted it. The list remains the way to convert other
81+
leaf operators, such as the scan of a Data Source V2 connector.
82+
6083
## Upgrading to Comet 1.1.0
6184

6285
Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 50 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,42 @@ object CometConf extends ShimCometConf {
203203
.booleanConf
204204
.createWithDefault(false)
205205

206+
val COMET_CONVERT_FROM_RANGE_ENABLED: ConfigEntry[Boolean] =
207+
conf("spark.comet.convert.range.enabled")
208+
.category(CATEGORY_EXEC)
209+
.doc(
210+
"When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate " +
211+
"natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format.")
212+
.booleanConf
213+
.createWithDefault(false)
214+
215+
val COMET_CONVERT_FROM_IN_MEMORY_CACHE_ENABLED: ConfigEntry[Boolean] =
216+
conf("spark.comet.convert.inMemoryCache.enabled")
217+
.category(CATEGORY_EXEC)
218+
.doc(
219+
"When enabled, data from in-memory cached tables that Comet's native cache scan does " +
220+
"not read (see `spark.comet.exec.inMemoryCache.enabled`), such as tables cached in " +
221+
"Spark's default format, will be converted to Arrow format.")
222+
.booleanConf
223+
.createWithDefault(false)
224+
225+
val COMET_CONVERT_FROM_RDD_ENABLED: ConfigEntry[Boolean] =
226+
conf("spark.comet.convert.rdd.enabled")
227+
.category(CATEGORY_EXEC)
228+
.doc("When enabled, data from a DataFrame created from an RDD of rows, for example with " +
229+
"`spark.createDataFrame(rdd, schema)`, will be converted to Arrow format.")
230+
.booleanConf
231+
.createWithDefault(false)
232+
233+
val COMET_CONVERT_FROM_ONE_ROW_RELATION_ENABLED: ConfigEntry[Boolean] =
234+
conf("spark.comet.convert.oneRowRelation.enabled")
235+
.category(CATEGORY_EXEC)
236+
.doc(
237+
"When enabled, the single row that a query without a FROM clause, such as `SELECT 1`, " +
238+
"reads will be converted to Arrow format.")
239+
.booleanConf
240+
.createWithDefault(false)
241+
206242
val COMET_EXEC_ENABLED: ConfigEntry[Boolean] = conf(s"$COMET_EXEC_CONFIG_PREFIX.enabled")
207243
.category(CATEGORY_EXEC)
208244
.doc(
@@ -1053,21 +1089,28 @@ object CometConf extends ShimCometConf {
10531089
val COMET_SPARK_TO_ARROW_ENABLED: ConfigEntry[Boolean] =
10541090
conf("spark.comet.sparkToColumnar.enabled")
10551091
.category(CATEGORY_EXEC)
1056-
.doc("Whether to enable Spark to Arrow columnar conversion. When this is turned on, " +
1057-
"Comet will convert operators in " +
1058-
"`spark.comet.sparkToColumnar.supportedOperatorList` into Arrow columnar format before " +
1059-
"processing.")
1092+
.doc("Whether to convert the output of the leaf operators named in " +
1093+
"`spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, " +
1094+
"in-memory cached tables, RDDs and queries without a FROM clause have " +
1095+
"`spark.comet.convert` configs of their own. When the list is not set, this converts " +
1096+
"those four instead, which is deprecated and will be removed in a future major release.")
10601097
.booleanConf
10611098
.createWithDefault(false)
10621099

10631100
val COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST: ConfigEntry[Seq[String]] =
10641101
conf("spark.comet.sparkToColumnar.supportedOperatorList")
10651102
.category(CATEGORY_EXEC)
1066-
.doc("A comma-separated list of operators that will be converted to Arrow columnar " +
1067-
s"format when `${COMET_SPARK_TO_ARROW_ENABLED.key}` is true.")
1103+
.doc(
1104+
"A comma-separated list of leaf operators to convert to Arrow format when " +
1105+
s"`${COMET_SPARK_TO_ARROW_ENABLED.key}` is true, each named by its Spark class name " +
1106+
"without the `Exec` suffix. It is for operators that have no `spark.comet.convert` " +
1107+
"config of their own, such as `BatchScan` for a Data Source V2 connector, or " +
1108+
"`FileSourceScan` for file formats other than Parquet, JSON and CSV. Naming `Range`, " +
1109+
"`InMemoryTableScan`, `RDDScan` or `OneRowRelation` is deprecated and will stop " +
1110+
"working in a future major release. Use their `spark.comet.convert` configs instead.")
10681111
.stringConf
10691112
.toSequence
1070-
.createWithDefault(Seq("Range,InMemoryTableScan,RDDScan,OneRowRelation"))
1113+
.createWithDefault(Nil)
10711114

10721115
val COMET_CASE_CONVERSION_ENABLED: ConfigEntry[Boolean] =
10731116
conf("spark.comet.caseConversion.enabled")

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 51 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
package org.apache.comet.rules
2121

22+
import java.util.concurrent.ConcurrentHashMap
23+
2224
import scala.collection.mutable.ListBuffer
2325
import scala.jdk.CollectionConverters._
2426

@@ -32,7 +34,7 @@ import org.apache.spark.sql.catalyst.util.sideBySide
3234
import org.apache.spark.sql.comet._
3335
import org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer
3436
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
35-
import org.apache.spark.sql.comet.shims.ShimCometEmptyRelation
37+
import org.apache.spark.sql.comet.shims.{ShimCometEmptyRelation, ShimCometOneRowRelation}
3638
import org.apache.spark.sql.comet.util.Utils
3739
import org.apache.spark.sql.execution._
3840
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, AQEShuffleReadExec, BroadcastQueryStageExec, LogicalQueryStage, QueryStageExec, ShuffleQueryStageExec}
@@ -53,7 +55,7 @@ import org.apache.spark.sql.execution.window.WindowExec
5355
import org.apache.spark.sql.internal.SQLConf
5456
import org.apache.spark.sql.types._
5557

56-
import org.apache.comet.{CometConf, CometExplainInfo, ExtendedExplainInfo}
58+
import org.apache.comet.{CometConf, CometExplainInfo, ConfigEntry, ExtendedExplainInfo}
5759
import org.apache.comet.CometConf.{COMET_SPARK_TO_ARROW_ENABLED, COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST}
5860
import org.apache.comet.CometSparkSessionExtensions._
5961
import org.apache.comet.rules.CometExecRule.allExecs
@@ -167,6 +169,17 @@ object CometExecRule {
167169
*/
168170
val SKIP_COMET_BROADCAST_TAG: org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit] =
169171
org.apache.spark.sql.catalyst.trees.TreeNodeTag[Unit]("comet.skipCometBroadcast")
172+
173+
/**
174+
* The operators `spark.comet.sparkToColumnar.supportedOperatorList` named by default, before
175+
* each had a `spark.comet.convert` config of its own. `spark.comet.sparkToColumnar.enabled`
176+
* still converts them when the list is not set, which is deprecated.
177+
*/
178+
private val deprecatedSparkToArrowOperators =
179+
Seq("Range", "InMemoryTableScan", "RDDScan", "OneRowRelation")
180+
181+
/** Keys of the `spark.comet.convert` configs whose deprecated alternative has been warned. */
182+
private[rules] val warnedDeprecatedConversions = ConcurrentHashMap.newKeySet[String]()
170183
}
171184

172185
/**
@@ -1285,6 +1298,14 @@ case class CometExecRule(session: SparkSession)
12851298
case _: ParquetScan => CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.get(conf)
12861299
case _ => isSparkToArrowEnabled(conf, op)
12871300
}
1301+
case _: RangeExec =>
1302+
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_RANGE_ENABLED)
1303+
case _: InMemoryTableScanExec =>
1304+
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_IN_MEMORY_CACHE_ENABLED)
1305+
case _ if ShimCometOneRowRelation.isOneRowRelation(op) =>
1306+
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_ONE_ROW_RELATION_ENABLED)
1307+
case _: RDDScanExec =>
1308+
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_RDD_ENABLED)
12881309
// other leaf nodes
12891310
case _: LeafExecNode =>
12901311
isSparkToArrowEnabled(conf, op)
@@ -1297,11 +1318,38 @@ case class CometExecRule(session: SparkSession)
12971318
}
12981319
}
12991320

1321+
/**
1322+
* Whether to convert a leaf operator that has a `spark.comet.convert` config of its own. The
1323+
* `spark.comet.sparkToColumnar` settings still convert it too, which is deprecated.
1324+
*/
1325+
private def isConversionEnabled(
1326+
conf: SQLConf,
1327+
op: SparkPlan,
1328+
entry: ConfigEntry[Boolean]): Boolean = {
1329+
entry.get(conf) || {
1330+
val deprecated = isSparkToArrowEnabled(conf, op)
1331+
if (deprecated && CometExecRule.warnedDeprecatedConversions.add(entry.key)) {
1332+
logWarning(
1333+
s"Using ${COMET_SPARK_TO_ARROW_ENABLED.key} to convert " +
1334+
s"${Utils.getSimpleName(op.getClass)} to Arrow is deprecated and will stop " +
1335+
s"working in a future major release. Set ${entry.key}=true instead.")
1336+
}
1337+
deprecated
1338+
}
1339+
}
1340+
13001341
private def isSparkToArrowEnabled(conf: SQLConf, op: SparkPlan) = {
13011342
COMET_SPARK_TO_ARROW_ENABLED.get(conf) && {
13021343
val simpleClassName = Utils.getSimpleName(op.getClass)
13031344
val nodeName = simpleClassName.replaceAll("Exec$", "")
1304-
COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST.get(conf).contains(nodeName)
1345+
// Without the list, the switch still converts the operators the list named by default.
1346+
val operators =
1347+
if (conf.getConfString(COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST.key, null) == null) {
1348+
CometExecRule.deprecatedSparkToArrowOperators
1349+
} else {
1350+
COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST.get(conf)
1351+
}
1352+
operators.contains(nodeName)
13051353
}
13061354
}
13071355

‎spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -52,12 +52,12 @@ import org.apache.comet.shims.ShimSQLConf
5252
// `CometExec`. However, for certain operators such as `CometCollectLimitExec` which overrides
5353
// `executeCollect`, the redundant `ColumnarToRowExec` makes the override ineffective.
5454
//
55-
// Note about the second case: When `spark.comet.sparkToColumnar.enabled` is set, Comet will add
56-
// `CometSparkToColumnarExec` on top of row-based operators first, but the downstream operator
57-
// only takes row-based input as it's a vanilla Spark operator(as Comet cannot convert it for
58-
// various reasons) or Spark requests row-based output such as a `collect` call. Spark will adds
59-
// another `ColumnarToRowExec` on top of `CometSparkToColumnarExec`. In this case, the pair could
60-
// be removed.
55+
// Note about the second case: When a Spark-to-Arrow conversion config is set, such as
56+
// `spark.comet.convert.parquet.enabled`, Comet will add `CometSparkToColumnarExec` on top of
57+
// row-based operators first, but the downstream operator only takes row-based input as it's a
58+
// vanilla Spark operator(as Comet cannot convert it for various reasons) or Spark requests
59+
// row-based output such as a `collect` call. Spark will adds another `ColumnarToRowExec` on top
60+
// of `CometSparkToColumnarExec`. In this case, the pair could be removed.
6161
case class EliminateRedundantTransitions(session: SparkSession)
6262
extends Rule[SparkPlan]
6363
with ShimCometMapInBatch
@@ -236,7 +236,7 @@ case class EliminateRedundantTransitions(session: SparkSession)
236236
* a row child and no Arrow producer at all;
237237
* - over a Spark-columnar source it keeps a `ColumnarToRowExec` but drops the Arrow bridge,
238238
* so the write would be handed Spark `ColumnarVector`s where the FFI adapter requires
239-
* `CometVector`s. That shape is reachable whenever `spark.comet.sparkToColumnar.enabled`
239+
* `CometVector`s. That shape is reachable whenever a Spark-to-Arrow conversion config
240240
* admits the write's source: `CometSparkToColumnarExec.createExec` wraps it in a
241241
* `CometScanWrapper` (a `CometNativeExec`, so `requiresNativeChildren` accepts it), and
242242
* `CometExecRule` then unwraps the placeholder, leaving the bridge directly beneath the
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.spark.sql.comet.shims
21+
22+
import org.apache.spark.sql.execution.{RDDScanExec, SparkPlan}
23+
24+
object ShimCometOneRowRelation {
25+
26+
/**
27+
* Whether `plan` produces the single row of a query without a FROM clause. Before Spark 4.1,
28+
* Spark plans that as an `RDDScanExec` named after the logical `OneRowRelation`.
29+
*/
30+
def isOneRowRelation(plan: SparkPlan): Boolean = plan match {
31+
case scan: RDDScanExec => scan.name == "OneRowRelation"
32+
case _ => false
33+
}
34+
}
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.spark.sql.comet.shims
21+
22+
import org.apache.spark.sql.execution.{RDDScanExec, SparkPlan}
23+
24+
object ShimCometOneRowRelation {
25+
26+
/**
27+
* Whether `plan` produces the single row of a query without a FROM clause. Before Spark 4.1,
28+
* Spark plans that as an `RDDScanExec` named after the logical `OneRowRelation`.
29+
*/
30+
def isOneRowRelation(plan: SparkPlan): Boolean = plan match {
31+
case scan: RDDScanExec => scan.name == "OneRowRelation"
32+
case _ => false
33+
}
34+
}
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.spark.sql.comet.shims
21+
22+
import org.apache.spark.sql.execution.{OneRowRelationExec, SparkPlan}
23+
24+
object ShimCometOneRowRelation {
25+
26+
/**
27+
* Whether `plan` produces the single row of a query without a FROM clause. From Spark 4.1,
28+
* Spark plans that as a `OneRowRelationExec`.
29+
*/
30+
def isOneRowRelation(plan: SparkPlan): Boolean = plan.isInstanceOf[OneRowRelationExec]
31+
}

0 commit comments

Comments
 (0)