Skip to content

Commit 0e66bc0

Browse files
committed
fix: drop the unused SparkConf import and narrow the min/max workaround
The suite no longer overrides sparkConf, so its SparkConf import failed Scalafix and the strict-warnings build. Reformat a CometExecRule comment that Spotless rejected. The three-config min/max workaround only keeps the decode out of the query when the cast is the aggregate's own argument. A cast in a projection below the aggregate runs natively and decodes first, so the guide now says so and keeps disabling Comet as the general answer.
1 parent 5104964 commit 0e66bc0

3 files changed

Lines changed: 9 additions & 6 deletions

File tree

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -123,9 +123,13 @@ reach Comet in these ways, which are the ones to check before relying on a resul
123123
With `b` holding `X'80'` and `X'C3A9'`, Spark returns `min(CAST(b AS STRING))` = the `0x80` string
124124
and `max` = `é`, whereas Comet returns `min` = `é` and `max` = `U+FFFD`. This is the ordering
125125
difference above. If the cast input can hold bytes that are not valid UTF-8, disable Comet for the
126-
query, or set `spark.comet.expression.Min.enabled=false`, `spark.comet.expression.Max.enabled=false`
127-
and `spark.comet.shuffle.mode=native`. All three are needed: with only the first two, the aggregate
128-
runs in Spark, but Comet's columnar shuffle still decodes its partial results.
126+
query. When the cast is the direct argument of `min` or `max`, a narrower alternative is to set
127+
`spark.comet.expression.Min.enabled=false`, `spark.comet.expression.Max.enabled=false` and
128+
`spark.comet.shuffle.mode=native`. All three are needed: with only the first two, the aggregate
129+
runs in Spark, but Comet's columnar shuffle still decodes its partial results. These settings are
130+
not enough when the cast is in a projection below the aggregate, for example
131+
`SELECT min(s) FROM (SELECT CAST(b AS STRING) AS s FROM t)`, because Comet runs that projection
132+
natively and decodes the bytes before the aggregate sees them.
129133

130134
Consistent handling of invalid UTF-8 across all native string paths is tracked by
131135
[#4764](https://github.com/apache/datafusion-comet/issues/4764).

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1540,8 +1540,8 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
15401540
* Parquet, and so is a Text scan on a broadcast build side: a Text file can hold bytes that are
15411541
* not valid UTF-8, for example a Latin-1 file, and the conversion replaces them with U+FFFD, so
15421542
* results that depend on the original bytes would differ from Spark. A Text build side that the
1543-
* bridge would otherwise convert records why it did not, so the explain output names the config.
1544-
* See issue #7118.
1543+
* bridge would otherwise convert records why it did not, so the explain output names the
1544+
* config. See issue #7118.
15451545
*/
15461546
private def shouldConvertTextScan(conf: SQLConf, op: SparkPlan): Boolean =
15471547
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_TEXT_ENABLED) || {

‎spark/src/test/scala/org/apache/comet/CometBroadcastBuildSideSuite.scala‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@ import java.io.File
2323
import java.nio.charset.StandardCharsets.UTF_8
2424
import java.nio.file.Files
2525

26-
import org.apache.spark.SparkConf
2726
import org.apache.spark.sql.{CometTestBase, DataFrame, Row}
2827
import org.apache.spark.sql.catalyst.expressions.{DynamicPruningExpression, Literal}
2928
import org.apache.spark.sql.comet._

0 commit comments

Comments
 (0)