Skip to content

Commit 2b1c6a3

Browse files
authored
fix: keep Text build sides on Spark by default and add Text conversion configs (#7123)
* fix: default broadcast build side conversion to off and document invalid UTF-8 cases Converting a Text scan on a broadcast build side to Arrow decodes invalid UTF-8 to U+FFFD, so 1.2.0 could return different results from 1.1.0. Default spark.comet.convert.broadcastBuildSide.enabled to false and name the Text-source and CAST(binary AS STRING) cases in the compatibility guide. * test: move the #6008 build-side tests into CometBroadcastBuildSideSuite CometBroadcastBuildSideSuite turns the build-side bridge on for every test, so the tests no longer need the bridgeTest helper, and a test added later cannot miss the setting. The two tests that failed because they were not wrapped in bridgeTest now run with the bridge on, and the five that asserted nothing was bridged now exercise the bridge again. The test bodies are unchanged. * fix: keep Text build sides on Spark by default and add Text conversion configs Converting a Text scan to Arrow replaces bytes that are not valid UTF-8 with U+FFFD, so bridging a Text build side by default (#6786) could return different results from 1.1.0 (#7118). The other formats that reach the bridge either define strings as UTF-8 (ORC, Avro, Iceberg) or are decoded by Spark, so spark.comet.convert.broadcastBuildSide.enabled goes back to on by default, and Text gets its own configs: - spark.comet.convert.broadcastBuildSide.text.enabled (default false) bridges a Text build side. Without it, the scan records a fallback reason that names the config. - spark.comet.convert.text.enabled (default false) converts every Text scan, like the CSV, JSON and Parquet configs. The sparkToColumnar operator list still converts Text as before. The compatibility guide now names Text rather than CSV and JSON, no longer says that Spark requires Parquet strings to be valid UTF-8, adds Comet columnar shuffle as a path, and gives the configs that keep min/max over CAST(binary AS STRING) in Spark. * test: drop the QueryStageExec import that the test move left unused * fix: convert a Text build side only with spark.comet.convert.text.enabled Drop spark.comet.convert.broadcastBuildSide.text.enabled. A Text scan, on a build side or not, is now converted only by spark.comet.convert.text.enabled, like CSV, JSON and Parquet, and the build-side fallback reason names that config. Text goes through isConversionEnabled, so naming FileSourceScan or BatchScan in spark.comet.sparkToColumnar.supportedOperatorList still converts it but is deprecated. The #6008 build-side tests read ORC instead of Text, since a global Text switch would convert their probes and no-bridge assertions too. * docs: describe Text conversion through its own config only Update the Text, build-side and compatibility sections for the removal of spark.comet.convert.broadcastBuildSide.text.enabled, and note in the migration guide that converting Text through sparkToColumnar is deprecated. * fix: drop the unused SparkConf import and narrow the min/max workaround The suite no longer overrides sparkConf, so its SparkConf import failed Scalafix and the strict-warnings build. Reformat a CometExecRule comment that Spotless rejected. The three-config min/max workaround only keeps the decode out of the query when the cast is the aggregate's own argument. A cast in a projection below the aggregate runs natively and decodes first, so the guide now says so and keeps disabling Comet as the general answer.
1 parent 69111f6 commit 2b1c6a3

7 files changed

Lines changed: 859 additions & 665 deletions

File tree

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,9 +97,41 @@ underlying bytes rather than on the rendered text:
9797
bytes) but `true` in Comet. Equality, joins, grouping, ordering, and byte-based string functions
9898
such as `contains` can therefore disagree with Spark when non-UTF-8 bytes are involved.
9999

100-
Both differences require string data that is not valid UTF-8, which does not occur for text read from
101-
Parquet or produced by string expressions. Consistent handling of invalid UTF-8 across all native
102-
string paths is tracked by
100+
Both differences require string data that is not valid UTF-8. Comet's native Parquet scan never
101+
produces such strings (it rejects them, see below), and neither do most string expressions. They can
102+
reach Comet in these ways, which are the ones to check before relying on a result:
103+
104+
- **Text files.** Spark's Text reader keeps the bytes of each line as they are, so a Text file in an
105+
encoding other than UTF-8, such as Latin-1, holds strings that are not valid UTF-8. Comet decodes
106+
them when it converts a Text scan to Arrow, which it does only when
107+
`spark.comet.convert.text.enabled=true` (see [Data sources](../datasources.md#text)), or, as a
108+
deprecated alternative, when `spark.comet.sparkToColumnar.enabled=true` and
109+
`spark.comet.sparkToColumnar.supportedOperatorList` names the scan. For example, with a Text file
110+
holding the bytes `1 caf\xE9`, `hex(value)` returns `3120636166EFBFBD` instead of Spark's
111+
`3120636166E9`, and `length` of `2 na\xEFve` is 7 instead of 5. Spark decodes CSV files as it reads
112+
them, so they never hold such strings, and its JSON reader treats a line with such bytes as
113+
malformed, so only the corrupt record column keeps them. Spark's ORC and Avro readers keep stored
114+
bytes without checking them, but those formats define strings as UTF-8, so only a malformed file is
115+
affected.
116+
- **Comet columnar shuffle.** When Comet shuffles rows that come from a Spark operator, such as a
117+
Text scan or a Spark partial aggregate, it converts them to Arrow and decodes their strings in the
118+
same way. It does this with the default `spark.comet.shuffle.mode=auto`, so, for example,
119+
repartitioning a Latin-1 Text table can return `U+FFFD` where Spark keeps the original bytes.
120+
Setting `spark.comet.shuffle.mode=native` keeps such shuffles on Spark.
121+
- **`CAST(binary AS STRING)`.** Comet decodes the bytes where Spark keeps them. Because `min` and `max`
122+
over strings run natively, a cast below them is decoded before the comparison, and the two can swap.
123+
With `b` holding `X'80'` and `X'C3A9'`, Spark returns `min(CAST(b AS STRING))` = the `0x80` string
124+
and `max` = `é`, whereas Comet returns `min` = `é` and `max` = `U+FFFD`. This is the ordering
125+
difference above. If the cast input can hold bytes that are not valid UTF-8, disable Comet for the
126+
query. When the cast is the direct argument of `min` or `max`, a narrower alternative is to set
127+
`spark.comet.expression.Min.enabled=false`, `spark.comet.expression.Max.enabled=false` and
128+
`spark.comet.shuffle.mode=native`. All three are needed: with only the first two, the aggregate
129+
runs in Spark, but Comet's columnar shuffle still decodes its partial results. These settings are
130+
not enough when the cast is in a projection below the aggregate, for example
131+
`SELECT min(s) FROM (SELECT CAST(b AS STRING) AS s FROM t)`, because Comet runs that projection
132+
natively and decodes the bytes before the aggregate sees them.
133+
134+
Consistent handling of invalid UTF-8 across all native string paths is tracked by
103135
[#4764](https://github.com/apache/datafusion-comet/issues/4764).
104136

105137
Separately, Comet's native Parquet scan currently rejects string columns whose stored bytes are not

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,15 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
4949
Comet does not provide a Rust-based JSON scan, but when `spark.comet.convert.json.enabled` is enabled, data is immediately
5050
converted into Arrow format, allowing the Comet pipeline to take over after that.
5151

52+
### Text
53+
54+
Comet does not provide a Rust-based Text scan, but when `spark.comet.convert.text.enabled` is enabled, data from Spark's
55+
Text reader is immediately converted into Arrow format, allowing the Comet pipeline to take over after that. Spark's Text
56+
reader keeps the bytes of each line as they are, so a file in an encoding other than UTF-8, such as Latin-1, holds strings
57+
that are not valid UTF-8. Arrow strings must be valid UTF-8, so the conversion replaces those bytes with `U+FFFD`, and
58+
results that depend on the original bytes can differ from Spark. Enable it only for Text files that are UTF-8. See
59+
[Strings with non-UTF-8 bytes](compatibility/index.md#strings-with-non-utf-8-bytes).
60+
5261
### Other Spark inputs
5362

5463
Comet can also convert the output of these Spark inputs to Arrow format, so that the operators
@@ -67,12 +76,15 @@ because the row it converts has no columns.
6776
such as JDBC tables, which Spark scans with `RowDataSourceScanExec`.
6877

6978
To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
70-
format other than Parquet, JSON and CSV, set `spark.comet.sparkToColumnar.enabled=true` and name the
71-
operator in `spark.comet.sparkToColumnar.supportedOperatorList` by its Spark class name without the
72-
`Exec` suffix, such as `BatchScan` or `FileSourceScan`.
79+
format other than Parquet, JSON, CSV and Text, set `spark.comet.sparkToColumnar.enabled=true` and name
80+
the operator in `spark.comet.sparkToColumnar.supportedOperatorList` by its Spark class name without the
81+
`Exec` suffix, such as `BatchScan` or `FileSourceScan`. Naming either to convert a Text scan still
82+
works but is deprecated, like naming `Range`, `InMemoryTableScan`, `RDDScan`, `OneRowRelation` or
83+
`RowDataSourceScan`, and will stop working in a future major release. Use
84+
`spark.comet.convert.text.enabled` instead.
7385

7486
One case is converted automatically: when the build (broadcast) side of a broadcast join is a file
75-
or V2 scan Comet cannot read natively - for example a small Text lookup table - and every scan on
87+
or V2 scan Comet cannot read natively - for example a small ORC lookup table - and every scan on
7688
the probe side is already native, Comet converts that build-side scan to Arrow so the whole join
7789
runs natively over the large probe. This is controlled by
7890
`spark.comet.convert.broadcastBuildSide.enabled`, which is **on by default**, unlike the switches
@@ -81,6 +93,10 @@ the scan can make the copied volume larger, and with dynamic partition pruning t
8193
a second time for the pruning subquery. Set `spark.comet.convert.broadcastBuildSide.enabled=false` to
8294
keep such build sides on Spark.
8395

96+
Parquet, JSON, CSV and Text build sides are converted only by their own switches above. A Text file
97+
can hold bytes that are not valid UTF-8 (see [Text](#text)), so a Text lookup table stays on Spark
98+
unless `spark.comet.convert.text.enabled` is set.
99+
84100
### Spark-to-Comet conversion types
85101

86102
Spark-to-Comet conversion supports `ARRAY<STRING>` and `MAP<STRING,STRING>` with binary

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,9 @@ it. `spark.comet.convert.oneRowRelation.enabled` is on by default, so such a que
8686
without either setting, and logs no warning. The list remains the way to convert other leaf
8787
operators, such as the scan of a Data Source V2 connector.
8888

89+
Converting a Text scan with `FileSourceScan` or `BatchScan` in the list is deprecated in the same
90+
way, and logs the same warning. Use `spark.comet.convert.text.enabled` instead.
91+
8992
### Iceberg Writes
9093

9194
Comet's Iceberg write path is now switched by one setting, `spark.comet.write.iceberg.enabled`,

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,18 @@ object CometConf extends ShimCometConf {
245245
.booleanConf
246246
.createWithDefault(false)
247247

248+
val COMET_CONVERT_FROM_TEXT_ENABLED: ConfigEntry[Boolean] =
249+
conf("spark.comet.convert.text.enabled")
250+
.category(CATEGORY_EXEC)
251+
.doc(
252+
"When enabled, data from Spark (non-native) Text v1 and v2 scans will be converted to " +
253+
"Arrow format. Arrow strings must be valid UTF-8, so Comet replaces bytes that are " +
254+
"not valid UTF-8, for example in a Latin-1 file, with U+FFFD, and results that " +
255+
"depend on the original bytes can differ from Spark. Enable this only for Text " +
256+
"sources that are UTF-8. See the compatibility guide.")
257+
.booleanConf
258+
.createWithDefault(false)
259+
248260
val COMET_CONVERT_FROM_RANGE_ENABLED: ConfigEntry[Boolean] =
249261
conf("spark.comet.convert.range.enabled")
250262
.category(CATEGORY_EXEC)
@@ -1343,10 +1355,11 @@ object CometConf extends ShimCometConf {
13431355
s"`${COMET_SPARK_TO_ARROW_ENABLED.key}` is true, each named by its Spark class name " +
13441356
"without the `Exec` suffix. It is for operators that have no `spark.comet.convert` " +
13451357
"config of their own, such as `BatchScan` for a Data Source V2 connector, or " +
1346-
"`FileSourceScan` for file formats other than Parquet, JSON and CSV. Naming `Range`, " +
1347-
"`InMemoryTableScan`, `RDDScan`, `OneRowRelation` or `RowDataSourceScan` is " +
1348-
"deprecated and will stop working in a future major release. Use their " +
1349-
"`spark.comet.convert` configs instead.")
1358+
"`FileSourceScan` for file formats other than Parquet, JSON, CSV and Text. Naming " +
1359+
"`Range`, `InMemoryTableScan`, `RDDScan`, `OneRowRelation` or `RowDataSourceScan`, " +
1360+
"or naming `FileSourceScan` or `BatchScan` to convert a Text scan, is deprecated " +
1361+
"and will stop working in a future major release. Use their `spark.comet.convert` " +
1362+
"configs instead.")
13501363
.stringConf
13511364
.toSequence
13521365
.createWithDefault(Nil)
@@ -1355,13 +1368,14 @@ object CometConf extends ShimCometConf {
13551368
conf("spark.comet.convert.broadcastBuildSide.enabled")
13561369
.category(CATEGORY_EXEC)
13571370
.doc("When enabled, Comet converts a file or V2 scan that it cannot scan natively (for " +
1358-
"example a Text source) to Arrow when that scan feeds a broadcast join's build side and " +
1371+
"example an ORC source) to Arrow when that scan feeds a broadcast join's build side and " +
13591372
"every scan on the probe side is already native, so the broadcast join runs natively " +
13601373
"over the large probe. The build side is usually small (auto-broadcasts are capped by " +
13611374
"the broadcast threshold), so the row to Arrow copy is usually cheap - though an " +
13621375
"explicit BROADCAST hint or a selective filter above the scan can make it larger. With " +
13631376
"dynamic partition pruning the dimension is also read a second time for the pruning " +
1364-
"subquery. Set this to false to keep such build sides on Spark.")
1377+
"subquery. Parquet, JSON, CSV and Text scans are converted only by their own " +
1378+
"`spark.comet.convert` configs. Set this to false to keep such build sides on Spark.")
13651379
.booleanConf
13661380
.createWithDefault(true)
13671381

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,12 @@ import org.apache.spark.sql.execution.datasources.{InsertIntoHadoopFsRelationCom
4545
import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
4646
import org.apache.spark.sql.execution.datasources.json.JsonFileFormat
4747
import org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
48+
import org.apache.spark.sql.execution.datasources.text.TextFileFormat
4849
import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, V2CommandExec}
4950
import org.apache.spark.sql.execution.datasources.v2.csv.CSVScan
5051
import org.apache.spark.sql.execution.datasources.v2.json.JsonScan
5152
import org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScan
53+
import org.apache.spark.sql.execution.datasources.v2.text.TextScan
5254
import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, BroadcastExchangeLike, Exchange, ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike}
5355
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, ShuffledHashJoinExec, SortMergeJoinExec}
5456
import org.apache.spark.sql.execution.window.WindowExec
@@ -193,6 +195,15 @@ object CometExecRule {
193195
"Spark to Arrow conversion is not compatible with input_file_name, " +
194196
"input_file_block_start, or input_file_block_length"
195197

198+
/**
199+
* Why Comet does not bridge a Text scan on a broadcast build side. A Text file can hold bytes
200+
* that are not valid UTF-8, and the conversion replaces them with U+FFFD. See issue #7118.
201+
*/
202+
private val TEXT_BUILD_SIDE_FALLBACK_REASON: String =
203+
"Comet converts a Text scan on a broadcast build side to Arrow only when " +
204+
s"${CometConf.COMET_CONVERT_FROM_TEXT_ENABLED.key}=true, because a Text " +
205+
"file can hold bytes that are not valid UTF-8"
206+
196207
/**
197208
* Tag set on a scan leaf that sits on a broadcast join's build side. When
198209
* `spark.comet.convert.broadcastBuildSide.enabled` is on, such a scan is bridged to Arrow with
@@ -990,7 +1001,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
9901001
tagUnsafePartialAggregates(planWithJoinRewritten)
9911002

9921003
// Tag leaves on broadcast join build sides so shouldApplySparkToColumnar can bridge an
993-
// unsupported build-side scan (e.g. Text) to Arrow and let the join run natively (#6008).
1004+
// unsupported build-side scan (e.g. ORC) to Arrow and let the join run natively (#6008).
9941005
tagBroadcastBuildSideLeaves(planWithJoinRewritten)
9951006

9961007
var transformed = transform(planWithJoinRewritten)
@@ -1462,8 +1473,9 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
14621473
case _: CSVFileFormat => CometConf.COMET_CONVERT_FROM_CSV_ENABLED.get(conf)
14631474
case _: JsonFileFormat => CometConf.COMET_CONVERT_FROM_JSON_ENABLED.get(conf)
14641475
case _: ParquetFileFormat => CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.get(conf)
1465-
// Unsupported format (e.g. Text): bridge it if the general opt-in is on or it feeds a
1466-
// broadcast build side. The per-format arms above keep honoring their own opt-outs.
1476+
case _: TextFileFormat => shouldConvertTextScan(conf, op)
1477+
// Unsupported format (e.g. ORC): bridge it if the general opt-in is on or it feeds a
1478+
// broadcast build side. The per-format arms above keep honoring their own switches.
14671479
case _ => shouldBridgeUnsupportedScan(conf, op)
14681480
}
14691481
// Convert Spark DS v2 scan to Arrow format
@@ -1472,6 +1484,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
14721484
case _: CSVScan => CometConf.COMET_CONVERT_FROM_CSV_ENABLED.get(conf)
14731485
case _: JsonScan => CometConf.COMET_CONVERT_FROM_JSON_ENABLED.get(conf)
14741486
case _: ParquetScan => CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.get(conf)
1487+
case _: TextScan => shouldConvertTextScan(conf, op)
14751488
case _ => shouldBridgeUnsupportedScan(conf, op)
14761489
}
14771490
case _: RangeExec =>
@@ -1519,12 +1532,28 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
15191532
/**
15201533
* An unsupported file-source scan is bridged to Arrow when the general sparkToColumnar opt-in
15211534
* covers it, or when it feeds a broadcast join's build side (see issue #6008). This is only
1522-
* reached from the unsupported-format arms, so the per-format `COMET_CONVERT_FROM_*` opt-outs
1523-
* for CSV/JSON/Parquet still take precedence.
1535+
* reached from the unsupported-format arms, so the per-format `COMET_CONVERT_FROM_*` switches
1536+
* for CSV/JSON/Parquet/Text still take precedence.
15241537
*/
15251538
private def shouldBridgeUnsupportedScan(conf: SQLConf, op: SparkPlan): Boolean =
15261539
isSparkToArrowEnabled(conf, op) || isBroadcastBuildSideLeaf(op)
15271540

1541+
/**
1542+
* A Text scan is converted only by `spark.comet.convert.text.enabled`, like CSV, JSON and
1543+
* Parquet, and so is a Text scan on a broadcast build side: a Text file can hold bytes that are
1544+
* not valid UTF-8, for example a Latin-1 file, and the conversion replaces them with U+FFFD, so
1545+
* results that depend on the original bytes would differ from Spark. A Text build side that the
1546+
* bridge would otherwise convert records why it did not, so the explain output names the
1547+
* config. See issue #7118.
1548+
*/
1549+
private def shouldConvertTextScan(conf: SQLConf, op: SparkPlan): Boolean =
1550+
isConversionEnabled(conf, op, CometConf.COMET_CONVERT_FROM_TEXT_ENABLED) || {
1551+
if (isBroadcastBuildSideLeaf(op)) {
1552+
withFallbackReason(op, CometExecRule.TEXT_BUILD_SIDE_FALLBACK_REASON)
1553+
}
1554+
false
1555+
}
1556+
15281557
/**
15291558
* True when the build-side auto-bridge should run: its own config is on and Comet broadcast
15301559
* exchange conversion is enabled. Without the latter the `BroadcastExchangeExec` cannot become
@@ -1553,7 +1582,7 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
15531582

15541583
/**
15551584
* Tag the file-source scan leaves on a broadcast join's build side so
1556-
* shouldApplySparkToColumnar can bridge an unsupported build-side scan (e.g. a Text scan) to
1585+
* shouldApplySparkToColumnar can bridge an unsupported build-side scan (e.g. an ORC scan) to
15571586
* Arrow, letting the whole join run natively. The build side is usually small (auto-broadcasts
15581587
* are capped by the broadcast threshold), so the row to Arrow copy is usually cheap - but that
15591588
* is a heuristic, not a bound: an explicit BROADCAST hint can force a larger build side, and a
@@ -1683,10 +1712,11 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
16831712
* Two guards keep the walk honest. It stops at a `CometBroadcastExchangeExec` - a nested
16841713
* broadcast join that went native keeps its bridge. And it only clears a tag (and so asks for a
16851714
* second transform) when the tagged scan sits under a `CometSparkToColumnarExec`, i.e. it was
1686-
* actually bridged. A CSV/JSON/Parquet scan whose per-format arm left it on Spark still carries
1687-
* the tag but was never bridged, so touching it would force a pointless second transform. The
1688-
* descent enters `CometScanWrapper`'s wrapped `originalPlan` by hand because it is a leaf, so a
1689-
* plain `foreach` would never reach the bridged scan inside it. See issue #6008.
1715+
* actually bridged. A CSV/JSON/Parquet/Text scan whose per-format arm left it on Spark still
1716+
* carries the tag but was never bridged, so touching it would force a pointless second
1717+
* transform. The descent enters `CometScanWrapper`'s wrapped `originalPlan` by hand because it
1718+
* is a leaf, so a plain `foreach` would never reach the bridged scan inside it. See issue
1719+
* #6008.
16901720
*/
16911721
private def clearWastedBuildSideBridges(plan: SparkPlan): Boolean = {
16921722
var cleared = false

0 commit comments

Comments
 (0)