You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2b1c6a3
Browse filesBrowse the repository at this point in the historyBrowse files
fix: keep Text build sides on Spark by default and add Text conversion configs (#7123)
* fix: default broadcast build side conversion to off and document invalid UTF-8 cases
Converting a Text scan on a broadcast build side to Arrow decodes invalid
UTF-8 to U+FFFD, so 1.2.0 could return different results from 1.1.0. Default
spark.comet.convert.broadcastBuildSide.enabled to false and name the
Text-source and CAST(binary AS STRING) cases in the compatibility guide.
* test: move the #6008 build-side tests into CometBroadcastBuildSideSuite
CometBroadcastBuildSideSuite turns the build-side bridge on for every test, so the tests no
longer need the bridgeTest helper, and a test added later cannot miss the setting. The two tests
that failed because they were not wrapped in bridgeTest now run with the bridge on, and the five
that asserted nothing was bridged now exercise the bridge again. The test bodies are unchanged.
* fix: keep Text build sides on Spark by default and add Text conversion configs
Converting a Text scan to Arrow replaces bytes that are not valid UTF-8 with U+FFFD, so
bridging a Text build side by default (#6786) could return different results from 1.1.0
(#7118). The other formats that reach the bridge either define strings as UTF-8 (ORC, Avro,
Iceberg) or are decoded by Spark, so spark.comet.convert.broadcastBuildSide.enabled goes back
to on by default, and Text gets its own configs:
- spark.comet.convert.broadcastBuildSide.text.enabled (default false) bridges a Text build
side. Without it, the scan records a fallback reason that names the config.
- spark.comet.convert.text.enabled (default false) converts every Text scan, like the CSV,
JSON and Parquet configs. The sparkToColumnar operator list still converts Text as before.
The compatibility guide now names Text rather than CSV and JSON, no longer says that Spark
requires Parquet strings to be valid UTF-8, adds Comet columnar shuffle as a path, and gives
the configs that keep min/max over CAST(binary AS STRING) in Spark.
* test: drop the QueryStageExec import that the test move left unused
* fix: convert a Text build side only with spark.comet.convert.text.enabled
Drop spark.comet.convert.broadcastBuildSide.text.enabled. A Text scan, on a
build side or not, is now converted only by spark.comet.convert.text.enabled,
like CSV, JSON and Parquet, and the build-side fallback reason names that
config. Text goes through isConversionEnabled, so naming FileSourceScan or
BatchScan in spark.comet.sparkToColumnar.supportedOperatorList still converts
it but is deprecated.
The #6008 build-side tests read ORC instead of Text, since a global Text switch
would convert their probes and no-bridge assertions too.
* docs: describe Text conversion through its own config only
Update the Text, build-side and compatibility sections for the removal of
spark.comet.convert.broadcastBuildSide.text.enabled, and note in the migration
guide that converting Text through sparkToColumnar is deprecated.
* fix: drop the unused SparkConf import and narrow the min/max workaround
The suite no longer overrides sparkConf, so its SparkConf import failed
Scalafix and the strict-warnings build. Reformat a CometExecRule comment
that Spotless rejected.
The three-config min/max workaround only keeps the decode out of the
query when the cast is the aggregate's own argument. A cast in a
projection below the aggregate runs natively and decodes first, so the
guide now says so and keeps disabling Comet as the general answer.
0 commit comments