Skip to content

Commit fb126fd

Browse files
committed
docs: document value-identity divergence and pin the decoder behavior in a Spark test
Addresses further review feedback on the CAST(binary AS string) UB fix: - Add an end-to-end CometCastSuite test over known invalid byte sequences (two invalid bytes, the surrogate-range collapse, a truncated lead) that checks Comet against Spark and pins the U+FFFD output, so a change in decode_utf8_spark_lossy is caught at the Spark level. - Document that decoding is not byte-preserving in a second way: distinct ill-formed byte sequences all decode to U+FFFD, so strings that differ in Spark can compare equal in Comet, which can affect equality, joins, grouping, ordering, and byte-based functions such as contains. Previously only the round-trip divergence was called out. - Fix the contributor-guide reference to the decoder, which now lives in datafusion-comet-common rather than the spark-expr utils module.
1 parent bdf9491 commit fb126fd

5 files changed

Lines changed: 65 additions & 14 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -486,12 +486,14 @@ must therefore **decode** them, not reinterpret them. Building a Rust `String`/`
486486
bytes is undefined behaviour, and an Arrow string array that holds invalid UTF-8 is unsound for the
487487
downstream kernels that read it via `StringArray::value` (which assumes the UTF-8 invariant).
488488

489-
Use `crate::utils::decode_utf8_spark_lossy`. It replaces each ill-formed sequence with `U+FFFD`
490-
exactly as the JVM's `new String(bytes, UTF_8)` does, so Comet's rendered output matches Spark
491-
(including the surrogate-range cases where `str::from_utf8_lossy` would differ). This is the decoder
492-
used by both `CAST(binary AS string)` and the native columnar shuffle. See the
489+
Use `datafusion_comet_common::decode_utf8_spark_lossy` (defined in `native/common/src/utils.rs`). It
490+
replaces each ill-formed sequence with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does,
491+
so Comet's rendered output matches Spark (including the surrogate-range cases where
492+
`str::from_utf8_lossy` would differ). This is the decoder used by both `CAST(binary AS string)` and
493+
the native columnar shuffle. See the
493494
[strings with non-UTF-8 bytes](../user-guide/latest/compatibility/index.md) note in the
494-
compatibility guide for the user-visible behavior.
495+
compatibility guide for the user-visible behavior, including the cases where decoding diverges from
496+
Spark.
495497

496498
### API Differences Between Spark Versions
497499

‎docs/source/contributor-guide/expression-audits/conversion_funcs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
- Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
2929
- Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
3030
- Known divergences and gaps:
31-
- `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output and diverges only under byte-level round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`.
31+
- `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`, and value identity, since distinct ill-formed byte sequences all decode to `U+FFFD` and can therefore compare equal in Comet where Spark compares the raw bytes (https://github.com/apache/datafusion-comet/issues/4764).
3232
- Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).
3333
- Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).
3434
- `CAST(<map> AS <map>)` falls back to Spark even though native `cast_map_to_map` exists (https://github.com/apache/datafusion-comet/issues/4491).

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,10 +82,22 @@ character `U+FFFD`. Spark itself applies the identical replacement whenever such
8282
materialized (collected, printed, or passed to most string functions), so the rendered result
8383
matches Spark.
8484

85-
The one observable difference is byte-preserving round-trips: Spark keeps the original bytes, so
86-
`CAST(CAST(X'FF' AS STRING) AS BINARY)` returns `X'FF'`, whereas Comet returns the UTF-8 encoding of
87-
`U+FFFD` (`X'EFBFBD'`). Operations that inspect the raw bytes (re-casting to binary, `octet_length`,
88-
hashing) can therefore differ for non-UTF-8 input.
85+
Decoding is not byte-preserving, so results can differ from Spark for any operation that works on the
86+
underlying bytes rather than on the rendered text:
87+
88+
- **Round-trips.** Spark keeps the original bytes, so `CAST(CAST(X'FF' AS STRING) AS BINARY)` returns
89+
`X'FF'`, whereas Comet returns the UTF-8 encoding of `U+FFFD` (`X'EFBFBD'`). `octet_length` and
90+
hashing of such a string differ for the same reason.
91+
- **Value identity.** Decoding maps every ill-formed sequence onto the same `U+FFFD`, so two Spark
92+
strings that hold different bytes can become equal in Comet. For example, with `b = X'FF'`,
93+
`CAST(b AS STRING) = CAST(X'EFBFBD' AS STRING)` is `false` in Spark (`UTF8String` compares the raw
94+
bytes) but `true` in Comet. Equality, joins, grouping, ordering, and byte-based string functions
95+
such as `contains` can therefore disagree with Spark when non-UTF-8 bytes are involved.
96+
97+
Both differences require string data that is not valid UTF-8, which does not occur for text read from
98+
Parquet or produced by string expressions. Consistent handling of invalid UTF-8 across all native
99+
string paths is tracked by
100+
[#4764](https://github.com/apache/datafusion-comet/issues/4764).
89101

90102
Separately, Comet's native Parquet scan currently rejects string columns whose stored bytes are not
91103
valid UTF-8 rather than reading them like Spark

‎native/spark-expr/src/conversion_funcs/cast.rs‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -866,10 +866,13 @@ fn cast_binary_formatter(value: &[u8]) -> Cow<'_, str> {
866866
// `decode_utf8_spark_lossy` replaces ill-formed sequences with U+FFFD exactly as Spark's
867867
// `new String(bytes, UTF_8)` does (the same decoder Comet's native shuffle uses, #4521). The
868868
// result is memory-safe valid UTF-8, never feeds invalid bytes into downstream native string
869-
// kernels, and matches Spark's rendered output byte-for-byte. It diverges from Spark only under
870-
// byte-level round-trips such as CAST(CAST(x AS string) AS binary), where Spark still has the
871-
// original bytes and Comet has the U+FFFD replacements. Returning `Cow` lets the valid path
872-
// borrow so the caller appends without an intermediate allocation.
869+
// kernels, and matches Spark's rendered output byte-for-byte. It diverges from Spark for
870+
// operations that read the underlying bytes rather than the rendered text: byte-level round-trips
871+
// such as CAST(CAST(x AS string) AS binary) (Spark still has the original bytes), and value
872+
// identity, since every ill-formed sequence decodes to the same U+FFFD and so two strings that
873+
// differ in Spark can compare equal in Comet. Both are documented in the compatibility guide and
874+
// tracked by #4764. Returning `Cow` lets the valid path borrow so the caller appends without an
875+
// intermediate allocation.
873876
decode_utf8_spark_lossy(value)
874877
}
875878

‎spark/src/test/scala/org/apache/comet/CometCastSuite.scala‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
package org.apache.comet
2121

2222
import java.io.File
23+
import java.nio.charset.StandardCharsets
2324

2425
import scala.collection.mutable.ListBuffer
2526
import scala.jdk.CollectionConverters._
@@ -1310,6 +1311,39 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
13101311
castTest(gen.generateStrings(dataSize, numericPattern, 8).toDF("a"), DataTypes.StringType)
13111312
}
13121313

1314+
test("cast BinaryType to StringType - invalid UTF-8 bytes are replaced") {
1315+
// Arrow strings must be valid UTF-8, so Comet decodes the bytes the way the JVM's
1316+
// `new String(bytes, UTF_8)` does, replacing each ill-formed sequence with U+FFFD (#4488).
1317+
// Spark applies the same replacement when the string is materialized, so the results match,
1318+
// but pin the replacement explicitly here so a change in the native decoder is caught.
1319+
val values: Seq[Array[Byte]] = Seq(
1320+
"abc".getBytes(StandardCharsets.UTF_8), // valid, preserved as is
1321+
Array[Byte](0xff.toByte, 0xfe.toByte), // two invalid bytes
1322+
Array[Byte](0xed.toByte, 0xa0.toByte, 0x80.toByte), // surrogate range, one replacement
1323+
Array[Byte]('a'.toByte, 0xc3.toByte), // truncated two-byte sequence
1324+
Array.emptyByteArray)
1325+
castTest(values.toDF("a"), DataTypes.StringType)
1326+
1327+
withTempPath { dir =>
1328+
// Carry an explicit index so the rows can be put back in input order after the round-trip.
1329+
val indexed = values.zipWithIndex.toDF("a", "i")
1330+
val df = roundtripParquet(indexed, dir)
1331+
.coalesce(1)
1332+
.orderBy(col("i"))
1333+
.select(col("a").cast(DataTypes.StringType).as("casted"))
1334+
checkSparkAnswerAndOperator(df)
1335+
1336+
val actual = df.collect().map(_.getString(0))
1337+
// The JVM decoder is the oracle: Comet must produce the same string Spark renders.
1338+
assert(actual.toSeq === values.map(new String(_, StandardCharsets.UTF_8)))
1339+
assert(actual(0) === "abc")
1340+
assert(actual(1) === "\uFFFD\uFFFD")
1341+
assert(actual(2) === "\uFFFD")
1342+
assert(actual(3) === "a\uFFFD")
1343+
assert(actual(4) === "")
1344+
}
1345+
}
1346+
13131347
// CAST from DateType
13141348

13151349
// Date to Boolean/Byte/Short/Int/Long/Float/Double/Decimal casts always return NULL

0 commit comments

Comments
 (0)