Repository navigation
Description
Describe the bug
Spark 4.0 refactored StringDecode from a BinaryExpression to a RuntimeReplaceable whose replacement is StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction). The two new boolean arguments control malformed-character handling: with legacyErrorAction = true, Spark substitutes replacement characters for invalid UTF-8 sequences (matching the Spark 3.x behavior); with legacyErrorAction = false (the default), Spark raises QueryExecutionErrors.malformedCharacterCoding(...).
Comet's Spark 4.0 shim (spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala) destructures the StaticInvoke arguments and discards both flags, then routes through CommonStringExprs.stringDecode which always lowers to Cast(bin, StringType, TRY). The Cast TRY path produces NULL on invalid UTF-8 in all cases. That means:
- Under Spark 4.0 default mode (
legacyErrorAction = false): Spark raises, Comet returns NULL. - Under Spark 4.0 legacy mode (
legacyErrorAction = true): Spark substitutes replacement characters, Comet returns NULL. - Under Spark 3.x: Spark substitutes replacement characters, Comet returns NULL.
Surfaced by the string-expressions audit in #4461.
Steps to reproduce
SET spark.sql.legacy.javaCharsets = true;
SELECT decode(X'FF', 'UTF-8');Spark 3.x: returns ? (Unicode replacement).
Spark 4.0 (legacy mode): same as 3.x.
Spark 4.0 (default mode): raises MALFORMED_CHARACTER_CODING.
Comet: returns NULL in all three cases.
Expected behavior
Honor legacyCharsets / legacyErrorAction when running under Spark 4.0+. At minimum, the flags should be propagated through the proto so the native impl can choose between the substitute/throw/null modes.
Additional context
- Shim location:
spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala(andspark-4.1,spark-4.2) - Helper:
CommonStringExprs.stringDecodeinspark/src/main/scala/org/apache/comet/serde/strings.scala - Related: [Bug] decode ignores Spark 4.0 legacyCharsets and legacyErrorAction flags #4465 (decode not surfaced in compatibility docs)
Activity
docs(audit): cross-reference filed follow-up issues from string audit…
chore: simplify decode fix per /simplify review