Skip to content

Commit 1755de3

Browse files
committed
fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test
Review follow-up. - On the released path, parse_string_to_decimal counts a zero's integral digits as 1 - scale with a wrapping i32 subtraction, as Spark's int arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers it, and two inputs whose counts don't wrap. - The sequence overflow test's released-Spark branch uses checkSparkError, which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather than the message. - The cast audit entry describes SPARK-60118's long digit count: released Spark throws for '1e2147483647' and similar strings, the fixed releases return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's ANSI error is #7188. - SparkFixes' scaladoc names the two fixes it covers and points to the other version checks.
1 parent 233f895 commit 1755de3

4 files changed

Lines changed: 21 additions & 15 deletions

File tree

‎docs/source/contributor-guide/expression-audits/conversion_funcs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
- Spark 3.5.8 (audited 2026-05-27): baseline. `Cast(child, dataType, timeZoneId, evalMode)`; eval modes are `LEGACY`, `ANSI`, `TRY`. The legacy `Cast.canCast` matrix and the `Cast.canAnsiCast` matrix decide acceptance per type pair. Comet routes via `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala`) using a per-source-type support matrix that returns `Compatible`, `Incompatible(reason)`, or `Unsupported(reason)`; literal children are short-circuited to `Compatible()` so `CometLiteral` validates them. The serialized `Cast` proto carries `datatype`, `evalMode`, `timezone` (default `UTC`), and `isSpark4Plus`; `allowIncompatible` is enforced only by the JVM planner and is not sent to the native cast kernel. The native side (`native/spark-expr/src/conversion_funcs/cast.rs`) implements explicit per-eval-mode branches for narrowing numeric casts that match Spark's overflow exceptions, and falls through to DataFusion `cast_with_options(safe = !ANSI)` for the rest.
2828
- Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
2929
- Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
30-
- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates; in 4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3): `Decimal.fromString` and `fromStringANSI` no longer count integral digits for zero (SPARK-60118), so a zero with a large exponent such as `'0e39'` casts to 0 instead of NULL (or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI). `CometCast` sets the `decimal_zero_any_exponent` cast option from `SparkFixes.castsZeroWithAnyExponentToDecimal`, and `parse_string_to_decimal` skips the zero limit when it is set. The same releases skip `BigDecimal.setScale` in `Decimal.changePrecision` when the value's magnitude already decides the result (SPARK-60119), so a string far below the target scale's ulp casts to the rounded value instead of throwing `ArithmeticException`. `'1e-2147483647'` to `DECIMAL(10,2)` gives `0.00`, which the native cast already returns, so that case of [#6966](https://github.com/apache/datafusion-comet/issues/6966) applies only to earlier releases. Its negative-scale case (`spark.sql.legacy.allowNegativeScaleOfDecimal=true`) still differs: `'1e-2147483647'` to `DECIMAL(10,-2)` gives 0 on these releases, while the native cast raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise.
30+
- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates; in 4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3): `Decimal.fromString` and `fromStringANSI` no longer count integral digits for zero (SPARK-60118), so a zero with a large exponent such as `'0e39'` casts to 0 instead of NULL (or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI). `CometCast` sets the `decimal_zero_any_exponent` cast option from `SparkFixes.castsZeroWithAnyExponentToDecimal`, and `parse_string_to_decimal` skips the zero limit when it is set. The same fix counts the integral digits in a long. Released Spark counts them in an int, which wraps for an exponent near the int limit, so `'1e2147483647'`, `'12e2147483647'` and `'1.5e2147483647'` skip the limit and throw a raw `ArithmeticException` in every eval mode, and `'0e2147483647'` casts to 0. On the fixed releases the first three return NULL, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI. The native cast returns NULL for them outside ANSI, as the fixed releases do, and raises `CAST_INVALID_INPUT` under ANSI ([#7188](https://github.com/apache/datafusion-comet/issues/7188)). Without `decimal_zero_any_exponent` it lets `'0e2147483647'` through as released Spark does. The same releases skip `BigDecimal.setScale` in `Decimal.changePrecision` when the value's magnitude already decides the result (SPARK-60119), so a string far below the target scale's ulp casts to the rounded value instead of throwing `ArithmeticException`. `'1e-2147483647'` to `DECIMAL(10,2)` gives `0.00`, which the native cast already returns, so that case of [#6966](https://github.com/apache/datafusion-comet/issues/6966) applies only to earlier releases. Its negative-scale case (`spark.sql.legacy.allowNegativeScaleOfDecimal=true`) still differs: `'1e-2147483647'` to `DECIMAL(10,-2)` gives 0 on these releases, while the native cast raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise.
3131
- Known divergences and gaps:
3232
- `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`, and value identity, since distinct ill-formed byte sequences all decode to `U+FFFD` and can therefore compare equal in Comet where Spark compares the raw bytes ([#4764](https://github.com/apache/datafusion-comet/issues/4764)).
3333
- Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test ([#4489](https://github.com/apache/datafusion-comet/issues/4489); umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)).

‎native/spark-expr/src/conversion_funcs/string.rs‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -710,9 +710,11 @@ fn parse_string_to_decimal(
710710

711711
// Early return mantissa 0, Spark checks if it fits digits and throw error in ansi. Spark
712712
// counts precision - scale integral digits for zero too, so "0e38" (scale -38) has 39 and is
713-
// over the 38-digit limit. Spark stopped applying that limit to zero in SPARK-60118.
713+
// over the 38-digit limit. Spark stopped applying that limit to zero in SPARK-60118. Before
714+
// that it counted in an int, which wraps negative for "0e2147483647" (scale -2147483647), so
715+
// that zero skipped the limit; the wrapping subtraction does the same.
714716
if mantissa == 0 {
715-
if !zero_any_exponent && exponent < -37 {
717+
if !zero_any_exponent && 1i32.wrapping_sub(exponent) > 38 {
716718
return Err(SparkError::NumericOutOfRange {
717719
value: input_str.to_string(),
718720
});
@@ -2773,11 +2775,17 @@ mod tests {
27732775
}
27742776

27752777
/// Spark used to count integral digits for zero, so a zero with an exponent above 37 was out
2776-
/// of range (NULL, or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). Since SPARK-60118 zero casts
2777-
/// to 0 whatever its exponent. The inputs include ones from Comet's cast fuzz tests.
2778+
/// of range (NULL, or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI), except "0e2147483647", whose
2779+
/// count wrapped in Spark's int arithmetic. Since SPARK-60118 zero casts to 0 whatever its
2780+
/// exponent. The inputs include ones from Comet's cast fuzz tests.
27782781
#[test]
27792782
fn test_parse_string_to_decimal_zero_with_large_exponent() {
2780-
for (s, precision, scale) in [("0e37", 10, 2), ("0e-50", 10, 2), ("-0.0e10", 38, 10)] {
2783+
for (s, precision, scale) in [
2784+
("0e37", 10, 2),
2785+
("0e-50", 10, 2),
2786+
("-0.0e10", 38, 10),
2787+
("0e2147483647", 10, 2),
2788+
] {
27812789
for zero_any_exponent in [false, true] {
27822790
assert_eq!(
27832791
parse_string_to_decimal(s, precision, scale, zero_any_exponent).unwrap(),
@@ -2793,6 +2801,8 @@ mod tests {
27932801
("0E2655", 10, 2),
27942802
("+0.E81", 10, 2),
27952803
("0e5887677", 38, 10),
2804+
("0e2147483646", 10, 2),
2805+
("0.0e2147483647", 10, 2),
27962806
] {
27972807
let err = parse_string_to_decimal(s, precision, scale, false).unwrap_err();
27982808
assert!(

‎spark/src/main/scala/org/apache/comet/serde/SparkFixes.scala‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@ package org.apache.comet.serde
2222
import org.apache.spark.sql.comet.util.Utils
2323

2424
/**
25-
* Spark fixes that change a result Comet reproduces natively, and the Spark releases that carry
26-
* them. The fixes leave nothing in the plan for Comet to detect, so the checks go by the version.
25+
* SPARK-58821 and SPARK-60118, which change results Comet reproduces natively, and the Spark
26+
* releases that carry them. The fixes leave nothing in the plan for Comet to detect, so the
27+
* checks go by the version. Other fixes Comet picks by version keep their own checks, such as
28+
* `RegrSparkVersions` and `ArraySetSupport.normalizesFloats`.
2729
*/
2830
private[comet] object SparkFixes {
2931

‎spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1803,13 +1803,7 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
18031803
if (SparkFixes.sequenceRecomputesOverflowedLength(org.apache.spark.SPARK_VERSION)) {
18041804
checkSparkAnswerAndOperator(query)
18051805
} else {
1806-
val df = sql(query)
1807-
checkCometOperators(stripAQEPlan(df.queryExecution.executedPlan))
1808-
val (sparkError, cometError) = checkSparkAnswerMaybeThrows(df)
1809-
Seq("Spark" -> sparkError, "Comet" -> cometError).foreach { case (engine, error) =>
1810-
assert(error.isDefined, s"$engine did not fail")
1811-
assert(error.get.getMessage.contains("Unreachable code reached"), error.get.getMessage)
1812-
}
1806+
checkSparkError(sql(query), "INTERNAL_ERROR")
18131807
}
18141808
}
18151809
}

0 commit comments

Comments
 (0)