Skip to content

Commit a3a135a

Browse files
viiryaviirya
committed
[SPARK-60118][SQL] Fix Int overflow in the fast-fail check of casting string to decimal
### What changes were proposed in this pull request? `Decimal.fromString` and `Decimal.fromStringANSI` fast-fail on strings whose number of integral digits exceeds `DecimalType.MAX_PRECISION`, computed as `precision - scale` of the parsed `java.math.BigDecimal`. This PR: - computes that value as a `Long` instead of an `Int`; - skips the check for zero, which fits any decimal type whatever its exponent, and drops a negative scale of zero first so that the result has precision 1 and scale 0 instead of a precision derived from the exponent; - moves the check into one helper shared by `fromString` and `fromStringANSI`. ### Why are the changes needed? For strings with an exponent near `Int.MaxValue`, e.g. `'1e2147483647'` (precision 1, scale -2147483647), `precision - scale` overflows `Int` to a negative number and the fast-fail check is skipped. The subsequent conversion then fails with a raw `java.lang.ArithmeticException` (`BigInteger would overflow supported range`, or `Underflow` on JDK 21+ where `'1e2147483648'` parses), so `try_cast`, legacy `CAST` and ANSI `CAST` all fail the task instead of returning NULL or raising `NUMERIC_OUT_OF_SUPPORTED_RANGE`. The check also counts integral digits for zero, so e.g. `'0e39'` is rejected as out of range although it fits any decimal type, while `'0e37'` returns `0.00`. Strings far below the target scale (e.g. `'1e-2147483647'`) and huge exponents under `spark.sql.legacy.allowNegativeScaleOfDecimal=true` fail later, in `Decimal.changePrecision`; they are fixed separately in SPARK-60119 (#59335). Other `BigDecimal` entry points of `Decimal` are tracked in SPARK-60132. ### Does this PR introduce _any_ user-facing change? Yes. Casting such strings to decimal now returns NULL (`try_cast` / non-ANSI `CAST`) or raises `NUMERIC_OUT_OF_SUPPORTED_RANGE` (ANSI `CAST`), the same as other out-of-range strings such as `'6E+38'`, instead of failing with `java.lang.ArithmeticException`. Zero with a large exponent, e.g. `'0e39'`, now returns `0` (precision 1, scale 0) instead of NULL / `NUMERIC_OUT_OF_SUPPORTED_RANGE`. ### How was this patch tested? New test cases in `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite` and `TryCastSuite` (inherited), covering interpreted and codegen evaluation. They fail without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59334 from viirya/SPARK-60118. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 79ab768) Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent cde5182 commit a3a135a

5 files changed

Lines changed: 83 additions & 17 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 32 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -623,26 +623,45 @@ object Decimal {
623623
}
624624
}
625625

626-
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Int =
627-
bigDecimal.precision - bigDecimal.scale
626+
// Computed in Long since `precision - scale` can overflow Int, e.g. for "1e2147483647".
627+
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Long =
628+
bigDecimal.precision.toLong - bigDecimal.scale
628629

629630
private def stringToJavaBigDecimal(str: UTF8String): JavaBigDecimal = {
630631
// According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
631632
// Please refer to https://github.com/apache/spark/pull/26640
632633
new JavaBigDecimal(str.toString.trim)
633634
}
634635

635-
def fromString(str: UTF8String): Decimal = {
636-
try {
637-
val bigDecimal = stringToJavaBigDecimal(str)
638-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
639-
// For example: Decimal("6.0790316E+25569151")
640-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
641-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
642-
null
636+
/**
637+
* Converts a `JavaBigDecimal` parsed from a string to `Decimal`, or returns null if its
638+
* integral part has more than `DecimalType.MAX_PRECISION` digits and negative scales are not
639+
* allowed. We fast fail because constructing a very large JavaBigDecimal to Decimal is very
640+
* slow. For example: Decimal("6.0790316E+25569151")
641+
*
642+
* This runs for every row of a cast, so this helper reads the conf only for a zero with a
643+
* negative scale and for a value with too many integral digits.
644+
*/
645+
private def stringBigDecimalToDecimal(bigDecimal: JavaBigDecimal): Decimal = {
646+
if (bigDecimal.signum == 0) {
647+
// Zero fits any decimal type whatever its exponent, e.g. "0e39". Drop a negative scale so
648+
// that the precision does not grow with a positive exponent.
649+
if (bigDecimal.scale < 0 && !SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
650+
Decimal(bigDecimal.setScale(0))
643651
} else {
644652
Decimal(bigDecimal)
645653
}
654+
} else if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
655+
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
656+
null
657+
} else {
658+
Decimal(bigDecimal)
659+
}
660+
}
661+
662+
def fromString(str: UTF8String): Decimal = {
663+
try {
664+
stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
646665
} catch {
647666
case _: NumberFormatException =>
648667
null
@@ -654,15 +673,11 @@ object Decimal {
654673
to: DecimalType = DecimalType.USER_DEFAULT,
655674
context: QueryContext = null): Decimal = {
656675
try {
657-
val bigDecimal = stringToJavaBigDecimal(str)
658-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
659-
// For example: Decimal("6.0790316E+25569151")
660-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
661-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
676+
val decimal = stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
677+
if (decimal == null) {
662678
throw DataTypeErrors.outOfDecimalTypeRangeError(str)
663-
} else {
664-
Decimal(bigDecimal)
665679
}
680+
decimal
666681
} catch {
667682
case _: NumberFormatException =>
668683
throw DataTypeErrors.invalidInputInCastToNumberError(to, str, context)

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
293293
assert(cast("abcdef", FloatType).nullable)
294294
}
295295

296+
test("SPARK-60118: cast string of zero with a large exponent to decimal") {
297+
Seq("0e38", "0e39", "-0E+100", "0e2147483647").foreach { s =>
298+
checkEvaluation(cast(s, DecimalType(10, 2)), Decimal("0.00"))
299+
}
300+
}
301+
296302
test("cast from timestamp") {
297303
val millis = 15 * 1000 + 3
298304
val seconds = millis * 1000 + 3

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -547,6 +547,9 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
547547
Decimal("60000000000000000000000000000000000000"))
548548
checkEvaluation(cast("6E+38", DecimalType(38, 0)), null)
549549
checkEvaluation(cast("6E+37", DecimalType(38, 1)), null)
550+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
551+
checkEvaluation(cast(s, DecimalType(10, 2)), null)
552+
}
550553

551554
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
552555
}

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,11 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
454454
checkExceptionInExpression[ArithmeticException](
455455
cast("6E+37", DecimalType(38, 1)),
456456
"cannot be represented as Decimal(38, 1)")
457+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
458+
checkExceptionInExpression[ArithmeticException](
459+
cast(s, DecimalType(10, 2)),
460+
"NUMERIC_OUT_OF_SUPPORTED_RANGE")
461+
}
457462

458463
checkExceptionInExpression[NumberFormatException](
459464
cast("abcd", DecimalType(38, 1)),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,43 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
418418
}
419419
}
420420

421+
test("SPARK-60118: UTF8String to Decimal with exponent near Int.MaxValue") {
422+
// The number of integral digits (precision - scale) of these values overflows Int.
423+
Seq("1e2147483647", "-1E+2147483647", "12e2147483647",
424+
"1.5e2147483647").foreach { string =>
425+
assert(Decimal.fromString(UTF8String.fromString(string)) === null)
426+
checkError(
427+
exception = intercept[SparkArithmeticException](
428+
Decimal.fromStringANSI(UTF8String.fromString(string))),
429+
condition = "NUMERIC_OUT_OF_SUPPORTED_RANGE",
430+
parameters = Map("value" -> string))
431+
}
432+
// Zero fits any decimal type whatever its exponent. "0e38" is the smallest exponent that
433+
// used to be rejected, and "0e37" used to get precision 38.
434+
Seq("0e37", "0e38", "0e39", "-0E+100", "0e2147483647").foreach { string =>
435+
Seq(Decimal.fromString(UTF8String.fromString(string)),
436+
Decimal.fromStringANSI(UTF8String.fromString(string))).foreach { d =>
437+
assert(d === Decimal(0))
438+
assert(d.precision === 1)
439+
assert(d.scale === 0)
440+
}
441+
}
442+
Seq(Decimal.fromString(UTF8String.fromString("0.00")),
443+
Decimal.fromStringANSI(UTF8String.fromString("0.00"))).foreach { d =>
444+
assert(d.precision === 2)
445+
assert(d.scale === 2)
446+
}
447+
// With negative scales allowed, zero keeps its negative scale.
448+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
449+
Seq(Decimal.fromString(UTF8String.fromString("0e39")),
450+
Decimal.fromStringANSI(UTF8String.fromString("0e39"))).foreach { d =>
451+
assert(d === Decimal(0))
452+
assert(d.precision === 1)
453+
assert(d.scale === -39)
454+
}
455+
}
456+
}
457+
421458
// 18 is a max number of digits in Decimal's compact long
422459
test("SPARK-41554: decrease/increase scale by 18 and more on compact decimal") {
423460
val unscaledNums = Seq(

0 commit comments

Comments
 (0)