Skip to content

Commit 5e43899

Browse files
committed
[SPARK-60118][SQL] Fix Int overflow in the fast-fail check of casting string to decimal
### What changes were proposed in this pull request? `Decimal.fromString` and `Decimal.fromStringANSI` fast-fail on strings whose number of integral digits exceeds `DecimalType.MAX_PRECISION`, computed as `precision - scale` of the parsed `java.math.BigDecimal`. This PR: - computes that value as a `Long` instead of an `Int`; - skips the check for zero, which fits any decimal type whatever its exponent, and drops a negative scale of zero first so that the result has precision 1 and scale 0 instead of a precision derived from the exponent; - moves the check into one helper shared by `fromString` and `fromStringANSI`. ### Why are the changes needed? For strings with an exponent near `Int.MaxValue`, e.g. `'1e2147483647'` (precision 1, scale -2147483647), `precision - scale` overflows `Int` to a negative number and the fast-fail check is skipped. The subsequent conversion then fails with a raw `java.lang.ArithmeticException` (`BigInteger would overflow supported range`, or `Underflow` on JDK 21+ where `'1e2147483648'` parses), so `try_cast`, legacy `CAST` and ANSI `CAST` all fail the task instead of returning NULL or raising `NUMERIC_OUT_OF_SUPPORTED_RANGE`. The check also counts integral digits for zero, so e.g. `'0e39'` is rejected as out of range although it fits any decimal type, while `'0e37'` returns `0.00`. Strings far below the target scale (e.g. `'1e-2147483647'`) and huge exponents under `spark.sql.legacy.allowNegativeScaleOfDecimal=true` fail later, in `Decimal.changePrecision`; they are fixed separately in SPARK-60119 (#59335). Other `BigDecimal` entry points of `Decimal` are tracked in SPARK-60132. ### Does this PR introduce _any_ user-facing change? Yes. Casting such strings to decimal now returns NULL (`try_cast` / non-ANSI `CAST`) or raises `NUMERIC_OUT_OF_SUPPORTED_RANGE` (ANSI `CAST`), the same as other out-of-range strings such as `'6E+38'`, instead of failing with `java.lang.ArithmeticException`. Zero with a large exponent, e.g. `'0e39'`, now returns `0` (precision 1, scale 0) instead of NULL / `NUMERIC_OUT_OF_SUPPORTED_RANGE`. ### How was this patch tested? New test cases in `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite` and `TryCastSuite` (inherited), covering interpreted and codegen evaluation. They fail without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59334 from viirya/SPARK-60118. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 79ab768) Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent 15ccb4b commit 5e43899

5 files changed

Lines changed: 83 additions & 17 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 32 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -632,26 +632,45 @@ object Decimal {
632632
}
633633
}
634634

635-
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Int =
636-
bigDecimal.precision - bigDecimal.scale
635+
// Computed in Long since `precision - scale` can overflow Int, e.g. for "1e2147483647".
636+
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Long =
637+
bigDecimal.precision.toLong - bigDecimal.scale
637638

638639
private def stringToJavaBigDecimal(str: UTF8String): JavaBigDecimal = {
639640
// According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
640641
// Please refer to https://github.com/apache/spark/pull/26640
641642
new JavaBigDecimal(str.toString.trim)
642643
}
643644

644-
def fromString(str: UTF8String): Decimal = {
645-
try {
646-
val bigDecimal = stringToJavaBigDecimal(str)
647-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
648-
// For example: Decimal("6.0790316E+25569151")
649-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
650-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
651-
null
645+
/**
646+
* Converts a `JavaBigDecimal` parsed from a string to `Decimal`, or returns null if its
647+
* integral part has more than `DecimalType.MAX_PRECISION` digits and negative scales are not
648+
* allowed. We fast fail because constructing a very large JavaBigDecimal to Decimal is very
649+
* slow. For example: Decimal("6.0790316E+25569151")
650+
*
651+
* This runs for every row of a cast, so this helper reads the conf only for a zero with a
652+
* negative scale and for a value with too many integral digits.
653+
*/
654+
private def stringBigDecimalToDecimal(bigDecimal: JavaBigDecimal): Decimal = {
655+
if (bigDecimal.signum == 0) {
656+
// Zero fits any decimal type whatever its exponent, e.g. "0e39". Drop a negative scale so
657+
// that the precision does not grow with a positive exponent.
658+
if (bigDecimal.scale < 0 && !SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
659+
Decimal(bigDecimal.setScale(0))
652660
} else {
653661
Decimal(bigDecimal)
654662
}
663+
} else if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
664+
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
665+
null
666+
} else {
667+
Decimal(bigDecimal)
668+
}
669+
}
670+
671+
def fromString(str: UTF8String): Decimal = {
672+
try {
673+
stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
655674
} catch {
656675
case _: NumberFormatException =>
657676
null
@@ -663,15 +682,11 @@ object Decimal {
663682
to: DecimalType = DecimalType.USER_DEFAULT,
664683
context: QueryContext = null): Decimal = {
665684
try {
666-
val bigDecimal = stringToJavaBigDecimal(str)
667-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
668-
// For example: Decimal("6.0790316E+25569151")
669-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
670-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
685+
val decimal = stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
686+
if (decimal == null) {
671687
throw DataTypeErrors.outOfDecimalTypeRangeError(str)
672-
} else {
673-
Decimal(bigDecimal)
674688
}
689+
decimal
675690
} catch {
676691
case _: NumberFormatException =>
677692
throw DataTypeErrors.invalidInputInCastToNumberError(to, str, context)

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
330330
assert(cast("abcdef", FloatType).nullable)
331331
}
332332

333+
test("SPARK-60118: cast string of zero with a large exponent to decimal") {
334+
Seq("0e38", "0e39", "-0E+100", "0e2147483647").foreach { s =>
335+
checkEvaluation(cast(s, DecimalType(10, 2)), Decimal("0.00"))
336+
}
337+
}
338+
333339
test("cast from timestamp") {
334340
val millis = 15 * 1000 + 3
335341
val seconds = millis * 1000 + 3

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -559,6 +559,9 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
559559
Decimal("60000000000000000000000000000000000000"))
560560
checkEvaluation(cast("6E+38", DecimalType(38, 0)), null)
561561
checkEvaluation(cast("6E+37", DecimalType(38, 1)), null)
562+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
563+
checkEvaluation(cast(s, DecimalType(10, 2)), null)
564+
}
562565

563566
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
564567
}

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,11 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
475475
checkExceptionInExpression[ArithmeticException](
476476
cast("6E+37", DecimalType(38, 1)),
477477
"cannot be represented as Decimal(38, 1)")
478+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
479+
checkExceptionInExpression[ArithmeticException](
480+
cast(s, DecimalType(10, 2)),
481+
"NUMERIC_OUT_OF_SUPPORTED_RANGE")
482+
}
478483

479484
checkExceptionInExpression[NumberFormatException](
480485
cast("abcd", DecimalType(38, 1)),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,43 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
445445
}
446446
}
447447

448+
test("SPARK-60118: UTF8String to Decimal with exponent near Int.MaxValue") {
449+
// The number of integral digits (precision - scale) of these values overflows Int.
450+
Seq("1e2147483647", "-1E+2147483647", "12e2147483647",
451+
"1.5e2147483647").foreach { string =>
452+
assert(Decimal.fromString(UTF8String.fromString(string)) === null)
453+
checkError(
454+
exception = intercept[SparkArithmeticException](
455+
Decimal.fromStringANSI(UTF8String.fromString(string))),
456+
condition = "NUMERIC_OUT_OF_SUPPORTED_RANGE",
457+
parameters = Map("value" -> string))
458+
}
459+
// Zero fits any decimal type whatever its exponent. "0e38" is the smallest exponent that
460+
// used to be rejected, and "0e37" used to get precision 38.
461+
Seq("0e37", "0e38", "0e39", "-0E+100", "0e2147483647").foreach { string =>
462+
Seq(Decimal.fromString(UTF8String.fromString(string)),
463+
Decimal.fromStringANSI(UTF8String.fromString(string))).foreach { d =>
464+
assert(d === Decimal(0))
465+
assert(d.precision === 1)
466+
assert(d.scale === 0)
467+
}
468+
}
469+
Seq(Decimal.fromString(UTF8String.fromString("0.00")),
470+
Decimal.fromStringANSI(UTF8String.fromString("0.00"))).foreach { d =>
471+
assert(d.precision === 2)
472+
assert(d.scale === 2)
473+
}
474+
// With negative scales allowed, zero keeps its negative scale.
475+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
476+
Seq(Decimal.fromString(UTF8String.fromString("0e39")),
477+
Decimal.fromStringANSI(UTF8String.fromString("0e39"))).foreach { d =>
478+
assert(d === Decimal(0))
479+
assert(d.precision === 1)
480+
assert(d.scale === -39)
481+
}
482+
}
483+
}
484+
448485
// 18 is a max number of digits in Decimal's compact long
449486
test("SPARK-41554: decrease/increase scale by 18 and more on compact decimal") {
450487
val unscaledNums = Seq(

0 commit comments

Comments
 (0)