Skip to content

Commit 770e31e

Browse files
committed
[SPARK-60118][SQL] Fix Int overflow in the fast-fail check of casting string to decimal
### What changes were proposed in this pull request? `Decimal.fromString` and `Decimal.fromStringANSI` fast-fail on strings whose number of integral digits exceeds `DecimalType.MAX_PRECISION`, computed as `precision - scale` of the parsed `java.math.BigDecimal`. This PR: - computes that value as a `Long` instead of an `Int`; - skips the check for zero, which fits any decimal type whatever its exponent, and drops a negative scale of zero first so that the result has precision 1 and scale 0 instead of a precision derived from the exponent; - moves the check into one helper shared by `fromString` and `fromStringANSI`. ### Why are the changes needed? For strings with an exponent near `Int.MaxValue`, e.g. `'1e2147483647'` (precision 1, scale -2147483647), `precision - scale` overflows `Int` to a negative number and the fast-fail check is skipped. The subsequent conversion then fails with a raw `java.lang.ArithmeticException` (`BigInteger would overflow supported range`, or `Underflow` on JDK 21+ where `'1e2147483648'` parses), so `try_cast`, legacy `CAST` and ANSI `CAST` all fail the task instead of returning NULL or raising `NUMERIC_OUT_OF_SUPPORTED_RANGE`. The check also counts integral digits for zero, so e.g. `'0e39'` is rejected as out of range although it fits any decimal type, while `'0e37'` returns `0.00`. Strings far below the target scale (e.g. `'1e-2147483647'`) and huge exponents under `spark.sql.legacy.allowNegativeScaleOfDecimal=true` fail later, in `Decimal.changePrecision`; they are fixed separately in SPARK-60119 (#59335). Other `BigDecimal` entry points of `Decimal` are tracked in SPARK-60132. ### Does this PR introduce _any_ user-facing change? Yes. Casting such strings to decimal now returns NULL (`try_cast` / non-ANSI `CAST`) or raises `NUMERIC_OUT_OF_SUPPORTED_RANGE` (ANSI `CAST`), the same as other out-of-range strings such as `'6E+38'`, instead of failing with `java.lang.ArithmeticException`. Zero with a large exponent, e.g. `'0e39'`, now returns `0` (precision 1, scale 0) instead of NULL / `NUMERIC_OUT_OF_SUPPORTED_RANGE`. ### How was this patch tested? New test cases in `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite` and `TryCastSuite` (inherited), covering interpreted and codegen evaluation. They fail without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59334 from viirya/SPARK-60118. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]> (cherry picked from commit 79ab768) Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent cb36303 commit 770e31e

5 files changed

Lines changed: 83 additions & 17 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 32 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -635,26 +635,45 @@ object Decimal {
635635
}
636636
}
637637

638-
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Int =
639-
bigDecimal.precision - bigDecimal.scale
638+
// Computed in Long since `precision - scale` can overflow Int, e.g. for "1e2147483647".
639+
private def numDigitsInIntegralPart(bigDecimal: JavaBigDecimal): Long =
640+
bigDecimal.precision.toLong - bigDecimal.scale
640641

641642
private def stringToJavaBigDecimal(str: UTF8String): JavaBigDecimal = {
642643
// According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
643644
// Please refer to https://github.com/apache/spark/pull/26640
644645
new JavaBigDecimal(str.toString.trim)
645646
}
646647

647-
def fromString(str: UTF8String): Decimal = {
648-
try {
649-
val bigDecimal = stringToJavaBigDecimal(str)
650-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
651-
// For example: Decimal("6.0790316E+25569151")
652-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
653-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
654-
null
648+
/**
649+
* Converts a `JavaBigDecimal` parsed from a string to `Decimal`, or returns null if its
650+
* integral part has more than `DecimalType.MAX_PRECISION` digits and negative scales are not
651+
* allowed. We fast fail because constructing a very large JavaBigDecimal to Decimal is very
652+
* slow. For example: Decimal("6.0790316E+25569151")
653+
*
654+
* This runs for every row of a cast, so this helper reads the conf only for a zero with a
655+
* negative scale and for a value with too many integral digits.
656+
*/
657+
private def stringBigDecimalToDecimal(bigDecimal: JavaBigDecimal): Decimal = {
658+
if (bigDecimal.signum == 0) {
659+
// Zero fits any decimal type whatever its exponent, e.g. "0e39". Drop a negative scale so
660+
// that the precision does not grow with a positive exponent.
661+
if (bigDecimal.scale < 0 && !SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
662+
Decimal(bigDecimal.setScale(0))
655663
} else {
656664
Decimal(bigDecimal)
657665
}
666+
} else if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
667+
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
668+
null
669+
} else {
670+
Decimal(bigDecimal)
671+
}
672+
}
673+
674+
def fromString(str: UTF8String): Decimal = {
675+
try {
676+
stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
658677
} catch {
659678
case _: NumberFormatException =>
660679
null
@@ -666,15 +685,11 @@ object Decimal {
666685
to: DecimalType = DecimalType.USER_DEFAULT,
667686
context: QueryContext = null): Decimal = {
668687
try {
669-
val bigDecimal = stringToJavaBigDecimal(str)
670-
// We fast fail because constructing a very large JavaBigDecimal to Decimal is very slow.
671-
// For example: Decimal("6.0790316E+25569151")
672-
if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
673-
!SqlApiConf.get.allowNegativeScaleOfDecimalEnabled) {
688+
val decimal = stringBigDecimalToDecimal(stringToJavaBigDecimal(str))
689+
if (decimal == null) {
674690
throw DataTypeErrors.outOfDecimalTypeRangeError(str)
675-
} else {
676-
Decimal(bigDecimal)
677691
}
692+
decimal
678693
} catch {
679694
case _: NumberFormatException =>
680695
throw DataTypeErrors.invalidInputInCastToNumberError(to, str, context)

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
330330
assert(cast("abcdef", FloatType).nullable)
331331
}
332332

333+
test("SPARK-60118: cast string of zero with a large exponent to decimal") {
334+
Seq("0e38", "0e39", "-0E+100", "0e2147483647").foreach { s =>
335+
checkEvaluation(cast(s, DecimalType(10, 2)), Decimal("0.00"))
336+
}
337+
}
338+
333339
test("cast from timestamp") {
334340
val millis = 15 * 1000 + 3
335341
val seconds = millis * 1000 + 3

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -559,6 +559,9 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
559559
Decimal("60000000000000000000000000000000000000"))
560560
checkEvaluation(cast("6E+38", DecimalType(38, 0)), null)
561561
checkEvaluation(cast("6E+37", DecimalType(38, 1)), null)
562+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
563+
checkEvaluation(cast(s, DecimalType(10, 2)), null)
564+
}
562565

563566
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
564567
}

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,11 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
475475
checkExceptionInExpression[ArithmeticException](
476476
cast("6E+37", DecimalType(38, 1)),
477477
"cannot be represented as Decimal(38, 1)")
478+
Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
479+
checkExceptionInExpression[ArithmeticException](
480+
cast(s, DecimalType(10, 2)),
481+
"NUMERIC_OUT_OF_SUPPORTED_RANGE")
482+
}
478483

479484
checkExceptionInExpression[NumberFormatException](
480485
cast("abcd", DecimalType(38, 1)),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -446,6 +446,43 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
446446
}
447447
}
448448

449+
test("SPARK-60118: UTF8String to Decimal with exponent near Int.MaxValue") {
450+
// The number of integral digits (precision - scale) of these values overflows Int.
451+
Seq("1e2147483647", "-1E+2147483647", "12e2147483647",
452+
"1.5e2147483647").foreach { string =>
453+
assert(Decimal.fromString(UTF8String.fromString(string)) === null)
454+
checkError(
455+
exception = intercept[SparkArithmeticException](
456+
Decimal.fromStringANSI(UTF8String.fromString(string))),
457+
condition = "NUMERIC_OUT_OF_SUPPORTED_RANGE",
458+
parameters = Map("value" -> string))
459+
}
460+
// Zero fits any decimal type whatever its exponent. "0e38" is the smallest exponent that
461+
// used to be rejected, and "0e37" used to get precision 38.
462+
Seq("0e37", "0e38", "0e39", "-0E+100", "0e2147483647").foreach { string =>
463+
Seq(Decimal.fromString(UTF8String.fromString(string)),
464+
Decimal.fromStringANSI(UTF8String.fromString(string))).foreach { d =>
465+
assert(d === Decimal(0))
466+
assert(d.precision === 1)
467+
assert(d.scale === 0)
468+
}
469+
}
470+
Seq(Decimal.fromString(UTF8String.fromString("0.00")),
471+
Decimal.fromStringANSI(UTF8String.fromString("0.00"))).foreach { d =>
472+
assert(d.precision === 2)
473+
assert(d.scale === 2)
474+
}
475+
// With negative scales allowed, zero keeps its negative scale.
476+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
477+
Seq(Decimal.fromString(UTF8String.fromString("0e39")),
478+
Decimal.fromStringANSI(UTF8String.fromString("0e39"))).foreach { d =>
479+
assert(d === Decimal(0))
480+
assert(d.precision === 1)
481+
assert(d.scale === -39)
482+
}
483+
}
484+
}
485+
449486
// 18 is a max number of digits in Decimal's compact long
450487
test("SPARK-41554: decrease/increase scale by 18 and more on compact decimal") {
451488
val unscaledNums = Seq(

0 commit comments

Comments
 (0)