Skip to content

Commit 212ff5a

Browse files
viiryadongjoon-hyun
authored andcommitted
[SPARK-60119][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
### What changes were proposed in this pull request? In the BigDecimal branch of `Decimal.changePrecision`, when the scale change exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits (`precision - scale`, computed in Long) is checked before calling `setScale`: - If the value is nonzero and has more integral digits than the target type allows, return `false` (overflow) directly. Rounding cannot bring it back into range. - If the value is below 0.01 ulp of the target scale, the result depends only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is rescaled instead. This gives the same result for every rounding mode. Otherwise the remaining scale change is bounded by the number of digits of the value, so `setScale` is unchanged. The extra check costs one Int comparison when the scale change is small, which is the common case. The Long branch of `changePrecision` now computes the scale change in Long. With negative scales allowed it can exceed the Int range and wrap, so `POW_10(diff)` threw `ArrayIndexOutOfBoundsException`, e.g. for a compact `1.00` changed to `DECIMAL(1, -2147483647)`. It now takes the existing rounding path or switches to BigDecimal. `NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose scale is below -1000 (only possible under `spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of `toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a String. Other values are rendered as before. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy, is removed and `CastUtils` calls `DataTypeErrors.cannotChangeDecimalPrecisionError` directly. ### Why are the changes needed? `setScale` takes time that grows with the scale change, and throws once 10^change exceeds BigInteger's range or the change exceeds the Int range: ```sql SELECT CAST('1e-2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: BigInteger would overflow supported range (LEGACY, ANSI and TRY) SELECT CAST('1e-100000000' AS DECIMAL(10,2)); -- 0.00, after ~40s SET spark.sql.legacy.allowNegativeScaleOfDecimal=true; SELECT CAST('1e2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: Underflow ``` This is the small-side counterpart of the fast-fail added in SPARK-35841/SPARK-37451. With negative scales allowed, `Decimal.fromString` intentionally skips that fast-fail, so huge values reach `changePrecision` too. ### Does this PR introduce _any_ user-facing change? Yes. - Casting a string whose value is far below the target scale's ulp now returns the rounded value (e.g. 0.00) instead of throwing `ArithmeticException`, and returns quickly. - With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string with a huge exponent returns NULL (non-ANSI, try_cast) or throws `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException: Underflow`. In that error, a value whose scale is below -1000 (e.g. `1E+2147483647`) is shown in scientific notation, since its plain form is too long to build. Other values are shown as before. - With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, changing a decimal stored as a long (precision up to 18, e.g. after an `UnsafeRow` round trip) by a scale change beyond the Int range no longer throws `ArrayIndexOutOfBoundsException`, and gives the same result as the same value stored as a BigDecimal. No input that previously produced a value changes its result. ### How was this patch tested? New tests in `DecimalSuite`, `CastSuiteBase` (so they run in `CastWithAnsiOffSuite`, `CastWithAnsiOnSuite` and `TryCastSuite`), `CastWithAnsiOffSuite` and `CastWithAnsiOnSuite`, covering interpreted and codegen evaluation. All of them fail without the fix. The ANSI test also checks the `value` parameter of the error, in scientific notation for `1e2147483647` and in plain notation for `1e40`. `DecimalSuite` also checks that compact (long) inputs, including scale changes beyond the Int range, round like the same values in BigDecimal form, and that values around the bounds of both shortcuts, with short and 81-digit mantissas, under every supported rounding mode and with and without negative scales, round exactly like `BigDecimal.setScale`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59335 from viirya/SPARK-60119. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 9c3a51f) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 4d46e1f commit 212ff5a

8 files changed

Lines changed: 167 additions & 24 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,15 +202,24 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
202202
numericValueOutOfRange(value, decimalPrecision, decimalScale, context)
203203
}
204204

205+
// A decimal whose scale is below -MAX_PLAIN_STRING_TRAILING_ZEROS (that is, with more than
206+
// this many zeros implied by a negative scale) is rendered in scientific notation in error
207+
// messages instead of plain notation.
208+
private val MAX_PLAIN_STRING_TRAILING_ZEROS = 1000
209+
205210
private def numericValueOutOfRange(
206211
value: Decimal,
207212
decimalPrecision: Int,
208213
decimalScale: Int,
209214
context: QueryContext): ArithmeticException = {
215+
// A huge negative scale (legacy mode only) makes the plain string too long to build,
216+
// e.g. for 1E+2147483647.
217+
val valueString =
218+
if (value.scale < -MAX_PLAIN_STRING_TRAILING_ZEROS) value.toString else value.toPlainString
210219
new SparkArithmeticException(
211220
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
212221
messageParameters = Map(
213-
"value" -> value.toPlainString,
222+
"value" -> valueString,
214223
"precision" -> decimalPrecision.toString,
215224
"scale" -> decimalScale.toString,
216225
"config" -> toSQLConf("spark.sql.ansi.enabled")),

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -398,8 +398,9 @@ final class Decimal extends Ordered[Decimal] with Serializable {
398398
// First, update our lv if we can, or transfer over to using a BigDecimal
399399
if (dv.eq(null)) {
400400
if (scale < _scale) {
401-
// Easier case: we just need to divide our scale down
402-
val diff = _scale - scale
401+
// Easier case: we just need to divide our scale down. The difference is computed in
402+
// Long since it can exceed the Int range when negative scales are allowed.
403+
val diff = _scale.toLong - scale
403404
// If diff is greater than max number of digits we store in Long, then
404405
// value becomes 0. Otherwise we calculate new value dividing by power of 10.
405406
// In both cases we apply rounding after that.
@@ -411,7 +412,7 @@ final class Decimal extends Ordered[Decimal] with Serializable {
411412
case _ => throw DataTypeErrors.unsupportedRoundingMode(roundMode)
412413
}
413414
} else {
414-
val pow10diff = POW_10(diff)
415+
val pow10diff = POW_10(diff.toInt)
415416
// % and / always round to 0
416417
val droppedDigits = lv % pow10diff
417418
lv /= pow10diff
@@ -442,11 +443,11 @@ final class Decimal extends Ordered[Decimal] with Serializable {
442443
} else if (scale > _scale) {
443444
// We might be able to multiply lv by a power of 10 and not overflow, but if not,
444445
// switch to using a BigDecimal
445-
val diff = scale - _scale
446-
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0))
446+
val diff = scale.toLong - _scale
447+
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0L).toInt)
447448
if (diff <= MAX_LONG_DIGITS && lv > -p && lv < p) {
448449
// Multiplying lv by POW_10(diff) will still keep it below MAX_LONG_DIGITS
449-
lv *= POW_10(diff)
450+
lv *= POW_10(diff.toInt)
450451
} else {
451452
// Give up on using Longs; switch to BigDecimal, which we'll modify below
452453
dv = BigDecimal(lv, _scale)
@@ -458,6 +459,20 @@ final class Decimal extends Ordered[Decimal] with Serializable {
458459
if (dv.ne(null)) {
459460
// We get here if either we started with a BigDecimal, or we switched to one because we would
460461
// have overflowed our Long; in either case we must rescale dv to the new scale.
462+
if (math.abs(dv.scale.toLong - scale) > DecimalType.MAX_PRECISION) {
463+
// setScale's cost grows with the scale change, and it throws once 10^change exceeds
464+
// BigInteger's range (e.g. for 1e-2147483647) or the change exceeds the Int range.
465+
// Skip it when the result is known.
466+
val numIntegralDigits = dv.precision.toLong - dv.scale
467+
if (dv.signum != 0 && numIntegralDigits > precision.toLong - scale) {
468+
return false
469+
}
470+
if (numIntegralDigits < -scale.toLong - 1) {
471+
// |dv| is below 0.01 ulp of the new scale, so the result depends only on the sign
472+
// and the rounding mode. Round a value of the same sign and 0.01 ulp instead.
473+
dv = BigDecimal(dv.signum, scale + 2)
474+
}
475+
}
461476
dv = dv.setScale(scale, roundMode)
462477
if (dv.precision > precision) {
463478
return false

‎sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/CastUtils.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.sql.catalyst.expressions;
1919

2020
import org.apache.spark.QueryContext;
21+
import org.apache.spark.sql.errors.DataTypeErrors;
2122
import org.apache.spark.sql.errors.QueryExecutionErrors;
2223
import org.apache.spark.sql.types.DataType;
2324
import org.apache.spark.sql.types.DataTypes;
@@ -107,7 +108,7 @@ public static short doubleToShortExact(double v) {
107108
public static Decimal changePrecisionExact(
108109
Decimal d, int precision, int scale, QueryContext context) {
109110
if (d.changePrecision(precision, scale)) return d;
110-
throw QueryExecutionErrors.cannotChangeDecimalPrecisionError(d, precision, scale, context);
111+
throw DataTypeErrors.cannotChangeDecimalPrecisionError(d, precision, scale, context);
111112
}
112113

113114
public static Decimal changePrecisionOrNull(Decimal d, int precision, int scale) {

‎sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -103,22 +103,6 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
103103
)
104104
}
105105

106-
def cannotChangeDecimalPrecisionError(
107-
value: Decimal,
108-
decimalPrecision: Int,
109-
decimalScale: Int,
110-
context: QueryContext = null): ArithmeticException = {
111-
new SparkArithmeticException(
112-
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
113-
messageParameters = Map(
114-
"value" -> value.toPlainString,
115-
"precision" -> decimalPrecision.toString,
116-
"scale" -> decimalScale.toString,
117-
"config" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
118-
context = getQueryContext(context),
119-
summary = getSummary(context))
120-
}
121-
122106
def invalidInputSyntaxForBooleanError(
123107
s: UTF8String,
124108
context: QueryContext): SparkRuntimeException = {

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2150,6 +2150,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
21502150
}
21512151
}
21522152

2153+
test("SPARK-60119: cast string with a huge negative exponent to decimal") {
2154+
Seq("1e-2147483647", "-1e-2147483647", "1e-100000000", "0e-2147483647").foreach { str =>
2155+
checkEvaluation(cast(str, DecimalType(10, 2)), Decimal("0.00"))
2156+
}
2157+
}
2158+
21532159
test("cast ANSI intervals to/from decimals") {
21542160
Seq(
21552161
(Duration.ZERO, DayTimeIntervalType(DAY), DecimalType(10, 3)) -> Decimal(0, 10, 3),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -566,6 +566,15 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
566566
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
567567
}
568568

569+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
570+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
571+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
572+
checkEvaluation(cast(str, DecimalType(10, 2)), null)
573+
}
574+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
575+
}
576+
}
577+
569578
test("data type casting II") {
570579
checkEvaluation(
571580
cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -486,6 +486,45 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
486486
castErrMsg("abcd", DecimalType(38, 1)))
487487
}
488488

489+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
490+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
491+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
492+
checkExceptionInExpression[ArithmeticException](
493+
cast(str, DecimalType(10, 2)),
494+
"cannot be represented as Decimal(10, 2)")
495+
}
496+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
497+
498+
// The value is shown in plain notation unless its scale is hugely negative.
499+
Seq("1e2147483647" -> "1E+2147483647", "1e40" -> ("1" + "0" * 40)).foreach {
500+
case (str, value) =>
501+
def expr: Expression = cast(str, DecimalType(10, 2))
502+
if (!isTryCast) {
503+
def checkOutOfRange(eval: => Any): Unit = {
504+
checkError(
505+
exception = intercept[SparkArithmeticException](eval),
506+
condition = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
507+
parameters = Map(
508+
"value" -> value,
509+
"precision" -> "10",
510+
"scale" -> "2",
511+
"config" -> """"spark.sql.ansi.enabled""""),
512+
queryContext = Array(ExpectedContext(fragment = "", start = -1, stop = -1)))
513+
}
514+
// Interpreted, and both codegen paths without falling back to interpreted.
515+
checkOutOfRange(evaluateWithoutCodegen(expr))
516+
withSQLConf(
517+
SQLConf.CODEGEN_FACTORY_MODE.key -> CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
518+
checkOutOfRange(evaluateWithMutableProjection(expr))
519+
checkOutOfRange(evaluateWithUnsafeProjection(expr))
520+
}
521+
} else {
522+
checkEvaluation(expr, null)
523+
}
524+
}
525+
}
526+
}
527+
489528
protected def checkCastToBooleanError(l: Literal, to: DataType, tryCastResult: Any): Unit = {
490529
checkExceptionInExpression[SparkRuntimeException](
491530
cast(l, to), """cannot be cast to "BOOLEAN"""")

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -529,4 +529,84 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
529529
s"unscaled: $unscaled, scaleFrom: $scaleFrom, scaleTo: $scaleTo, mode: $roundMode")
530530
}
531531
}
532+
533+
test("SPARK-60119: changePrecision with a source scale far from the target scale") {
534+
// A value below 0.01 ulp of the target scale rounds to 0 or +/-1 ulp depending only on its
535+
// sign and the rounding mode, so it must round like a small value of the same sign.
536+
Seq("1e-2147483647", "-1e-2147483647", "9.99e-100000000", "-1e-100000000",
537+
"0e-2147483647").foreach { str =>
538+
val bd = new java.math.BigDecimal(str)
539+
allSupportedRoundModes.foreach { mode =>
540+
val expected = BigDecimal(bd.signum, 50).setScale(2, mode)
541+
val d = Decimal(bd)
542+
assert(d.changePrecision(10, 2, mode), s"$str, $mode")
543+
assert(d.toString === expected.toString, s"$str, $mode")
544+
}
545+
}
546+
547+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
548+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
549+
allSupportedRoundModes.foreach { mode =>
550+
val d = Decimal(new java.math.BigDecimal(str))
551+
assert(d.toPrecision(10, 2, mode) === null, s"$str, $mode")
552+
}
553+
}
554+
val zero = Decimal(new java.math.BigDecimal("0e2147483647"))
555+
assert(zero.changePrecision(10, 2))
556+
assert(zero.toString === "0.00")
557+
558+
// A compact (long) input takes the Long branch first, where the scale change can exceed
559+
// the Int range. These switch to the BigDecimal branch, so check their results directly.
560+
Seq(
561+
(1L, 1, -40, 10, 2, null),
562+
(0L, 1, -2147483647, 10, 2, "0.00"),
563+
(1L, 1, Int.MinValue, 38, 20, null)).foreach { case (unscaled, p, s, toP, toS, expected) =>
564+
allSupportedRoundModes.foreach { mode =>
565+
val compact = Decimal.createUnsafe(unscaled, p, s)
566+
checkCompact(compact, true)
567+
assert(Option(compact.toPrecision(toP, toS, mode)).map(_.toString).orNull === expected,
568+
s"$unscaled, ($p, $s) -> ($toP, $toS), $mode")
569+
}
570+
}
571+
// These stay in the Long branch, so compare with the same value in BigDecimal form.
572+
Seq(100L, -100L).foreach { unscaled =>
573+
allSupportedRoundModes.foreach { mode =>
574+
val compact = Decimal.createUnsafe(unscaled, 3, 2)
575+
checkCompact(compact, true)
576+
val expanded = Decimal(java.math.BigDecimal.valueOf(unscaled, 2))
577+
checkCompact(expanded, false)
578+
assert(
579+
Option(compact.toPrecision(1, -2147483647, mode)).map(_.toString) ===
580+
Option(expanded.toPrecision(1, -2147483647, mode)).map(_.toString),
581+
s"$unscaled, $mode")
582+
}
583+
}
584+
}
585+
586+
// Values around the bounds of the shortcuts must round as BigDecimal.setScale does.
587+
Seq(false, true).foreach { allowNegativeScale =>
588+
withSQLConf(
589+
SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> allowNegativeScale.toString) {
590+
val targets =
591+
Seq((10, 2), (3, 3), (5, 0)) ++ (if (allowNegativeScale) Seq((5, -3)) else Nil)
592+
for {
593+
// The long mantissas make the scale change large enough to take the shortcuts.
594+
mantissa <- Seq("1", "4.9", "5", "5.1", "9.99", "9.995",
595+
"1." + "0" * 79 + "1", "4." + "9" * 80, "5." + "0" * 80, "5." + "0" * 79 + "1",
596+
"9." + "9" * 80)
597+
exponent <- -8 to 12
598+
sign <- Seq("", "-")
599+
mode <- allSupportedRoundModes
600+
(precision, scale) <- targets
601+
} {
602+
val bd = new java.math.BigDecimal(s"$sign${mantissa}e$exponent")
603+
val rounded = BigDecimal(bd).setScale(scale, mode)
604+
val expected = if (rounded.precision > precision) null else rounded.toString
605+
val d = Decimal(bd).toPrecision(precision, scale, mode)
606+
assert(Option(d).map(_.toString).orNull === expected,
607+
s"$bd, $mode, ($precision, $scale)")
608+
}
609+
}
610+
}
611+
}
532612
}

0 commit comments

Comments
 (0)