Skip to content

Commit eaed2f0

Browse files
committed
[SPARK-60119][4.2][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
### What changes were proposed in this pull request? Backport of #59335 to branch-4.2. The cherry-pick conflicted only on `CastUtils.java`, which does not exist on this branch: `Cast` calls `QueryExecutionErrors.cannotChangeDecimalPrecisionError` directly in both the interpreted and generated code. So instead of removing that method and switching `CastUtils`, this backport keeps it and makes it delegate to `DataTypeErrors.cannotChangeDecimalPrecisionError`, which gives the same error message. The rest of the change is identical to #59335. In the BigDecimal branch of `Decimal.changePrecision`, when the scale change exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits (`precision - scale`, computed in Long) is checked before calling `setScale`: - If the value is nonzero and has more integral digits than the target type allows, return `false` (overflow) directly. Rounding cannot bring it back into range. - If the value is below 0.01 ulp of the target scale, the result depends only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is rescaled instead. This gives the same result for every rounding mode. The Long branch of `changePrecision` now computes the scale change in Long, so it no longer wraps and throws `ArrayIndexOutOfBoundsException` from `POW_10(diff)` when negative scales are allowed. `NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose scale is below -1000 (only possible under `spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of `toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a String. Other values are rendered as before. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy, now delegates to `DataTypeErrors.cannotChangeDecimalPrecisionError`. ### Why are the changes needed? `setScale` takes time that grows with the scale change, and throws once 10^change exceeds BigInteger's range or the change exceeds the Int range: ```sql SELECT CAST('1e-2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: BigInteger would overflow supported range (LEGACY, ANSI and TRY) SELECT CAST('1e-100000000' AS DECIMAL(10,2)); -- 0.00, after ~40s SET spark.sql.legacy.allowNegativeScaleOfDecimal=true; SELECT CAST('1e2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: Underflow ``` This branch already has the related fix SPARK-60118, so backporting this keeps decimal casts consistent with master. ### Does this PR introduce _any_ user-facing change? Yes, the same as #59335: - Casting a string whose value is far below the target scale's ulp now returns the rounded value (e.g. 0.00) instead of throwing `ArithmeticException`, and returns quickly. - With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string with a huge exponent returns NULL (non-ANSI, try_cast) or throws `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException: Underflow`, and a decimal stored as a long no longer throws `ArrayIndexOutOfBoundsException` for a scale change beyond the Int range. In that error, a value whose scale is below -1000 is shown in scientific notation. No input that previously produced a value changes its result. ### How was this patch tested? The tests from #59335 in `DecimalSuite`, `CastSuiteBase`, `CastWithAnsiOnSuite` and `CastWithAnsiOffSuite` (also run through `TryCastSuite`). On this branch with JDK 17, {'4.3': '512 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`, plus `UdfSerializationSuite` (16 tests)', '4.2': '410 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.1': '403 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.0': '354 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`'} pass, and scalastyle (`sql-api`, `catalyst`, `catalyst` tests) and the `sql/api` scalafmt check pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59513 from viirya/SPARK-60119-4.2. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent 46b0da4 commit eaed2f0

7 files changed

Lines changed: 167 additions & 16 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,15 +196,24 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
196196
numericValueOutOfRange(value, decimalPrecision, decimalScale, context)
197197
}
198198

199+
// A decimal whose scale is below -MAX_PLAIN_STRING_TRAILING_ZEROS (that is, with more than
200+
// this many zeros implied by a negative scale) is rendered in scientific notation in error
201+
// messages instead of plain notation.
202+
private val MAX_PLAIN_STRING_TRAILING_ZEROS = 1000
203+
199204
private def numericValueOutOfRange(
200205
value: Decimal,
201206
decimalPrecision: Int,
202207
decimalScale: Int,
203208
context: QueryContext): ArithmeticException = {
209+
// A huge negative scale (legacy mode only) makes the plain string too long to build,
210+
// e.g. for 1E+2147483647.
211+
val valueString =
212+
if (value.scale < -MAX_PLAIN_STRING_TRAILING_ZEROS) value.toString else value.toPlainString
204213
new SparkArithmeticException(
205214
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
206215
messageParameters = Map(
207-
"value" -> value.toPlainString,
216+
"value" -> valueString,
208217
"precision" -> decimalPrecision.toString,
209218
"scale" -> decimalScale.toString,
210219
"config" -> toSQLConf("spark.sql.ansi.enabled")),

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -398,8 +398,9 @@ final class Decimal extends Ordered[Decimal] with Serializable {
398398
// First, update our lv if we can, or transfer over to using a BigDecimal
399399
if (dv.eq(null)) {
400400
if (scale < _scale) {
401-
// Easier case: we just need to divide our scale down
402-
val diff = _scale - scale
401+
// Easier case: we just need to divide our scale down. The difference is computed in
402+
// Long since it can exceed the Int range when negative scales are allowed.
403+
val diff = _scale.toLong - scale
403404
// If diff is greater than max number of digits we store in Long, then
404405
// value becomes 0. Otherwise we calculate new value dividing by power of 10.
405406
// In both cases we apply rounding after that.
@@ -411,7 +412,7 @@ final class Decimal extends Ordered[Decimal] with Serializable {
411412
case _ => throw DataTypeErrors.unsupportedRoundingMode(roundMode)
412413
}
413414
} else {
414-
val pow10diff = POW_10(diff)
415+
val pow10diff = POW_10(diff.toInt)
415416
// % and / always round to 0
416417
val droppedDigits = lv % pow10diff
417418
lv /= pow10diff
@@ -440,11 +441,11 @@ final class Decimal extends Ordered[Decimal] with Serializable {
440441
} else if (scale > _scale) {
441442
// We might be able to multiply lv by a power of 10 and not overflow, but if not,
442443
// switch to using a BigDecimal
443-
val diff = scale - _scale
444-
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0))
444+
val diff = scale.toLong - _scale
445+
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0L).toInt)
445446
if (diff <= MAX_LONG_DIGITS && lv > -p && lv < p) {
446447
// Multiplying lv by POW_10(diff) will still keep it below MAX_LONG_DIGITS
447-
lv *= POW_10(diff)
448+
lv *= POW_10(diff.toInt)
448449
} else {
449450
// Give up on using Longs; switch to BigDecimal, which we'll modify below
450451
dv = BigDecimal(lv, _scale)
@@ -456,6 +457,20 @@ final class Decimal extends Ordered[Decimal] with Serializable {
456457
if (dv.ne(null)) {
457458
// We get here if either we started with a BigDecimal, or we switched to one because we would
458459
// have overflowed our Long; in either case we must rescale dv to the new scale.
460+
if (math.abs(dv.scale.toLong - scale) > DecimalType.MAX_PRECISION) {
461+
// setScale's cost grows with the scale change, and it throws once 10^change exceeds
462+
// BigInteger's range (e.g. for 1e-2147483647) or the change exceeds the Int range.
463+
// Skip it when the result is known.
464+
val numIntegralDigits = dv.precision.toLong - dv.scale
465+
if (dv.signum != 0 && numIntegralDigits > precision.toLong - scale) {
466+
return false
467+
}
468+
if (numIntegralDigits < -scale.toLong - 1) {
469+
// |dv| is below 0.01 ulp of the new scale, so the result depends only on the sign
470+
// and the rounding mode. Round a value of the same sign and 0.01 ulp instead.
471+
dv = BigDecimal(dv.signum, scale + 2)
472+
}
473+
}
459474
dv = dv.setScale(scale, roundMode)
460475
if (dv.precision > precision) {
461476
return false

‎sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -108,15 +108,8 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
108108
decimalPrecision: Int,
109109
decimalScale: Int,
110110
context: QueryContext = null): ArithmeticException = {
111-
new SparkArithmeticException(
112-
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
113-
messageParameters = Map(
114-
"value" -> value.toPlainString,
115-
"precision" -> decimalPrecision.toString,
116-
"scale" -> decimalScale.toString,
117-
"config" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
118-
context = getQueryContext(context),
119-
summary = getSummary(context))
111+
DataTypeErrors.cannotChangeDecimalPrecisionError(
112+
value, decimalPrecision, decimalScale, context)
120113
}
121114

122115
def invalidInputSyntaxForBooleanError(

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1352,6 +1352,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
13521352
}
13531353
}
13541354

1355+
test("SPARK-60119: cast string with a huge negative exponent to decimal") {
1356+
Seq("1e-2147483647", "-1e-2147483647", "1e-100000000", "0e-2147483647").foreach { str =>
1357+
checkEvaluation(cast(str, DecimalType(10, 2)), Decimal("0.00"))
1358+
}
1359+
}
1360+
13551361
test("cast ANSI intervals to/from decimals") {
13561362
Seq(
13571363
(Duration.ZERO, DayTimeIntervalType(DAY), DecimalType(10, 3)) -> Decimal(0, 10, 3),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -554,6 +554,15 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
554554
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
555555
}
556556

557+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
558+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
559+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
560+
checkEvaluation(cast(str, DecimalType(10, 2)), null)
561+
}
562+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
563+
}
564+
}
565+
557566
test("data type casting II") {
558567
checkEvaluation(
559568
cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -485,6 +485,45 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
485485
castErrMsg("abcd", DecimalType(38, 1)))
486486
}
487487

488+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
489+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
490+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
491+
checkExceptionInExpression[ArithmeticException](
492+
cast(str, DecimalType(10, 2)),
493+
"cannot be represented as Decimal(10, 2)")
494+
}
495+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
496+
497+
// The value is shown in plain notation unless its scale is hugely negative.
498+
Seq("1e2147483647" -> "1E+2147483647", "1e40" -> ("1" + "0" * 40)).foreach {
499+
case (str, value) =>
500+
def expr: Expression = cast(str, DecimalType(10, 2))
501+
if (!isTryCast) {
502+
def checkOutOfRange(eval: => Any): Unit = {
503+
checkError(
504+
exception = intercept[SparkArithmeticException](eval),
505+
condition = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
506+
parameters = Map(
507+
"value" -> value,
508+
"precision" -> "10",
509+
"scale" -> "2",
510+
"config" -> """"spark.sql.ansi.enabled""""),
511+
queryContext = Array(ExpectedContext(fragment = "", start = -1, stop = -1)))
512+
}
513+
// Interpreted, and both codegen paths without falling back to interpreted.
514+
checkOutOfRange(evaluateWithoutCodegen(expr))
515+
withSQLConf(
516+
SQLConf.CODEGEN_FACTORY_MODE.key -> CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
517+
checkOutOfRange(evaluateWithMutableProjection(expr))
518+
checkOutOfRange(evaluateWithUnsafeProjection(expr))
519+
}
520+
} else {
521+
checkEvaluation(expr, null)
522+
}
523+
}
524+
}
525+
}
526+
488527
protected def checkCastToBooleanError(l: Literal, to: DataType, tryCastResult: Any): Unit = {
489528
checkExceptionInExpression[SparkRuntimeException](
490529
cast(l, to), """cannot be cast to "BOOLEAN"""")

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -501,4 +501,84 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
501501
s"unscaled: $unscaled, scaleFrom: $scaleFrom, scaleTo: $scaleTo, mode: $roundMode")
502502
}
503503
}
504+
505+
test("SPARK-60119: changePrecision with a source scale far from the target scale") {
506+
// A value below 0.01 ulp of the target scale rounds to 0 or +/-1 ulp depending only on its
507+
// sign and the rounding mode, so it must round like a small value of the same sign.
508+
Seq("1e-2147483647", "-1e-2147483647", "9.99e-100000000", "-1e-100000000",
509+
"0e-2147483647").foreach { str =>
510+
val bd = new java.math.BigDecimal(str)
511+
allSupportedRoundModes.foreach { mode =>
512+
val expected = BigDecimal(bd.signum, 50).setScale(2, mode)
513+
val d = Decimal(bd)
514+
assert(d.changePrecision(10, 2, mode), s"$str, $mode")
515+
assert(d.toString === expected.toString, s"$str, $mode")
516+
}
517+
}
518+
519+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
520+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
521+
allSupportedRoundModes.foreach { mode =>
522+
val d = Decimal(new java.math.BigDecimal(str))
523+
assert(d.toPrecision(10, 2, mode) === null, s"$str, $mode")
524+
}
525+
}
526+
val zero = Decimal(new java.math.BigDecimal("0e2147483647"))
527+
assert(zero.changePrecision(10, 2))
528+
assert(zero.toString === "0.00")
529+
530+
// A compact (long) input takes the Long branch first, where the scale change can exceed
531+
// the Int range. These switch to the BigDecimal branch, so check their results directly.
532+
Seq(
533+
(1L, 1, -40, 10, 2, null),
534+
(0L, 1, -2147483647, 10, 2, "0.00"),
535+
(1L, 1, Int.MinValue, 38, 20, null)).foreach { case (unscaled, p, s, toP, toS, expected) =>
536+
allSupportedRoundModes.foreach { mode =>
537+
val compact = Decimal.createUnsafe(unscaled, p, s)
538+
checkCompact(compact, true)
539+
assert(Option(compact.toPrecision(toP, toS, mode)).map(_.toString).orNull === expected,
540+
s"$unscaled, ($p, $s) -> ($toP, $toS), $mode")
541+
}
542+
}
543+
// These stay in the Long branch, so compare with the same value in BigDecimal form.
544+
Seq(100L, -100L).foreach { unscaled =>
545+
allSupportedRoundModes.foreach { mode =>
546+
val compact = Decimal.createUnsafe(unscaled, 3, 2)
547+
checkCompact(compact, true)
548+
val expanded = Decimal(java.math.BigDecimal.valueOf(unscaled, 2))
549+
checkCompact(expanded, false)
550+
assert(
551+
Option(compact.toPrecision(1, -2147483647, mode)).map(_.toString) ===
552+
Option(expanded.toPrecision(1, -2147483647, mode)).map(_.toString),
553+
s"$unscaled, $mode")
554+
}
555+
}
556+
}
557+
558+
// Values around the bounds of the shortcuts must round as BigDecimal.setScale does.
559+
Seq(false, true).foreach { allowNegativeScale =>
560+
withSQLConf(
561+
SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> allowNegativeScale.toString) {
562+
val targets =
563+
Seq((10, 2), (3, 3), (5, 0)) ++ (if (allowNegativeScale) Seq((5, -3)) else Nil)
564+
for {
565+
// The long mantissas make the scale change large enough to take the shortcuts.
566+
mantissa <- Seq("1", "4.9", "5", "5.1", "9.99", "9.995",
567+
"1." + "0" * 79 + "1", "4." + "9" * 80, "5." + "0" * 80, "5." + "0" * 79 + "1",
568+
"9." + "9" * 80)
569+
exponent <- -8 to 12
570+
sign <- Seq("", "-")
571+
mode <- allSupportedRoundModes
572+
(precision, scale) <- targets
573+
} {
574+
val bd = new java.math.BigDecimal(s"$sign${mantissa}e$exponent")
575+
val rounded = BigDecimal(bd).setScale(scale, mode)
576+
val expected = if (rounded.precision > precision) null else rounded.toString
577+
val d = Decimal(bd).toPrecision(precision, scale, mode)
578+
assert(Option(d).map(_.toString).orNull === expected,
579+
s"$bd, $mode, ($precision, $scale)")
580+
}
581+
}
582+
}
583+
}
504584
}

0 commit comments

Comments
 (0)