Skip to content

Commit a3476e6

Browse files
committed
[SPARK-60119][4.3][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
### What changes were proposed in this pull request? Backport of #59335 to branch-4.3. The cherry-pick applied cleanly. In the BigDecimal branch of `Decimal.changePrecision`, when the scale change exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits (`precision - scale`, computed in Long) is checked before calling `setScale`: - If the value is nonzero and has more integral digits than the target type allows, return `false` (overflow) directly. Rounding cannot bring it back into range. - If the value is below 0.01 ulp of the target scale, the result depends only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is rescaled instead. This gives the same result for every rounding mode. The Long branch of `changePrecision` now computes the scale change in Long, so it no longer wraps and throws `ArrayIndexOutOfBoundsException` from `POW_10(diff)` when negative scales are allowed. `NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose scale is below -1000 (only possible under `spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of `toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a String. Other values are rendered as before. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy, is removed and `CastUtils` calls `DataTypeErrors.cannotChangeDecimalPrecisionError` directly. ### Why are the changes needed? `setScale` takes time that grows with the scale change, and throws once 10^change exceeds BigInteger's range or the change exceeds the Int range: ```sql SELECT CAST('1e-2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: BigInteger would overflow supported range (LEGACY, ANSI and TRY) SELECT CAST('1e-100000000' AS DECIMAL(10,2)); -- 0.00, after ~40s SET spark.sql.legacy.allowNegativeScaleOfDecimal=true; SELECT CAST('1e2147483647' AS DECIMAL(10,2)); -- java.lang.ArithmeticException: Underflow ``` This branch already has the related fix SPARK-60118, so backporting this keeps decimal casts consistent with master. ### Does this PR introduce _any_ user-facing change? Yes, the same as #59335: - Casting a string whose value is far below the target scale's ulp now returns the rounded value (e.g. 0.00) instead of throwing `ArithmeticException`, and returns quickly. - With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string with a huge exponent returns NULL (non-ANSI, try_cast) or throws `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException: Underflow`, and a decimal stored as a long no longer throws `ArrayIndexOutOfBoundsException` for a scale change beyond the Int range. In that error, a value whose scale is below -1000 is shown in scientific notation. No input that previously produced a value changes its result. ### How was this patch tested? The tests from #59335 in `DecimalSuite`, `CastSuiteBase`, `CastWithAnsiOnSuite` and `CastWithAnsiOffSuite` (also run through `TryCastSuite`). On this branch with JDK 17, {'4.3': '512 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`, plus `UdfSerializationSuite` (16 tests)', '4.2': '410 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.1': '403 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.0': '354 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`'} pass, and scalastyle (`sql-api`, `catalyst`, `catalyst` tests) and the `sql/api` scalafmt check pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59512 from viirya/SPARK-60119-4.3. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent e292b41 commit a3476e6

8 files changed

Lines changed: 167 additions & 24 deletions

File tree

‎sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,15 +202,24 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
202202
numericValueOutOfRange(value, decimalPrecision, decimalScale, context)
203203
}
204204

205+
// A decimal whose scale is below -MAX_PLAIN_STRING_TRAILING_ZEROS (that is, with more than
206+
// this many zeros implied by a negative scale) is rendered in scientific notation in error
207+
// messages instead of plain notation.
208+
private val MAX_PLAIN_STRING_TRAILING_ZEROS = 1000
209+
205210
private def numericValueOutOfRange(
206211
value: Decimal,
207212
decimalPrecision: Int,
208213
decimalScale: Int,
209214
context: QueryContext): ArithmeticException = {
215+
// A huge negative scale (legacy mode only) makes the plain string too long to build,
216+
// e.g. for 1E+2147483647.
217+
val valueString =
218+
if (value.scale < -MAX_PLAIN_STRING_TRAILING_ZEROS) value.toString else value.toPlainString
210219
new SparkArithmeticException(
211220
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
212221
messageParameters = Map(
213-
"value" -> value.toPlainString,
222+
"value" -> valueString,
214223
"precision" -> decimalPrecision.toString,
215224
"scale" -> decimalScale.toString,
216225
"config" -> toSQLConf("spark.sql.ansi.enabled")),

‎sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -398,8 +398,9 @@ final class Decimal extends Ordered[Decimal] with Serializable {
398398
// First, update our lv if we can, or transfer over to using a BigDecimal
399399
if (dv.eq(null)) {
400400
if (scale < _scale) {
401-
// Easier case: we just need to divide our scale down
402-
val diff = _scale - scale
401+
// Easier case: we just need to divide our scale down. The difference is computed in
402+
// Long since it can exceed the Int range when negative scales are allowed.
403+
val diff = _scale.toLong - scale
403404
// If diff is greater than max number of digits we store in Long, then
404405
// value becomes 0. Otherwise we calculate new value dividing by power of 10.
405406
// In both cases we apply rounding after that.
@@ -411,7 +412,7 @@ final class Decimal extends Ordered[Decimal] with Serializable {
411412
case _ => throw DataTypeErrors.unsupportedRoundingMode(roundMode)
412413
}
413414
} else {
414-
val pow10diff = POW_10(diff)
415+
val pow10diff = POW_10(diff.toInt)
415416
// % and / always round to 0
416417
val droppedDigits = lv % pow10diff
417418
lv /= pow10diff
@@ -440,11 +441,11 @@ final class Decimal extends Ordered[Decimal] with Serializable {
440441
} else if (scale > _scale) {
441442
// We might be able to multiply lv by a power of 10 and not overflow, but if not,
442443
// switch to using a BigDecimal
443-
val diff = scale - _scale
444-
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0))
444+
val diff = scale.toLong - _scale
445+
val p = POW_10(math.max(MAX_LONG_DIGITS - diff, 0L).toInt)
445446
if (diff <= MAX_LONG_DIGITS && lv > -p && lv < p) {
446447
// Multiplying lv by POW_10(diff) will still keep it below MAX_LONG_DIGITS
447-
lv *= POW_10(diff)
448+
lv *= POW_10(diff.toInt)
448449
} else {
449450
// Give up on using Longs; switch to BigDecimal, which we'll modify below
450451
dv = BigDecimal(lv, _scale)
@@ -456,6 +457,20 @@ final class Decimal extends Ordered[Decimal] with Serializable {
456457
if (dv.ne(null)) {
457458
// We get here if either we started with a BigDecimal, or we switched to one because we would
458459
// have overflowed our Long; in either case we must rescale dv to the new scale.
460+
if (math.abs(dv.scale.toLong - scale) > DecimalType.MAX_PRECISION) {
461+
// setScale's cost grows with the scale change, and it throws once 10^change exceeds
462+
// BigInteger's range (e.g. for 1e-2147483647) or the change exceeds the Int range.
463+
// Skip it when the result is known.
464+
val numIntegralDigits = dv.precision.toLong - dv.scale
465+
if (dv.signum != 0 && numIntegralDigits > precision.toLong - scale) {
466+
return false
467+
}
468+
if (numIntegralDigits < -scale.toLong - 1) {
469+
// |dv| is below 0.01 ulp of the new scale, so the result depends only on the sign
470+
// and the rounding mode. Round a value of the same sign and 0.01 ulp instead.
471+
dv = BigDecimal(dv.signum, scale + 2)
472+
}
473+
}
459474
dv = dv.setScale(scale, roundMode)
460475
if (dv.precision > precision) {
461476
return false

‎sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/CastUtils.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.sql.catalyst.expressions;
1919

2020
import org.apache.spark.QueryContext;
21+
import org.apache.spark.sql.errors.DataTypeErrors;
2122
import org.apache.spark.sql.errors.QueryExecutionErrors;
2223
import org.apache.spark.sql.types.DataType;
2324
import org.apache.spark.sql.types.DataTypes;
@@ -107,7 +108,7 @@ public static short doubleToShortExact(double v) {
107108
public static Decimal changePrecisionExact(
108109
Decimal d, int precision, int scale, QueryContext context) {
109110
if (d.changePrecision(precision, scale)) return d;
110-
throw QueryExecutionErrors.cannotChangeDecimalPrecisionError(d, precision, scale, context);
111+
throw DataTypeErrors.cannotChangeDecimalPrecisionError(d, precision, scale, context);
111112
}
112113

113114
public static Decimal changePrecisionOrNull(Decimal d, int precision, int scale) {

‎sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -103,22 +103,6 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
103103
)
104104
}
105105

106-
def cannotChangeDecimalPrecisionError(
107-
value: Decimal,
108-
decimalPrecision: Int,
109-
decimalScale: Int,
110-
context: QueryContext = null): ArithmeticException = {
111-
new SparkArithmeticException(
112-
errorClass = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
113-
messageParameters = Map(
114-
"value" -> value.toPlainString,
115-
"precision" -> decimalPrecision.toString,
116-
"scale" -> decimalScale.toString,
117-
"config" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
118-
context = getQueryContext(context),
119-
summary = getSummary(context))
120-
}
121-
122106
def invalidInputSyntaxForBooleanError(
123107
s: UTF8String,
124108
context: QueryContext): SparkRuntimeException = {

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2158,6 +2158,12 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
21582158
}
21592159
}
21602160

2161+
test("SPARK-60119: cast string with a huge negative exponent to decimal") {
2162+
Seq("1e-2147483647", "-1e-2147483647", "1e-100000000", "0e-2147483647").foreach { str =>
2163+
checkEvaluation(cast(str, DecimalType(10, 2)), Decimal("0.00"))
2164+
}
2165+
}
2166+
21612167
test("cast ANSI intervals to/from decimals") {
21622168
Seq(
21632169
(Duration.ZERO, DayTimeIntervalType(DAY), DecimalType(10, 3)) -> Decimal(0, 10, 3),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -566,6 +566,15 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
566566
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
567567
}
568568

569+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
570+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
571+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
572+
checkEvaluation(cast(str, DecimalType(10, 2)), null)
573+
}
574+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
575+
}
576+
}
577+
569578
test("data type casting II") {
570579
checkEvaluation(
571580
cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -486,6 +486,45 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
486486
castErrMsg("abcd", DecimalType(38, 1)))
487487
}
488488

489+
test("SPARK-60119: cast string with a huge exponent to decimal with negative scale allowed") {
490+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
491+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
492+
checkExceptionInExpression[ArithmeticException](
493+
cast(str, DecimalType(10, 2)),
494+
"cannot be represented as Decimal(10, 2)")
495+
}
496+
checkEvaluation(cast("0e2147483647", DecimalType(10, 2)), Decimal("0.00"))
497+
498+
// The value is shown in plain notation unless its scale is hugely negative.
499+
Seq("1e2147483647" -> "1E+2147483647", "1e40" -> ("1" + "0" * 40)).foreach {
500+
case (str, value) =>
501+
def expr: Expression = cast(str, DecimalType(10, 2))
502+
if (!isTryCast) {
503+
def checkOutOfRange(eval: => Any): Unit = {
504+
checkError(
505+
exception = intercept[SparkArithmeticException](eval),
506+
condition = "NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION",
507+
parameters = Map(
508+
"value" -> value,
509+
"precision" -> "10",
510+
"scale" -> "2",
511+
"config" -> """"spark.sql.ansi.enabled""""),
512+
queryContext = Array(ExpectedContext(fragment = "", start = -1, stop = -1)))
513+
}
514+
// Interpreted, and both codegen paths without falling back to interpreted.
515+
checkOutOfRange(evaluateWithoutCodegen(expr))
516+
withSQLConf(
517+
SQLConf.CODEGEN_FACTORY_MODE.key -> CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
518+
checkOutOfRange(evaluateWithMutableProjection(expr))
519+
checkOutOfRange(evaluateWithUnsafeProjection(expr))
520+
}
521+
} else {
522+
checkEvaluation(expr, null)
523+
}
524+
}
525+
}
526+
}
527+
489528
protected def checkCastToBooleanError(l: Literal, to: DataType, tryCastResult: Any): Unit = {
490529
checkExceptionInExpression[SparkRuntimeException](
491530
cast(l, to), """cannot be cast to "BOOLEAN"""")

‎sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -528,4 +528,84 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester with SQLHelper
528528
s"unscaled: $unscaled, scaleFrom: $scaleFrom, scaleTo: $scaleTo, mode: $roundMode")
529529
}
530530
}
531+
532+
test("SPARK-60119: changePrecision with a source scale far from the target scale") {
533+
// A value below 0.01 ulp of the target scale rounds to 0 or +/-1 ulp depending only on its
534+
// sign and the rounding mode, so it must round like a small value of the same sign.
535+
Seq("1e-2147483647", "-1e-2147483647", "9.99e-100000000", "-1e-100000000",
536+
"0e-2147483647").foreach { str =>
537+
val bd = new java.math.BigDecimal(str)
538+
allSupportedRoundModes.foreach { mode =>
539+
val expected = BigDecimal(bd.signum, 50).setScale(2, mode)
540+
val d = Decimal(bd)
541+
assert(d.changePrecision(10, 2, mode), s"$str, $mode")
542+
assert(d.toString === expected.toString, s"$str, $mode")
543+
}
544+
}
545+
546+
withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> "true") {
547+
Seq("1e2147483647", "-1e2147483646", "12e2147483647", "1e100000000").foreach { str =>
548+
allSupportedRoundModes.foreach { mode =>
549+
val d = Decimal(new java.math.BigDecimal(str))
550+
assert(d.toPrecision(10, 2, mode) === null, s"$str, $mode")
551+
}
552+
}
553+
val zero = Decimal(new java.math.BigDecimal("0e2147483647"))
554+
assert(zero.changePrecision(10, 2))
555+
assert(zero.toString === "0.00")
556+
557+
// A compact (long) input takes the Long branch first, where the scale change can exceed
558+
// the Int range. These switch to the BigDecimal branch, so check their results directly.
559+
Seq(
560+
(1L, 1, -40, 10, 2, null),
561+
(0L, 1, -2147483647, 10, 2, "0.00"),
562+
(1L, 1, Int.MinValue, 38, 20, null)).foreach { case (unscaled, p, s, toP, toS, expected) =>
563+
allSupportedRoundModes.foreach { mode =>
564+
val compact = Decimal.createUnsafe(unscaled, p, s)
565+
checkCompact(compact, true)
566+
assert(Option(compact.toPrecision(toP, toS, mode)).map(_.toString).orNull === expected,
567+
s"$unscaled, ($p, $s) -> ($toP, $toS), $mode")
568+
}
569+
}
570+
// These stay in the Long branch, so compare with the same value in BigDecimal form.
571+
Seq(100L, -100L).foreach { unscaled =>
572+
allSupportedRoundModes.foreach { mode =>
573+
val compact = Decimal.createUnsafe(unscaled, 3, 2)
574+
checkCompact(compact, true)
575+
val expanded = Decimal(java.math.BigDecimal.valueOf(unscaled, 2))
576+
checkCompact(expanded, false)
577+
assert(
578+
Option(compact.toPrecision(1, -2147483647, mode)).map(_.toString) ===
579+
Option(expanded.toPrecision(1, -2147483647, mode)).map(_.toString),
580+
s"$unscaled, $mode")
581+
}
582+
}
583+
}
584+
585+
// Values around the bounds of the shortcuts must round as BigDecimal.setScale does.
586+
Seq(false, true).foreach { allowNegativeScale =>
587+
withSQLConf(
588+
SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> allowNegativeScale.toString) {
589+
val targets =
590+
Seq((10, 2), (3, 3), (5, 0)) ++ (if (allowNegativeScale) Seq((5, -3)) else Nil)
591+
for {
592+
// The long mantissas make the scale change large enough to take the shortcuts.
593+
mantissa <- Seq("1", "4.9", "5", "5.1", "9.99", "9.995",
594+
"1." + "0" * 79 + "1", "4." + "9" * 80, "5." + "0" * 80, "5." + "0" * 79 + "1",
595+
"9." + "9" * 80)
596+
exponent <- -8 to 12
597+
sign <- Seq("", "-")
598+
mode <- allSupportedRoundModes
599+
(precision, scale) <- targets
600+
} {
601+
val bd = new java.math.BigDecimal(s"$sign${mantissa}e$exponent")
602+
val rounded = BigDecimal(bd).setScale(scale, mode)
603+
val expected = if (rounded.precision > precision) null else rounded.toString
604+
val d = Decimal(bd).toPrecision(precision, scale, mode)
605+
assert(Option(d).map(_.toString).orNull === expected,
606+
s"$bd, $mode, ($precision, $scale)")
607+
}
608+
}
609+
}
610+
}
531611
}

0 commit comments

Comments
 (0)