You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit eaed2f0
Browse filesBrowse the repository at this point in the historyBrowse files
[SPARK-60119][4.2][SQL] Skip rescaling in Decimal.changePrecision when the result is known from the magnitude
### What changes were proposed in this pull request?
Backport of #59335 to branch-4.2.
The cherry-pick conflicted only on `CastUtils.java`, which does not exist on this branch: `Cast` calls `QueryExecutionErrors.cannotChangeDecimalPrecisionError` directly in both the interpreted and generated code. So instead of removing that method and switching `CastUtils`, this backport keeps it and makes it delegate to `DataTypeErrors.cannotChangeDecimalPrecisionError`, which gives the same error message. The rest of the change is identical to #59335.
In the BigDecimal branch of `Decimal.changePrecision`, when the scale change exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits (`precision - scale`, computed in Long) is checked before calling `setScale`:
- If the value is nonzero and has more integral digits than the target type allows, return `false` (overflow) directly. Rounding cannot bring it back into range.
- If the value is below 0.01 ulp of the target scale, the result depends only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is rescaled instead. This gives the same result for every rounding mode.
The Long branch of `changePrecision` now computes the scale change in Long, so it no longer wraps and throws `ArrayIndexOutOfBoundsException` from `POW_10(diff)` when negative scales are allowed.
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose scale is below -1000 (only possible under `spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of `toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a String. Other values are rendered as before. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy, now delegates to `DataTypeErrors.cannotChangeDecimalPrecisionError`.
### Why are the changes needed?
`setScale` takes time that grows with the scale change, and throws once 10^change exceeds BigInteger's range or the change exceeds the Int range:
```sql
SELECT CAST('1e-2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: BigInteger would overflow supported range (LEGACY, ANSI and TRY)
SELECT CAST('1e-100000000' AS DECIMAL(10,2));
-- 0.00, after ~40s
SET spark.sql.legacy.allowNegativeScaleOfDecimal=true;
SELECT CAST('1e2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: Underflow
```
This branch already has the related fix SPARK-60118, so backporting this keeps decimal casts consistent with master.
### Does this PR introduce _any_ user-facing change?
Yes, the same as #59335:
- Casting a string whose value is far below the target scale's ulp now returns the rounded value (e.g. 0.00) instead of throwing `ArithmeticException`, and returns quickly.
- With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string with a huge exponent returns NULL (non-ANSI, try_cast) or throws `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException: Underflow`, and a decimal stored as a long no longer throws `ArrayIndexOutOfBoundsException` for a scale change beyond the Int range. In that error, a value whose scale is below -1000 is shown in scientific notation.
No input that previously produced a value changes its result.
### How was this patch tested?
The tests from #59335 in `DecimalSuite`, `CastSuiteBase`, `CastWithAnsiOnSuite` and `CastWithAnsiOffSuite` (also run through `TryCastSuite`). On this branch with JDK 17, {'4.3': '512 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`, plus `UdfSerializationSuite` (16 tests)', '4.2': '410 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.1': '403 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`', '4.0': '354 tests across `DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`'} pass, and scalastyle (`sql-api`, `catalyst`, `catalyst` tests) and the `sql/api` scalafmt check pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes#59513 from viirya/SPARK-60119-4.2.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
0 commit comments