Casting a string whose exponent is near Int.MaxValue to a decimal type fails with a raw java.lang.ArithmeticException under all cast modes, instead of returning NULL (try_cast / legacy cast) or raising NUMERIC_OUT_OF_SUPPORTED_RANGE (ANSI cast):
SELECT try_cast('1e2147483647' AS DECIMAL(10,2));
java.lang.ArithmeticException: BigInteger would overflow supported range at java.base/java.math.BigInteger.reportOverflow(BigInteger.java:1178) at java.base/java.math.BigInteger.pow(BigInteger.java:2627) at java.base/java.math.BigDecimal.bigTenToThe(BigDecimal.java:4375) at java.base/java.math.BigDecimal.bigMultiplyPowerTen(BigDecimal.java:4510) at java.base/java.math.BigDecimal.setScale(BigDecimal.java:2936) at java.base/java.math.BigDecimal.setScale(BigDecimal.java:3003) at scala.math.BigDecimal.setScale(BigDecimal.scala:592) at org.apache.spark.sql.types.Decimal.set(Decimal.scala:155) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:610) at org.apache.spark.sql.types.Decimal$.fromString(Decimal.scala:656)
Decimal.fromString / fromStringANSI fast-fail when the number of integral digits (precision - scale of the parsed java.math.BigDecimal) exceeds 38. That value is computed as an Int; for '1e2147483647' it is 1 - (-2147483647), which overflows to a negative number, so the check is skipped and converting the BigDecimal to Decimal throws.
Other affected inputs: '-1E+2147483647', '12e2147483647', '1.5e2147483647', and '1e2147483648' (on JDK 21+, where it parses; throws "Underflow"). Relatedly, the check also counts integral digits for zero, so a zero with a large exponent such as '0e39' or '0e100' is rejected as out of range (NULL, or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI) although it fits any decimal type, while '0e37' returns 0.00. '0e2147483647' returns 0.00 only because of the Int overflow above. The code dates back to SPARK-35841 and is the same on all maintained branches.
- is related to
-
SPARK-60132 Bound the integral-digit count of BigDecimal values in Decimal.set to avoid raw ArithmeticException for huge exponents
-
- Open
-
- links to