Casting a string with an exponent near Int.MaxValue to decimal throws ArithmeticException instead of NULL / NUMERIC_OUT_OF_SUPPORTED_RANGE

XMLWordPrintableJSON

      Casting a string whose exponent is near Int.MaxValue to a decimal type fails with a raw java.lang.ArithmeticException under all cast modes, instead of returning NULL (try_cast / legacy cast) or raising NUMERIC_OUT_OF_SUPPORTED_RANGE (ANSI cast):

      SELECT try_cast('1e2147483647' AS DECIMAL(10,2));
      
      java.lang.ArithmeticException: BigInteger would overflow supported range
        at java.base/java.math.BigInteger.reportOverflow(BigInteger.java:1178)
        at java.base/java.math.BigInteger.pow(BigInteger.java:2627)
        at java.base/java.math.BigDecimal.bigTenToThe(BigDecimal.java:4375)
        at java.base/java.math.BigDecimal.bigMultiplyPowerTen(BigDecimal.java:4510)
        at java.base/java.math.BigDecimal.setScale(BigDecimal.java:2936)
        at java.base/java.math.BigDecimal.setScale(BigDecimal.java:3003)
        at scala.math.BigDecimal.setScale(BigDecimal.scala:592)
        at org.apache.spark.sql.types.Decimal.set(Decimal.scala:155)
        at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:610)
        at org.apache.spark.sql.types.Decimal$.fromString(Decimal.scala:656)
      

      Decimal.fromString / fromStringANSI fast-fail when the number of integral digits (precision - scale of the parsed java.math.BigDecimal) exceeds 38. That value is computed as an Int; for '1e2147483647' it is 1 - (-2147483647), which overflows to a negative number, so the check is skipped and converting the BigDecimal to Decimal throws.

      Other affected inputs: '-1E+2147483647', '12e2147483647', '1.5e2147483647', and '1e2147483648' (on JDK 21+, where it parses; throws "Underflow"). Relatedly, the check also counts integral digits for zero, so a zero with a large exponent such as '0e39' or '0e100' is rejected as out of range (NULL, or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI) although it fits any decimal type, while '0e37' returns 0.00. '0e2147483647' returns 0.00 only because of the Int overflow above. The code dates back to SPARK-35841 and is the same on all maintained branches.

            Assignee:
            L. C. Hsieh
            Reporter:
            L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: