Skip to content

Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException #6966

Open

Description

@viirya

Describe the bug

Spark parses a string to decimal with new java.math.BigDecimal(str) and then rescales it to the target type with BigDecimal.setScale. For extreme exponents that rescale fails with a raw java.lang.ArithmeticException in every eval mode (ANSI, legacy and try_cast), while the native cast returns a value or NULL. Found while reviewing #6826.

Two shapes:

  1. Positive target scale. '1e-2147483647' cast to DECIMAL(10,2): the parsed scale (2147483647) fits in an int, so the string is valid, but the scale difference to the target is about 2.1e9. Spark fails with BigInteger would overflow supported range. The native cast returns 0.00, because parse_string_to_decimal returns zero whenever the value has to be divided by more than 10^38.
  2. Negative target scale (spark.sql.legacy.allowNegativeScaleOfDecimal=true). '1e-2147483647' cast to DECIMAL(10,-2): the scale difference does not fit in an int, and Spark fails with Underflow. Since fix: do not fail planning on a cast of a literal that fails to evaluate #6826 the native cast reports CAST_INVALID_INPUT under ANSI and NULL otherwise, so both engines fail under ANSI but with different exceptions, and they differ outside ANSI.

On JDK 17 and 21, setScale returns zero for scale differences up to about 1e8 (after computing a very large power of ten) and throws from about 6e8. The exact threshold depends on the JDK's BigInteger limits, so the native cast cannot reproduce it exactly.

Steps to reproduce

CREATE TABLE t(s string) USING parquet;
INSERT INTO t VALUES ('1e-2147483647');
SELECT try_cast(s AS DECIMAL(10,2)) FROM t;

Spark: java.lang.ArithmeticException: BigInteger would overflow supported range. Comet: 0.00.

Expected behavior

To be decided. Spark raising a raw ArithmeticException even from try_cast looks like a Spark-side issue. Comet should at least not silently return a value where Spark fails.

Activity

  1. added
    bugSomething isn't working
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Oct 18, 2026
  2. snmvaughan commented on Nov 5, 2026

    @snmvaughan
    Contributor

    Two details from the #7522 review, found by @comphead with a small Java program and by reading Decimal.set and changePrecision at v4.1.3:

    • The slowdown starts well before the exception. On a release without SPARK-60119, new BigDecimal("1e-50000000").setScale(2, HALF_UP) returns 0.00 after about 12 seconds on JDK 17, while 1e-600000000 throws BigInteger would overflow supported range.
    • The int boundary is off by one. '1e-2147483647' cast to DECIMAL(10,-1) throws Underflow as well, because setScale needs the difference (2147483648) to fit in an int. parse_string_to_decimal computes a scale_adjustment of -2147483648, which fits in an i32, so the native cast returns 0.

    Both gaps close on Spark 4.0.5, 4.1.4 and 4.2.1, where SPARK-60119 makes these cases return 0. #7187 and #7522 cover the negative-scale difference that overflows an int.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions