You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException #6966
Spark parses a string to decimal with new java.math.BigDecimal(str) and then rescales it to the target type with BigDecimal.setScale. For extreme exponents that rescale fails with a raw java.lang.ArithmeticException in every eval mode (ANSI, legacy and try_cast), while the native cast returns a value or NULL. Found while reviewing #6826.
Two shapes:
Positive target scale.'1e-2147483647' cast to DECIMAL(10,2): the parsed scale (2147483647) fits in an int, so the string is valid, but the scale difference to the target is about 2.1e9. Spark fails with BigInteger would overflow supported range. The native cast returns 0.00, because parse_string_to_decimal returns zero whenever the value has to be divided by more than 10^38.
Negative target scale (spark.sql.legacy.allowNegativeScaleOfDecimal=true). '1e-2147483647' cast to DECIMAL(10,-2): the scale difference does not fit in an int, and Spark fails with Underflow. Since fix: do not fail planning on a cast of a literal that fails to evaluate #6826 the native cast reports CAST_INVALID_INPUT under ANSI and NULL otherwise, so both engines fail under ANSI but with different exceptions, and they differ outside ANSI.
On JDK 17 and 21, setScale returns zero for scale differences up to about 1e8 (after computing a very large power of ten) and throws from about 6e8. The exact threshold depends on the JDK's BigInteger limits, so the native cast cannot reproduce it exactly.
Steps to reproduce
CREATETABLEt(s string) USING parquet;
INSERT INTO t VALUES ('1e-2147483647');
SELECT try_cast(s ASDECIMAL(10,2)) FROM t;
Spark: java.lang.ArithmeticException: BigInteger would overflow supported range. Comet: 0.00.
Expected behavior
To be decided. Spark raising a raw ArithmeticException even from try_cast looks like a Spark-side issue. Comet should at least not silently return a value where Spark fails.
Two details from the #7522 review, found by @comphead with a small Java program and by reading Decimal.set and changePrecision at v4.1.3:
The slowdown starts well before the exception. On a release without SPARK-60119, new BigDecimal("1e-50000000").setScale(2, HALF_UP) returns 0.00 after about 12 seconds on JDK 17, while 1e-600000000 throws BigInteger would overflow supported range.
The int boundary is off by one. '1e-2147483647' cast to DECIMAL(10,-1) throws Underflow as well, because setScale needs the difference (2147483648) to fit in an int. parse_string_to_decimal computes a scale_adjustment of -2147483648, which fits in an i32, so the native cast returns 0.
Both gaps close on Spark 4.0.5, 4.1.4 and 4.2.1, where SPARK-60119 makes these cases return 0. #7187 and #7522 cover the negative-scale difference that overflows an int.
Activity
Link the remaining string to decimal gap to apache#6966
fix: do not fail planning on a cast of a literal that fails to evalua…
docs: describe SPARK-60119 and the version-number caveat, and share o…
fix: match SPARK-58821 and SPARK-60118 in sequence and string to deci…
fix: match SPARK-58821 and SPARK-60118 in sequence and string to deci…
fix: match SPARK-58821 and SPARK-60118 in sequence and string to deci…
snmvaughan commentedon Nov 5, 2026
Two details from the #7522 review, found by @comphead with a small Java program and by reading
Decimal.setandchangePrecisionat v4.1.3:new BigDecimal("1e-50000000").setScale(2, HALF_UP)returns0.00after about 12 seconds on JDK 17, while1e-600000000throwsBigInteger would overflow supported range.'1e-2147483647'cast toDECIMAL(10,-1)throwsUnderflowas well, becausesetScaleneeds the difference (2147483648) to fit in an int.parse_string_to_decimalcomputes ascale_adjustmentof -2147483648, which fits in ani32, so the native cast returns 0.Both gaps close on Spark 4.0.5, 4.1.4 and 4.2.1, where SPARK-60119 makes these cases return 0. #7187 and #7522 cover the negative-scale difference that overflows an int.