-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major
-
None
-
Affects Version/s: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
-
Component/s: SQL
-
None
Follow-up of SPARK-60119, which bounds the rescaling in `Decimal.changePrecision`.
`Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)` still calls `decimal.setScale(scale, ROUND_HALF_UP)` without bounding the scale change. It is used by the JSON, CSV and XML parsers for decimal fields (`JacksonParser`, `UnivocityParser`, `StaxXmlParser`), and also by Avro, JDBC and `ToNumberParser`. With a `DECIMAL(10,2)` schema:
- `from_json('
{"a": 1e-2147483647}
', 'a DECIMAL(10,2)')` and `from_csv('1e-2147483647', 'a DECIMAL(10,2)')` fail with `ArithmeticException: BigInteger would overflow supported range`, so the record is treated as malformed (null in PERMISSIVE, error in FAILFAST), while `1e-10000000` correctly gives 0.00.
- `1e-10000000` takes ~1.4s per value and `1e10000000` ~5s per value; exponents around 1e8 take ~40s per value.
Overflow in this method is reported as `NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION` with a `roundedValue`, so the large side needs a decision on what to report without computing the rounded value.
`Decimal.set(decimal: BigDecimal)` has a similar issue: with `spark.sql.legacy.allowNegativeScaleOfDecimal=false` it calls `decimal.setScale(0)` for a negative scale, which fails for e.g. a `java.math.BigDecimal("1e2147483647")` coming from a non-string input. That method is tracked in SPARK-60132; this ticket covers `Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)`. The string cast path is guarded by the fast-fail in `Decimal.fromString` (see SPARK-60118).
- is related to
-
SPARK-60132 Bound the integral-digit count of BigDecimal values in Decimal.set to avoid raw ArithmeticException for huge exponents
-
- Open
-