Decimal.set fails or is slow when the value scale is far from the target scale

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
    • Component/s: SQL
    • None

      Follow-up of SPARK-60119, which bounds the rescaling in `Decimal.changePrecision`.

      `Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)` still calls `decimal.setScale(scale, ROUND_HALF_UP)` without bounding the scale change. It is used by the JSON, CSV and XML parsers for decimal fields (`JacksonParser`, `UnivocityParser`, `StaxXmlParser`), and also by Avro, JDBC and `ToNumberParser`. With a `DECIMAL(10,2)` schema:

      • `from_json(' {"a": 1e-2147483647}

        ', 'a DECIMAL(10,2)')` and `from_csv('1e-2147483647', 'a DECIMAL(10,2)')` fail with `ArithmeticException: BigInteger would overflow supported range`, so the record is treated as malformed (null in PERMISSIVE, error in FAILFAST), while `1e-10000000` correctly gives 0.00.

      • `1e-10000000` takes ~1.4s per value and `1e10000000` ~5s per value; exponents around 1e8 take ~40s per value.

      Overflow in this method is reported as `NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION` with a `roundedValue`, so the large side needs a decision on what to report without computing the rounded value.

      `Decimal.set(decimal: BigDecimal)` has a similar issue: with `spark.sql.legacy.allowNegativeScaleOfDecimal=false` it calls `decimal.setScale(0)` for a negative scale, which fails for e.g. a `java.math.BigDecimal("1e2147483647")` coming from a non-string input. That method is tracked in SPARK-60132; this ticket covers `Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)`. The string cast path is guarded by the fast-fail in `Decimal.fromString` (see SPARK-60118).

            Assignee:
            L. C. Hsieh
            Reporter:
            L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: