Skip to content

[Bug] abs reports a DataFusion-specific ANSI overflow message instead of Spark's ARITHMETIC_OVERFLOW text #24145

Closed

Description

@andygrove

Describe the bug

Under datafusion.execution.enable_ansi_mode = true, datafusion-spark's abs
correctly raises on integral overflow, but the message is one DataFusion
invents rather than Spark's:

DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648)

Spark 4.2.0 raises:

[ARITHMETIC_OVERFLOW] overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22003

The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.

This one matters more than a typical message mismatch because abs.rs is the
in-repo precedent that new datafusion-spark ANSI implementations are pointed
at. Its wording gets copied.

Cross-version note

The message text is version dependent, and 4.2.0 is the version that changed it:

Spark Message for abs(-2147483648)
3.5.8 [ARITHMETIC_OVERFLOW] integer overflow. ...
4.0.4 [ARITHMETIC_OVERFLOW] integer overflow. ...
4.1.3 [ARITHMETIC_OVERFLOW] integer overflow. ...
4.2.0 [ARITHMETIC_OVERFLOW] overflow. ...

Spark 4.2.0 added a canonicalization step to
ExecutionErrors.arithmeticOverflowError that rewrites any message matching
\w+ overflow to plain overflow, so the JDK strings integer overflow and
long overflow and the MathUtils strings byte overflow and
short overflow all collapse to the same text. The reason given in the source
is JIT hot-throw behavior, JDK-8367990.

Before 4.2.0 the width appeared in the message, so DataFusion's current text is
closer in spirit to those versions than to 4.2.0, though equal to none of them.
DataFusion has no mechanism for version-specific expectations, tracked by
apache/datafusion-spark#24.

To Reproduce

set datafusion.execution.enable_ansi_mode = true;

select abs((-2147483648)::INT);
-- DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648)

The array path produces a second spelling, because the macro behind it
stringifies the Arrow array type rather than the scalar type:

select abs(a) FROM (VALUES (-2147483647::INT), ((-2147483648)::INT)) AS t(a);
-- DataFusion error: Arrow error: Compute error: Int32Array overflow on abs(-2147483648)

Both spellings are asserted today in
datafusion/sqllogictest/test_files/spark/math/abs.slt, four of each. Spark's
message does not vary by input shape.

Expected behavior

A message matching Spark 4.2.0's ARITHMETIC_OVERFLOW text, and the same
message on the scalar and array paths.

Scope

This is the abs half of the repository-wide question raised in
apache/datafusion-spark#27. That issue notes that
changing pmod alone would leave the crate less consistent while abs keeps
its own wording. The two should be decided together.

The two paths in abs are not equally easy, and the array one is the reason
this is worth its own issue rather than a drive-by fix.

  • Scalar path. The message is built by scalar_compute_op! in
    datafusion/spark/src/function/math/abs.rs. It is local to the spark crate
    and is two format! calls.
  • Array path. The message is built by make_try_abs_function! in
    datafusion/functions/src/math/abs.rs, which datafusion-spark imports and
    reuses. Core DataFusion's own abs raises through the same macro, so the
    text cannot be changed there without changing the error message core
    DataFusion users see. Matching Spark on the array path means datafusion-spark
    needs its own kernel rather than borrowing core's.

That second point generalizes past abs: wherever a spark function reuses a
core macro or kernel, Spark-specific error text is not available without first
un-sharing the code.

Relevant code

  • datafusion/spark/src/function/math/abs.rs
  • datafusion/functions/src/math/abs.rs, make_try_abs_function!
  • datafusion/sqllogictest/test_files/spark/math/abs.slt

Surfaced by the audit-datafusion-spark-expression skill.

Activity

  1. added
    bugSomething isn't working
    sparkChanges to datafusion-spark crate
    on Aug 6, 2026
  2. goutamadwant commented on Aug 7, 2026

    @goutamadwant
    Contributor

    take

  3. goutamadwant commented on Aug 7, 2026

    @goutamadwant
    Contributor

    @andygrove I have a fix with #24156.

    It adds a Spark-local checked array kernel so scalar and array ANSI overflow paths return the Spark 4.2 ARITHMETIC_OVERFLOW message without changing core DataFusion behavior. Let me know if any suggestions. thanks!

  4. added a commit that references this issue on Aug 19, 2026
    76a4513
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingsparkChanges to datafusion-spark crate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions