Repository navigation
Description
Describe the bug
Under datafusion.execution.enable_ansi_mode = true, datafusion-spark's abs
correctly raises on integral overflow, but the message is one DataFusion
invents rather than Spark's:
DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648)
Spark 4.2.0 raises:
[ARITHMETIC_OVERFLOW] overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22003
The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.
This one matters more than a typical message mismatch because abs.rs is the
in-repo precedent that new datafusion-spark ANSI implementations are pointed
at. Its wording gets copied.
Cross-version note
The message text is version dependent, and 4.2.0 is the version that changed it:
| Spark | Message for abs(-2147483648) |
|---|---|
| 3.5.8 | [ARITHMETIC_OVERFLOW] integer overflow. ... |
| 4.0.4 | [ARITHMETIC_OVERFLOW] integer overflow. ... |
| 4.1.3 | [ARITHMETIC_OVERFLOW] integer overflow. ... |
| 4.2.0 | [ARITHMETIC_OVERFLOW] overflow. ... |
Spark 4.2.0 added a canonicalization step to
ExecutionErrors.arithmeticOverflowError that rewrites any message matching
\w+ overflow to plain overflow, so the JDK strings integer overflow and
long overflow and the MathUtils strings byte overflow and
short overflow all collapse to the same text. The reason given in the source
is JIT hot-throw behavior, JDK-8367990.
Before 4.2.0 the width appeared in the message, so DataFusion's current text is
closer in spirit to those versions than to 4.2.0, though equal to none of them.
DataFusion has no mechanism for version-specific expectations, tracked by
apache/datafusion-spark#24.
To Reproduce
set datafusion.execution.enable_ansi_mode = true;
select abs((-2147483648)::INT);
-- DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648)The array path produces a second spelling, because the macro behind it
stringifies the Arrow array type rather than the scalar type:
select abs(a) FROM (VALUES (-2147483647::INT), ((-2147483648)::INT)) AS t(a);
-- DataFusion error: Arrow error: Compute error: Int32Array overflow on abs(-2147483648)Both spellings are asserted today in
datafusion/sqllogictest/test_files/spark/math/abs.slt, four of each. Spark's
message does not vary by input shape.
Expected behavior
A message matching Spark 4.2.0's ARITHMETIC_OVERFLOW text, and the same
message on the scalar and array paths.
Scope
This is the abs half of the repository-wide question raised in
apache/datafusion-spark#27. That issue notes that
changing pmod alone would leave the crate less consistent while abs keeps
its own wording. The two should be decided together.
The two paths in abs are not equally easy, and the array one is the reason
this is worth its own issue rather than a drive-by fix.
- Scalar path. The message is built by
scalar_compute_op!in
datafusion/spark/src/function/math/abs.rs. It is local to the spark crate
and is twoformat!calls. - Array path. The message is built by
make_try_abs_function!in
datafusion/functions/src/math/abs.rs, whichdatafusion-sparkimports and
reuses. Core DataFusion's ownabsraises through the same macro, so the
text cannot be changed there without changing the error message core
DataFusion users see. Matching Spark on the array path meansdatafusion-spark
needs its own kernel rather than borrowing core's.
That second point generalizes past abs: wherever a spark function reuses a
core macro or kernel, Spark-specific error text is not available without first
un-sharing the code.
Relevant code
datafusion/spark/src/function/math/abs.rsdatafusion/functions/src/math/abs.rs,make_try_abs_function!datafusion/sqllogictest/test_files/spark/math/abs.slt
Surfaced by the audit-datafusion-spark-expression skill.
Activity
goutamadwant commentedon Aug 7, 2026
take
goutamadwant commentedon Aug 7, 2026
@andygrove I have a fix with #24156.
It adds a Spark-local checked array kernel so scalar and array ANSI overflow paths return the Spark 4.2
ARITHMETIC_OVERFLOWmessage without changing core DataFusion behavior. Let me know if any suggestions. thanks!fix(spark): match Spark abs overflow errors (apache#24156)