Repository navigation
Closed
Labels
area:expressionsExpression evaluationExpression evaluationbugSomething isn't workingSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features
Milestone
Description
Describe the bug
For several arithmetic expressions, Comet's ANSI throw/NULL decision is correct, but the error class, error message, or exception parameters differ from Spark. All of these surface to users who catch or match on Spark error conditions.
- Byte/Short overflow raises the wrong error class.
Add/Subtract/Multiplyon TINYINT/SMALLINT under ANSI: Spark 4.1 raisesBINARY_ARITHMETIC_OVERFLOWcarrying the two operand values and the operator symbol (types/numerics.scala, codegen inarithmetic.scala). Comet raisesARITHMETIC_OVERFLOWwithfrom_type: "integer"(native/spark-expr/src/math_funcs/checked_arithmetic.rs:69-71, 93-96).SparkError::BinaryArithmeticOverflowalready exists (native/common/src/error.rs:87-93) and is wired throughShimSparkErrorConverter, but no kernel constructs it. - Long overflow says "integer overflow".
from_typeis hard-coded to"integer"regardless of Int32 vs Int64 (checked_arithmetic.rs:70, 94). Spark says"long overflow"for BIGINT. The same applies toSumInteger(native/spark-expr/src/agg_funcs/sum_int.rs), where Spark's sum result type is always LONG. - The
try_suggestion is always empty.ShimSparkErrorConverter.scala:118passes""for the alternative, so Comet's message omitsUse `try_add` to tolerate overflow and return NULL instead. UnaryMinusscalar path for Byte/Short passes" caused"as the type (native/spark-expr/src/math_funcs/negative.rs:157, 160), producing"[ARITHMETIC_OVERFLOW] caused overflow."instead of"byte overflow"/"short overflow". The array path uses the correct names.Abspasses Rust type names ("Int8","Int64","Decimal128", ...) asfrom_type(native/spark-expr/src/math_funcs/abs.rs:98, 106, 114, 122, 156, 186), so Comet says"Int64 overflow"where Spark says"long overflow". Contrast withnegative.rs, which uses the Spark names.avgoverflow suggestion:avg_decimal.rsusesdecimal_sum_overflow_error(...); verify the function-name parameter producestry_avg(Spark's message) rather thantry_sum(ShimSparkErrorConverter.scala:123-127buildstry_$functionName).
Steps to reproduce
SET spark.sql.ansi.enabled=true;
SELECT CAST(127 AS TINYINT) + CAST(1 AS TINYINT); -- Spark: BINARY_ARITHMETIC_OVERFLOW; Comet: ARITHMETIC_OVERFLOW "integer overflow"
SELECT CAST(9223372036854775807 AS LONG) + 1L; -- Spark: "long overflow"; Comet: "integer overflow"
SELECT -CAST(-128 AS TINYINT); -- literal/scalar path: Comet message is "[ARITHMETIC_OVERFLOW] caused overflow."
SELECT abs(CAST(-9223372036854775808 AS LONG)); -- Spark: "long overflow"; Comet: "Int64 overflow"Expected behavior
Error classes, type names, and try_ suggestions match Spark exactly. Existing tests pass because they assert message substrings only; assertions on getCondition() / error class would catch these.
Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Possibly related to the Spark 4.2 ANSI test failures in #4967.
Metadata
Metadata
Assignees
Labels
area:expressionsExpression evaluationExpression evaluationbugSomething isn't workingSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features
Activity
docs: document known correctness issues in the compatibility guide (#…
Smallfu666 commentedon Jul 31, 2026
take
fix: skip op for null slots and single pass construction in checked_b…
fix: align ANSI arithmetic overflow error classes, type names, and su…
fix: align ANSI arithmetic overflow error classes, type names, and su…
docs: drop compatibility notes for bugs that are now fixed
docs: drop compatibility notes for bugs that are now fixed (#5154)
Smallfu666 commentedon Aug 10, 2026
I took another look at #5071 after #5162 landed. It looks like item 4 is fixed, but a few of the original issues are still present on curent main:
Absstill uses Arrow/Rust type names for overflow errors, e.g.Int64 overflowinstead of Spark'slong overflow(item 5).integer overflowinstead oflong overflow(item 2).try_add/try_sub/try_mulsuggestions are still missing (item 3).I also rechecked item 6 and that one seems to be fine.
Would you prefer to reopen #5071 for the remaining items, or track them separately as smaller issues?
docs: document known correctness issues in the compatibility guide (a…
docs: drop compatibility notes for bugs that are now fixed (apache#5154)