You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Three native ANSI error paths raise raw Arrow errors instead of SparkError, so they bypass the JSON error converter (SparkErrorConverter.convertToSparkException bails when the message is not JSON) and surface as generic CometNativeException / CometQueryExecutionException instead of SparkArithmeticException with the proper error class, SQLSTATE, and query context.
Wide-decimal arithmetic overflow under ANSI: native/spark-expr/src/wide_decimal_binary_expr.rs:343 returns ArrowError::ComputeError("Arithmetic overflow"). Note that native/core/src/execution/planner.rs:519-526 deliberately skips the wrapping CheckOverflow (which would produce the correct NUMERIC_VALUE_OUT_OF_RANGE) when the child is a WideDecimalBinaryExpr, so this Arrow error is the only signal the user gets.
Decimal divide/remainder by zero under ANSI: native/spark-expr/src/math_funcs/div.rs:96, 123 raises ArrowError::ComputeError(divide_by_zero_error().to_string()). The message text contains [DIVIDE_BY_ZERO] but the exception type is wrong. The existing test (CometExpressionSuite.scala:2984-2988) only asserts on message text, so it passes.
Decimal-to-decimal cast overflow under ANSI: no dedicated arm in cast_array; the fall-through DataFusion cast with safe=false propagates ArrowError::InvalidArgumentError("... too large to store in a Decimal128 of precision ...") instead of Spark's [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] ... cannot be represented as Decimal(p, s). This divergence is explicitly special-cased in CometCastSuite.scala:2402-2405; fixing it allows deleting that special case.
Steps to reproduce
SETspark.sql.ansi.enabled=true;
-- decimal div by zero (correct message text, wrong exception type):SELECT CAST(1ASDECIMAL(10,2)) / CAST(0ASDECIMAL(10,2));
-- decimal to decimal overflow (wrong class and message):SELECT CAST(CAST(12345.678ASDECIMAL(20,3)) ASDECIMAL(4,3));
Expected behavior
All three paths raise SparkError variants (NumericValueOutOfRange, DivideByZero) so the converter produces genuine Spark exceptions, as already happens for e.g. integer overflow and array-index errors.
Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1.
Activity
docs: document known correctness issues in the compatibility guide (#…
peterxcli commentedon Jul 31, 2026
take
docs: drop compatibility notes for bugs that are now fixed
docs: drop compatibility notes for bugs that are now fixed (#5154)
docs: document known correctness issues in the compatibility guide (a…
docs: drop compatibility notes for bugs that are now fixed (apache#5154)
address review: drop stale docs bullets, harden CheckedBinaryExpr unw…
fix: native ansi-errors raised as spark error (apache#5169)