You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
negative: ANSI overflow check reads null slots and can raise spurious overflow errors #5093
In native/spark-expr/src/math_funcs/negative.rs, the ANSI-mode overflow check (check_overflow! macro) loops over Int8/16/32/64 and interval arrays comparing every slot's value to MIN, then calls neg_wrapping. The scan reads value(i) without checking validity, so a MIN value sitting in a null slot (arbitrary bytes left behind by filter/slice/FFI) raises a spurious ARITHMETIC_OVERFLOW error even though the row is null.
It is also a two-pass re-implementation of arrow's checked neg kernel, which negates with overflow detection in a single pass and correctly skips null slots.
Steps to reproduce
Construct an Int64 array where a null slot's underlying value is i64::MIN (for example by masking a valid i64::MIN entry to null via a validity buffer) and evaluate negative() on it in ANSI mode. The expression errors; Spark returns null for the null row and succeeds.
Expected behavior
Null slots must be ignored by the overflow check. Replacing the ANSI path with arrow::compute::kernels::numeric::neg fixes this, since the kernel skips invalid slots.
Additional context
Notes for the fix:
The Spark error message per type must be reconstructed in a map_err; if the exact offending value is required for Int8/Int16 wording, a rescan on the (cold) error path can recover it.
IntervalUnit::MonthDayNano is deliberately not overflow-checked today, and arrow's checked neg would check it, so that type must keep neg_wrapping unconditionally.
Legacy (non-ANSI) mode keeps neg_wrapping as today.
Found during an audit of native code that replicates existing arrow-rs kernels. The corresponding null-slot behavior is already pinned correctly for add in checked_arithmetic.rs tests.
Activity
Smallfu666 commentedon Jul 31, 2026
take
fix: skip null slots when checking overflow in unary negation (apache…
fix: skip null slots when checking overflow in unary negation (apache…
fix: skip null slots when checking overflow in unary negation (apache…
fix: skip null slots when checking overflow in unary negation (#5162)
fix: [branch-1.0] backport four bug fixes for 1.0.1 (#5321)
fix: skip null slots when checking overflow in unary negation (apache…