Repository navigation
Commit 2da78d6
authored
fix(spark): correct mod ANSI zero divisor and negative zero handling (apache#23987)
## Which issue does this PR close?
- Closes apache#23894
## Rationale for this change
`datafusion-spark`'s `mod` routes through the shared `try_rem` helper in
`datafusion/spark/src/function/math/modulus.rs`, which has two gaps in
its
zero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the
issue):
1. **ANSI mode, floating-point divisor.** `try_rem` delegates to Arrow's
`rem`
kernel in ANSI mode, but Arrow only reports division by zero for integer
and
decimal types. Floating-point divisors follow IEEE 754 and quietly
produce
`NaN`, while Spark raises `REMAINDER_BY_ZERO` for a zero divisor of any
numeric type:
```sql
set datafusion.execution.enable_ansi_mode = true;
SELECT mod(10.5::float8, 0.0::float8); -- NaN, Spark raises
```
2. **`-0.0` divisor, both modes.** The legacy path nulls out zero
divisors via
`eq(right, 0)`, but Arrow's floating-point comparisons use a total order
in
which `-0.0` is distinct from `0.0`, so a `-0.0` divisor goes
unrecognised.
Spark's `isZero` is a numeric comparison and treats `-0.0` as zero:
```sql
SELECT mod(10.5::float8, -0.0::float8); -- NaN, Spark returns NULL
(legacy)
```
## What changes are included in this PR?
`try_rem` now detects zero divisors itself, mirroring the shape apache#23898
established for `pmod`:
- A new `is_zero` helper counts `-0.0` as zero for the floating-point
types
(via a `negative_zero` companion, same as apache#23898).
- In ANSI mode, any row with a zero divisor raises
`ArrowError::DivideByZero`
— the same error Arrow's `rem` already produces for integers today, so
the
message stays uniform across types. The check is masked by the validity
of
the dividend because Spark's remainder expressions are null intolerant:
a
NULL dividend short-circuits to NULL before the divisor is validated, so
`mod(NULL, 0)` must return NULL rather than raise.
- Both modes substitute NULL for zero divisors before calling Arrow's
`rem`,
so the kernel never sees a zero divisor: legacy mode gets NULLs, and
ANSI
mode has already raised on the rows that required it.
Note on overlap with apache#23898: that PR rewrites `pmod` to no longer use
`try_rem` and adds identical `is_zero`/`negative_zero` helpers. This PR
is
independent of it — `mod` is fixed either way — but whichever lands
second
should dedupe the helpers in a rebase. Until apache#23898 lands, `pmod` also
picks
up the `-0.0` and ANSI floating-point zero-divisor fixes through the
shared
helper.
Out of scope: #23897 (reproducing Spark's exact ANSI error text) is a
repository-wide error-message policy question, as noted in that issue.
## Are these changes tested?
Yes:
- New unit tests in `modulus.rs`: ANSI floating-point zero divisor
raises;
`-0.0` divisor returns NULL in legacy mode and raises in ANSI mode; a
NULL
dividend with a zero divisor returns NULL in ANSI mode (integer and
float).
- New sqllogictest cases in `spark/math/mod.slt` covering the same
behavior at
SQL level.
- Verified `cargo test -p datafusion-spark`, the `spark/math/mod.slt`
and
`spark/math/pmod.slt` sqllogictests, `./dev/rust_lint.sh`, and the
extended
workspace suite (ci profile with
`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`)
— all green.
## Are there any user-facing changes?
Only the bug fixes, and only for the Spark `mod`/`pmod` functions: in
ANSI
mode a floating-point zero divisor now raises instead of returning NaN,
and a
`-0.0` divisor is treated as zero in both modes (NULL in legacy mode, an
error
in ANSI mode), matching Spark. No API changes.1 parent bf1c17a commit 2da78d6
3 files changed
Lines changed: 302 additions & 19 deletions
File tree
- datafusion
- spark/src/function/math
- sqllogictest/test_files/spark/math
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | | - | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
31 | 60 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
36 | 66 | | |
37 | | - | |
38 | | - | |
| 67 | + | |
| 68 | + | |
39 | 69 | | |
40 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
41 | 78 | | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
52 | 87 | | |
| 88 | + | |
53 | 89 | | |
54 | 90 | | |
55 | 91 | | |
| |||
418 | 454 | | |
419 | 455 | | |
420 | 456 | | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
421 | 552 | | |
422 | 553 | | |
423 | 554 | | |
| |||
645 | 776 | | |
646 | 777 | | |
647 | 778 | | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
648 | 875 | | |
649 | 876 | | |
650 | 877 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
163 | 175 | | |
164 | 176 | | |
165 | 177 | | |
| |||
170 | 182 | | |
171 | 183 | | |
172 | 184 | | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
173 | 204 | | |
174 | 205 | | |
175 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
77 | 83 | | |
78 | 84 | | |
79 | 85 | | |
| |||
84 | 90 | | |
85 | 91 | | |
86 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
87 | 112 | | |
88 | 113 | | |
89 | 114 | | |
| |||
0 commit comments