Repository navigation
Commit 262936e
fix(spark): derive pmod's decimal result type from the declared arguments (#24409)
## Which issue does this PR close?
- Closes #23895.
## Rationale for this change
`pmod` reports a wider decimal type than Spark does. Spark derives the
result
type of `pmod` with `Pmod.resultDecimalType`, which applies the
`Remainder`
rule to the *declared* argument types:
```text
scale = max(s1, s2)
precision = min(p1 - s1, p2 - s2) + scale
```
For `pmod(decimal(3,1), decimal(2,1))` Spark reports `decimal(2,1)`, but
DataFusion reported `Decimal128(3, 1)`.
The cause is coercion. `SparkPmod` used `Signature::numeric`, which
collapses
both arguments to a common decimal before `return_type` runs, so the two
precisions `return_type` saw were already equal and the rule degenerated
to the
input precision.
## What changes are included in this PR?
- `SparkPmod` moves to `Signature::user_defined` with a `coerce_types`
that
leaves a decimal/decimal argument pair intact, following the precedent
set by
`try_sum`. Every other argument combination keeps the coercion
`Signature::numeric` performed — including its null handling, where a
null
argument is skipped and an all-null call falls back to `Float64` — so
only the
decimal pair changes behaviour.
- `return_type` applies Spark's `Pmod.resultDecimalType` rule for
decimal
arguments and is unchanged for everything else.
- Because Spark's result type is *narrower* than the dividend, the
operands
cannot be cast to it before the remainder is taken without overflowing
the
dividend — `pmod(99.9::decimal(3,1), 2.5::decimal(2,1))` returns
`decimal(2,1)`, which cannot hold `99.9`. `spark_pmod` therefore widens
the
operands to a common computation type, takes the remainder there, and
narrows
the result afterwards.
### Overflow semantics
The remainder is bounded by the divisor, but the result type only
carries
`min(p1 - s1, p2 - s2)` integer digits, so the narrowing step can
overflow when
the divisor is wider than the dividend:
```sql
-- result type decimal(3,1), true value 9999.8
SELECT pmod(-0.1::decimal(3,1), 9999.9::decimal(5,1));
```
Spark wraps decimal arithmetic in `CheckOverflow(nullOnOverflow =
!ansiEnabled)`,
so the narrowing cast returns NULL in legacy mode and raises under ANSI.
The
widening cast uses `safe: false` in both modes, since the computation
type is
chosen to fit both operands and a silent NULL there would hide a real
bug.
### Scope
Deliberately limited to `pmod` over two `Decimal128` arguments, which is
what
#23895 reports. Three adjacent gaps are left alone and are happy to be
follow-ups
if you would rather see them here:
- **`SparkMod`** has the same bug, since `Remainder.resultDecimalType`
is the
same rule. It is the easier of the two: arrow's `Op::Rem` already
computes
`min(p1-s1, p2-s2) + max(s1, s2)`, so `mod` needs only the
`coerce_types`
pass-through and the matching `return_type`, with no widen/narrow step.
- **Decimal mixed with integer** still diverges:
`pmod(2.5::decimal(3,1), 3)`
reports `Decimal128(21, 1)` where Spark casts INT to `decimal(10,0)` and
reports `decimal(3,1)`.
- **`Decimal256`, `Decimal64` and `Decimal32`** pairs fall through to
the
previous behaviour. Spark has no equivalent of the wider types.
## Are these changes tested?
Yes.
`datafusion/sqllogictest/test_files/spark/math/pmod.slt` gains:
- four `arrow_typeof` assertions covering equal scales, differing
precisions,
differing scales, and the narrowing case;
- a value test for `pmod(99.9::decimal(3,1), 2.5::decimal(2,1))`, the
case that
would regress if the operands were narrowed before the remainder;
- the overflow case above, asserted as NULL in legacy mode and as an
error in
the ANSI block; and
- null-argument cases pinning the coercion parity described above.
`modulus.rs` gains a unit test for `pmod_decimal_result_type` covering
the rule
directly, independent of the planner.
The existing `pmod` and `mod` value tests are unchanged and still pass.
Verified
locally: `cargo test -p datafusion-spark --all-features` (279 passed),
all 244
`spark/` sqllogictest files, `cargo clippy --all-targets --all-features
-D
warnings`, and `cargo fmt --all --check`.
## Are there any user-facing changes?
Yes, and it is the point of the fix: `pmod` over two decimals now
reports the
same result type Spark does. Values that fit the Spark result type are
unchanged. Values that do not fit were previously returned at the wider
type and
are now NULL (legacy) or an error (ANSI), matching Spark. No public API
changes.
---------
Co-authored-by: Jeffrey Vo <[email protected]>1 parent 50cab14 commit 262936e
2 files changed
Lines changed: 228 additions & 25 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
18 | 20 | | |
| 21 | + | |
19 | 22 | | |
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
26 | | - | |
| 29 | + | |
27 | 30 | | |
28 | | - | |
| 31 | + | |
29 | 32 | | |
30 | | - | |
| 33 | + | |
| 34 | + | |
31 | 35 | | |
32 | 36 | | |
33 | 37 | | |
| |||
101 | 105 | | |
102 | 106 | | |
103 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
104 | 125 | | |
105 | 126 | | |
106 | 127 | | |
107 | 128 | | |
108 | 129 | | |
109 | 130 | | |
| 131 | + | |
110 | 132 | | |
111 | 133 | | |
112 | 134 | | |
113 | | - | |
114 | | - | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
115 | 168 | | |
116 | 169 | | |
117 | 170 | | |
118 | 171 | | |
119 | 172 | | |
120 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
121 | 189 | | |
122 | 190 | | |
123 | 191 | | |
| |||
182 | 250 | | |
183 | 251 | | |
184 | 252 | | |
185 | | - | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
186 | 266 | | |
187 | 267 | | |
188 | 268 | | |
| |||
203 | 283 | | |
204 | 284 | | |
205 | 285 | | |
206 | | - | |
207 | | - | |
208 | | - | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
209 | 296 | | |
210 | 297 | | |
211 | 298 | | |
212 | | - | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
213 | 304 | | |
214 | 305 | | |
215 | 306 | | |
| |||
558 | 649 | | |
559 | 650 | | |
560 | 651 | | |
561 | | - | |
| 652 | + | |
| 653 | + | |
562 | 654 | | |
563 | 655 | | |
564 | 656 | | |
| |||
581 | 673 | | |
582 | 674 | | |
583 | 675 | | |
584 | | - | |
| 676 | + | |
| 677 | + | |
585 | 678 | | |
586 | 679 | | |
587 | 680 | | |
| |||
625 | 718 | | |
626 | 719 | | |
627 | 720 | | |
628 | | - | |
| 721 | + | |
| 722 | + | |
629 | 723 | | |
630 | 724 | | |
631 | 725 | | |
| |||
683 | 777 | | |
684 | 778 | | |
685 | 779 | | |
686 | | - | |
| 780 | + | |
| 781 | + | |
687 | 782 | | |
688 | 783 | | |
689 | 784 | | |
| |||
718 | 813 | | |
719 | 814 | | |
720 | 815 | | |
721 | | - | |
| 816 | + | |
| 817 | + | |
722 | 818 | | |
723 | 819 | | |
724 | 820 | | |
| |||
737 | 833 | | |
738 | 834 | | |
739 | 835 | | |
740 | | - | |
| 836 | + | |
| 837 | + | |
741 | 838 | | |
742 | 839 | | |
743 | 840 | | |
| |||
750 | 847 | | |
751 | 848 | | |
752 | 849 | | |
753 | | - | |
| 850 | + | |
| 851 | + | |
754 | 852 | | |
755 | 853 | | |
756 | 854 | | |
| |||
772 | 870 | | |
773 | 871 | | |
774 | 872 | | |
775 | | - | |
| 873 | + | |
| 874 | + | |
776 | 875 | | |
777 | 876 | | |
778 | 877 | | |
| |||
788 | 887 | | |
789 | 888 | | |
790 | 889 | | |
791 | | - | |
| 890 | + | |
| 891 | + | |
792 | 892 | | |
793 | 893 | | |
794 | 894 | | |
| |||
802 | 902 | | |
803 | 903 | | |
804 | 904 | | |
805 | | - | |
| 905 | + | |
| 906 | + | |
806 | 907 | | |
807 | 908 | | |
808 | 909 | | |
| |||
825 | 926 | | |
826 | 927 | | |
827 | 928 | | |
828 | | - | |
| 929 | + | |
| 930 | + | |
829 | 931 | | |
830 | 932 | | |
831 | 933 | | |
| |||
840 | 942 | | |
841 | 943 | | |
842 | 944 | | |
843 | | - | |
| 945 | + | |
| 946 | + | |
844 | 947 | | |
845 | 948 | | |
846 | 949 | | |
| |||
858 | 961 | | |
859 | 962 | | |
860 | 963 | | |
861 | | - | |
| 964 | + | |
| 965 | + | |
862 | 966 | | |
863 | 967 | | |
864 | 968 | | |
| |||
881 | 985 | | |
882 | 986 | | |
883 | 987 | | |
884 | | - | |
| 988 | + | |
| 989 | + | |
885 | 990 | | |
886 | 991 | | |
887 | 992 | | |
| |||
919 | 1024 | | |
920 | 1025 | | |
921 | 1026 | | |
922 | | - | |
| 1027 | + | |
| 1028 | + | |
923 | 1029 | | |
924 | 1030 | | |
925 | 1031 | | |
| |||
935 | 1041 | | |
936 | 1042 | | |
937 | 1043 | | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
938 | 1075 | | |
0 commit comments