Repository navigation
Commit 76a4513
authored
fix(spark): match Spark abs overflow errors (apache#24156)
## Which issue does this PR close?
- Closes apache#24145.
## Rationale for this change
When ANSI mode is enabled, `datafusion-spark` reports type-specific
DataFusion errors for integral `abs` overflow. The scalar and array
paths also produce different messages. Spark 4.2 uses one canonical
`ARITHMETIC_OVERFLOW` message for both paths.
## What changes are included in this PR?
- Add a Spark-local checked `abs` array kernel so Spark-specific error
text does not change core DataFusion behavior.
- Use the Spark 4.2 `ARITHMETIC_OVERFLOW` message for scalar and array
overflows.
- Update the existing Spark SQL logic tests for all signed integer
widths and both input shapes.
## Are these changes tested?
Yes. The following checks pass:
- `cargo fmt --all -- --check`
- `cargo clippy -p datafusion-spark --all-targets --all-features -- -D
warnings`
- `cargo test -p datafusion-spark --all-features`
- `cargo test --profile=ci --test sqllogictests -- spark/math/abs.slt`
- `cargo test --profile=ci --test sqllogictests -- math.slt`
- `RUST_BACKTRACE=1 cargo test --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli --workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
## Are there any user-facing changes?
Yes. With `datafusion.execution.enable_ansi_mode = true`, Spark `abs`
overflow errors now use Spark 4.2's canonical `ARITHMETIC_OVERFLOW`
message for scalar and array inputs. There are no public API changes.
---------
Signed-off-by: goutamadwant <[email protected]>1 parent 5c846af commit 76a4513
2 files changed
Lines changed: 40 additions & 33 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | | - | |
| 27 | + | |
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
32 | 37 | | |
33 | 38 | | |
34 | 39 | | |
| |||
91 | 96 | | |
92 | 97 | | |
93 | 98 | | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
| 99 | + | |
101 | 100 | | |
102 | 101 | | |
103 | 102 | | |
| |||
107 | 106 | | |
108 | 107 | | |
109 | 108 | | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
| 109 | + | |
117 | 110 | | |
118 | 111 | | |
119 | 112 | | |
| |||
125 | 118 | | |
126 | 119 | | |
127 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
128 | 133 | | |
129 | 134 | | |
130 | 135 | | |
| |||
142 | 147 | | |
143 | 148 | | |
144 | 149 | | |
145 | | - | |
| 150 | + | |
146 | 151 | | |
147 | 152 | | |
148 | 153 | | |
149 | 154 | | |
150 | 155 | | |
151 | 156 | | |
152 | 157 | | |
153 | | - | |
| 158 | + | |
154 | 159 | | |
155 | 160 | | |
156 | 161 | | |
157 | 162 | | |
158 | 163 | | |
159 | 164 | | |
160 | 165 | | |
161 | | - | |
| 166 | + | |
162 | 167 | | |
163 | 168 | | |
164 | 169 | | |
165 | 170 | | |
166 | 171 | | |
167 | 172 | | |
168 | 173 | | |
169 | | - | |
| 174 | + | |
170 | 175 | | |
171 | 176 | | |
172 | 177 | | |
| |||
182 | 187 | | |
183 | 188 | | |
184 | 189 | | |
185 | | - | |
| 190 | + | |
186 | 191 | | |
187 | 192 | | |
188 | 193 | | |
189 | 194 | | |
190 | 195 | | |
191 | 196 | | |
192 | 197 | | |
193 | | - | |
| 198 | + | |
194 | 199 | | |
195 | 200 | | |
196 | 201 | | |
| |||
361 | 366 | | |
362 | 367 | | |
363 | 368 | | |
364 | | - | |
365 | | - | |
366 | | - | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
367 | 374 | | |
368 | 375 | | |
369 | 376 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
| 129 | + | |
130 | 130 | | |
131 | 131 | | |
132 | | - | |
| 132 | + | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
| 135 | + | |
136 | 136 | | |
137 | 137 | | |
138 | | - | |
| 138 | + | |
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
| |||
0 commit comments