Repository navigation
Commit 4ec76ad
authored
fix(spark): floor instead of truncate in unix_millis/unix_seconds for pre-epoch timestamp (apache#26129)
## Which issue does this PR close?
- Closes apache#26128.
## Rationale for this change
The Spark-compatible `unix_millis` and `unix_seconds` functions return
different results from Spark for timestamps before 1970-01-01 that have
a sub-unit component. Spark floors the result, but DataFusion truncates
toward zero. For example, `unix_millis` of a timestamp 1 µs before the
epoch is `-1` in Spark and `0` in DataFusion. Results for non-negative
timestamps and exact multiples already match.
The cause is that `SparkUnixTimestamp::simplify` rewrites the call as a
cast to a coarser `Timestamp` unit, and Arrow's cast truncates toward
zero. Spark computes these values with `Math.floorDiv`
(`TimestampToLongBase`).
## What changes are included in this PR?
In `SparkUnixTimestamp::simplify`
(`datafusion/spark/src/function/datetime/unix.rs`):
- When the input timestamp unit is finer than the target unit, the raw
`Int64` ticks are divided by the unit ratio (1e3, 1e6 or 1e9). Then `1`
is subtracted when the remainder is negative: `ticks / d - CAST(ticks %
d < 0 AS BIGINT)`. This is an exact floor division over the full `i64`
range, with no intermediate unit conversion, so it adds no new overflow
cases.
- When the input unit is the same as or coarser than the target, the
existing direct cast is kept, so results for those inputs don't change.
- NULL handling doesn't change; all operators involved propagate NULL.
## What is the testing strategy for this PR?
New sqllogictest cases in
`datafusion/sqllogictest/test_files/spark/datetime/unix.slt`:
- **Runtime path:** a table column (not just literals, so constant
folding isn't the only path tested) holding pre-epoch values with and
without a remainder, a positive value and NULL. Each is checked for both
`unix_millis` and `unix_seconds`.
- **Constant-folded path:** the same cases written as literals.
- **Other input units:** `Timestamp(Nanosecond)` to millis and micros,
`Timestamp(Millisecond)` to seconds, and an unchanged conversion to a
finer unit (`Millisecond` to micros).
- **Extremes:** `i64::MIN` microseconds to seconds, and a large
millisecond value to seconds, to guard against overflow.
Expected values for microsecond inputs came from Spark 4.0.0 (pyspark
with `spark.sql.session.timeZone=UTC`), not from this implementation.
Spark has no nanosecond timestamps, so the nanosecond cases use the same
floor rule.
Local runs: all `spark/*` sqllogictests, `cargo test -p
datafusion-spark`, `cargo fmt --all -- --check` and `cargo clippy -p
datafusion-spark --all-targets -- -D warnings` pass
## Are there any user-facing changes?
Yes, as a bug fix. `unix_millis` and `unix_seconds` now return the
Spark-compatible (floored) value for pre-epoch timestamps with a
sub-unit component. No public API changes, so no `api change` label is
needed.
---
AI assistance: I used an AI coding assistant to help find this
divergence (by differential testing against Spark) and to draft the
change. A separate AI review pass caught overflow edge cases, which the
tests now cover. I reproduced the behavior against Spark 4.0.0 myself
and reviewed the change line by line.1 parent d75fd7b commit 4ec76ad
2 files changed
Lines changed: 192 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | | - | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
111 | 112 | | |
112 | 113 | | |
113 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
114 | 133 | | |
115 | 134 | | |
116 | 135 | | |
| |||
131 | 150 | | |
132 | 151 | | |
133 | 152 | | |
134 | | - | |
135 | | - | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
136 | 177 | | |
137 | 178 | | |
138 | 179 | | |
| |||
141 | 182 | | |
142 | 183 | | |
143 | 184 | | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
144 | 191 | | |
145 | 192 | | |
146 | 193 | | |
| |||
150 | 197 | | |
151 | 198 | | |
152 | 199 | | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
0 commit comments