Repository navigation
Commit 6fc7baa
authored
Unify temporal field parsing with DatePart (apache#24906)
## Which issue does this PR close?
- Closes apache#22033.
## Rationale for this change
Temporal functions currently parse field names independently, which
produces inconsistent user-visible behavior. For example, `date_part`
accepts aliases such as `mon`, `mons`, `month`, and `months`, while
`date_trunc` accepts only `month`.
Using Arrow's `DatePart` parser as the common representation makes these
aliases consistent across temporal functions and SQL unparsing.
## What changes are included in this PR?
- Use `DatePart::from_str` directly in `date_part` instead of parsing
through `IntervalUnit` and maintaining additional manual mappings.
- Replace the private `DateTruncGranularity` enum and its string parser
with `DatePart`.
- Restrict `date_trunc` to the `DatePart` variants that represent valid
truncation granularities.
- Update the Spark wrappers to delegate field parsing to the shared
implementation while retaining their function-specific semantic
restrictions.
- Map Spark's Sunday-based `dayofweek` directly to `dow1` instead of
adding one to the result expression.
- Use `DatePart` when canonicalizing temporal fields in the SQL
unparser.
## What is the testing strategy for this PR?
SQL logic tests cover the newly shared aliases, including:
- `date_part`: `QUARTERS`, `ISOWEEK`, `DAYOFYEAR`, and `DAYOFWEEK`
- `date_trunc`: `mon`, `months`, `qtr`, and `yrs`
- Spark wrappers: `QUARTERS`, `DAYOFYEAR`, `MONTHS`, and `MINS`
- Unparser canonicalization of aliases such as `MONS` and `YRS`
The following checks passed:
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- Extended workspace tests with
`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
- Spark datetime SQL logic tests
- `cargo test -p datafusion-spark --lib`
## Are there any user-facing changes?
Yes. Temporal functions now consistently accept the field-name aliases
recognized by Arrow's `DatePart` parser. This broadens accepted input
spellings without changing existing accepted inputs or truncation
behavior.
The Spark wrappers inherit these aliases as well, making them slightly
more permissive than Spark's exact spelling tables.1 parent 9fc7a4d commit 6fc7baa
14 files changed
Lines changed: 245 additions & 272 deletions
File tree
- datafusion
- functions/src/datetime
- spark/src/function/datetime
- sqllogictest/test_files
- datetime
- spark/datetime
- sql/src/unparser
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | 24 | | |
26 | 25 | | |
27 | 26 | | |
| |||
216 | 215 | | |
217 | 216 | | |
218 | 217 | | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
250 | 226 | | |
251 | 227 | | |
252 | 228 | | |
| |||
256 | 232 | | |
257 | 233 | | |
258 | 234 | | |
259 | | - | |
| 235 | + | |
260 | 236 | | |
261 | 237 | | |
262 | 238 | | |
| |||
268 | 244 | | |
269 | 245 | | |
270 | 246 | | |
271 | | - | |
272 | | - | |
| 247 | + | |
| 248 | + | |
273 | 249 | | |
274 | 250 | | |
275 | 251 | | |
276 | | - | |
| 252 | + | |
277 | 253 | | |
278 | 254 | | |
279 | | - | |
280 | | - | |
| 255 | + | |
| 256 | + | |
281 | 257 | | |
282 | 258 | | |
283 | 259 | | |
| |||
336 | 312 | | |
337 | 313 | | |
338 | 314 | | |
339 | | - | |
340 | | - | |
| 315 | + | |
| 316 | + | |
341 | 317 | | |
342 | 318 | | |
343 | 319 | | |
| |||
0 commit comments