Repository navigation
Commit d2bd93c
authored
fix(spark): accept STRING and TIMESTAMP start dates in next_day (apache#26127)
## Which issue does this PR close?
- Closes apache#23889.
## Rationale for this change
Spark's `NextDay` casts its first argument to DATE, so
`next_day('2015-07-23', 'Mon')` and `next_day(TIMESTAMP '2015-07-23
12:12:12', 'Mon')` both work. `datafusion-spark` declared `Exact(Date32,
Utf8)` and failed at planning for STRING and TIMESTAMP inputs.
## What changes are included in this PR?
- `next_day` now uses a coercible signature: the first argument is
implicitly cast to DATE from STRING or TIMESTAMP, and the day-of-week is
any string type (numeric is still coerced to string).
- The kernel is unchanged.
- In `next_day.slt`, three existing `query error` expectations changed
to the new planner messages (arity and non-date first argument), and the
unparseable-string case now returns a cast error. They still fail at
planning or execution as before.
## What is the testing strategy for this PR?
New cases in `spark/datetime/next_day.slt`: string, string with a time
part, timestamp literal, string column with NULL, timestamp column, and
`LargeUtf8`/`Utf8View` day-of-week. I compared the expected results with
`pyspark==4.0.0`, ANSI on and off.
## Are there any user-facing changes?
`next_day` accepts more argument types. No API changes.
## Known limitations / unknowns
- An unparseable date string always errors. Spark returns NULL with ANSI
off and errors with ANSI on. This looks tied to the ANSI work in #23890
and apache#23929.
- I did not verify timestamps with time zones near midnight against
Spark's session-time-zone cast.
- apache#23892 touches the same files, so I expect a small rebase if it merges
first.
- AI assistance was used to implement this. I reviewed and understand
the change, and checked the results against Spark.
---------
Signed-off-by: Harsh Ande <[email protected]>1 parent 80dda10 commit d2bd93c
2 files changed
Lines changed: 103 additions & 9 deletions
File tree
- datafusion
- spark/src/function/datetime
- sqllogictest/test_files/spark/datetime
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | | - | |
13 | | - | |
| 13 | + | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | | - | |
32 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
33 | 50 | | |
34 | 51 | | |
35 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
55 | 66 | | |
56 | 67 | | |
57 | 68 | | |
| |||
87 | 98 | | |
88 | 99 | | |
89 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
0 commit comments