Repository navigation
Commit 5a7420e
## Which issue does this PR close?
- Closes apache#25169
## Rationale for this change
### What a user hits
Build a timestamp series from second-precision bounds. Second and
millisecond precision are common in Parquet, so this is what a series
over a column read from storage looks like.
```sql
SELECT * FROM generate_series(
arrow_cast(TIMESTAMP '2024-01-01 00:00:00', 'Timestamp(Second, None)'),
arrow_cast(TIMESTAMP '2024-01-03 00:00:00', 'Timestamp(Second, None)'),
INTERVAL '1 day');
```
On `main` the call fails, and the message says the argument is not a
timestamp while it prints a timestamp:
```
Error: Error during planning: First argument must be a timestamp or NULL,
got Literal(TimestampSecond(1704067200, None), None)
```
With this PR the same call works:
```
+---------------------+
| value |
+---------------------+
| 2024-01-01T00:00:00 |
| 2024-01-02T00:00:00 |
| 2024-01-03T00:00:00 |
+---------------------+
```
### What changes for a user
`generate_series` and `range` accept all four timestamp precisions, not
nanoseconds alone. The same query written with a second-precision or
millisecond-precision bound now returns a series instead of an error.
Queries that worked before keep working, and the result type does not
change.
Error messages also name the type they rejected, so a reader can act on
them.
### The technical detail
`generate_series` and `range` over timestamps accepted
`Timestamp(Nanosecond, _)` alone. Any other `TimeUnit` failed. The
dispatch in `call_with_args` already routed every
`DataType::Timestamp(_, _)` into `call_timestamp`, and `call_timestamp`
then hard-matched `ScalarValue::TimestampNanosecond`. So a coarser bound
reached a match arm that could not accept it, and fell through to a
message about the wrong thing.
There is no design question about the result precision. The output
schema is already fixed at `Timestamp(Nanosecond, tz)`, so the function
already produced nanoseconds. It refused only coarser *inputs*.
### Field research
Both reference engines accept coarser bounds, so this change moves
DataFusion toward them. I measured both rather than quote their
documentation.
PostgreSQL 17.11 (`postgres:17`):
```
=> SELECT pg_typeof(g), g FROM generate_series('2024-01-01'::timestamp(0), '2024-01-03'::timestamp(0), INTERVAL '1 day') g;
pg_typeof | g
-----------------------------+---------------------
timestamp without time zone | 2024-01-01 00:00:00
timestamp without time zone | 2024-01-02 00:00:00
timestamp without time zone | 2024-01-03 00:00:00
(3 rows)
```
Mixed precisions also work there:
```
=> SELECT g FROM generate_series('2024-01-01'::timestamp(0), '2024-01-03'::timestamp(3), INTERVAL '1 day') g; -- 3 rows
```
DuckDB 1.5.2:
```
D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY); -- 3 rows, type TIMESTAMP
D SELECT * FROM generate_series(TIMESTAMP_MS '2024-01-01', TIMESTAMP_MS '2024-01-03', INTERVAL 1 DAY); -- 3 rows
D SELECT * FROM generate_series(TIMESTAMP_NS '2024-01-01', TIMESTAMP_NS '2024-01-03', INTERVAL 1 DAY); -- 3 rows
D SELECT * FROM range(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY); -- 2 rows
D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_MS '2024-01-03', INTERVAL 1 DAY); -- 3 rows
```
Two points carry over from both engines:
- Neither engine rejects a coarser bound.
- Neither engine lets the bound's unit reach the result type. PostgreSQL
returns `timestamp`, DuckDB returns `TIMESTAMP`, and this PR keeps
`Timestamp(Nanosecond, tz)`.
Neither engine can arbitrate the timezone question below. Both store a
`timestamptz` as an instant with no per-value zone, so both render the
series in the session zone. Arrow puts the zone in the value, so
DataFusion must pick one, and no cross-engine comparison is meaningful
there.
## What changes are included in this PR?
Both timestamp bounds in `call_timestamp`, which `generate_series` and
`range` share, go through a new `timestamp_arg_to_nanos` helper instead
of a hard match on `ScalarValue::TimestampNanosecond`.
- **All four `TimeUnit`s are accepted** and widened to nanoseconds. The
output stays `Timestamp(Nanosecond, tz)`.
- **Overflow is checked.** `Second`, `Millisecond` and `Microsecond`
span far more than an `i64` of nanoseconds, which covers roughly 1677 to
2262. The widening is a `checked_mul`, so an out-of-range bound is a
plan error that names the argument, the value and the window:
```
First argument for generate_series is out of range of nanosecond
timestamps:
9223372036854775807 (Timestamp(Second, None)) is outside
1677-09-21T00:12:43.145224192 to 2262-04-11T23:47:16.854775807
```
An unchecked multiply wraps in silence in a release build and emits a
plausible but wrong series. No panic was reachable from SQL through this
path before: `call_timestamp` did no arithmetic on the bounds, and the
`DATE` overload already used `checked_mul`. The new multiplication is
checked so that stays true now that coarser units get through.
- **Mixed precisions work.** The two bounds are read independently, so
`generate_series(ts_second, ts_micro, INTERVAL '1 day')` is fine. A
timestamp denotes an instant whatever unit holds it, and both sides
reach the same nanosecond scale before the comparison.
- **Timezone handling does not change, but a comment now explains it.**
The output timezone still comes from the start argument. Two different
timezones on the two bounds are deliberately not an error: an Arrow
timezone changes how an instant is rendered, not which instant it is.
The start's zone is the one kept because it also anchors the calendar
arithmetic that advances the series. Month and day steps apply in local
time, so they follow that zone's DST rules.
- **Error messages name the type they rejected** instead of a dump of
the whole `Expr`, for example `Second argument for generate_series must
be a TIMESTAMP or NULL constant, got Int64`. This covers the timestamp
and `DATE` overloads. A non-literal argument gets the same message, with
the expression after "got".
Two limits stay, and both are worth a mention.
- The step argument still accepts `Interval(MonthDayNano)` alone. That
is not reachable as a limit from SQL, because `arrow_cast` to
`Interval(DayTime)` and `Interval(YearMonth)` is itself unimplemented.
So it is left alone rather than widened on speculation.
- A `DATE` bound and a `TIMESTAMP` bound still cannot be mixed.
PostgreSQL and DuckDB both accept that mix. It is out of scope for a
precision fix, and the new message at least names the type now.
## What is the testing strategy for this PR?
All coverage is in sqllogictest, in a new section of
`datafusion/sqllogictest/test_files/table_functions.slt` beside the
existing timestamp-range tests:
- all four units, for `generate_series` and for `range`
- a sub-second step over second-precision bounds
- mixed-precision bounds in both directions
- `arrow_typeof` assertions that the output is `Timestamp(Nanosecond,
tz)` whatever the input unit
- bounds with two different timezones, in both orders (the swapped order
returns fewer rows, which shows that the bounds compare as instants)
- typed and untyped NULL bounds
- the edge of the nanosecond range for `Second`, `Millisecond` and
`Microsecond`, in both directions, and one unit past each edge
- the error messages for a wrong type and for a non-literal argument
```
cargo test -p datafusion-sqllogictest --test sqllogictests -- table_functions
cargo clippy -p datafusion-functions-table --all-targets -- -D warnings
```
## Are there any user-facing changes?
Yes. Both of them widen behaviour rather than break it.
- `generate_series` and `range` accept
`Timestamp(Second|Millisecond|Microsecond, _)` bounds, which errored
before. Queries that worked before keep working, and the result type
does not change.
- Several plan error messages are reworded to name the rejected data
type. There are no public API changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 56cad07 commit 5a7420e
2 files changed
Lines changed: 268 additions & 27 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
534 | 534 | | |
535 | 535 | | |
536 | 536 | | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
537 | 585 | | |
538 | 586 | | |
539 | 587 | | |
| |||
673 | 721 | | |
674 | 722 | | |
675 | 723 | | |
676 | | - | |
677 | | - | |
678 | | - | |
679 | | - | |
680 | | - | |
681 | | - | |
682 | | - | |
683 | | - | |
684 | | - | |
685 | | - | |
686 | | - | |
687 | | - | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
688 | 728 | | |
689 | | - | |
690 | | - | |
691 | | - | |
692 | | - | |
693 | | - | |
694 | | - | |
695 | | - | |
696 | | - | |
697 | | - | |
698 | | - | |
699 | | - | |
| 729 | + | |
| 730 | + | |
700 | 731 | | |
701 | 732 | | |
702 | 733 | | |
703 | 734 | | |
704 | 735 | | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
705 | 743 | | |
706 | 744 | | |
707 | | - | |
| 745 | + | |
| 746 | + | |
708 | 747 | | |
709 | 748 | | |
710 | 749 | | |
| |||
765 | 804 | | |
766 | 805 | | |
767 | 806 | | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
768 | 814 | | |
769 | 815 | | |
770 | | - | |
| 816 | + | |
| 817 | + | |
771 | 818 | | |
772 | 819 | | |
773 | 820 | | |
| |||
783 | 830 | | |
784 | 831 | | |
785 | 832 | | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
786 | 840 | | |
787 | 841 | | |
788 | | - | |
| 842 | + | |
| 843 | + | |
789 | 844 | | |
790 | 845 | | |
791 | 846 | | |
| |||
803 | 858 | | |
804 | 859 | | |
805 | 860 | | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
806 | 868 | | |
807 | 869 | | |
808 | | - | |
| 870 | + | |
| 871 | + | |
809 | 872 | | |
810 | 873 | | |
811 | 874 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
259 | 259 | | |
260 | 260 | | |
261 | 261 | | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
262 | 440 | | |
263 | 441 | | |
264 | 442 | | |
| |||
0 commit comments