Repository navigation
Commit c18a192
fix: truncate scalar timestamps in their own unit in date_trunc (apache#25498)
## Which issue does this PR close?
- Closes apache#25432.
## Rationale for this change
`process_scalar` always went through `general_date_trunc`, which
converts the value to nanoseconds before truncating. A
`Timestamp(Second)`, `Timestamp(Millisecond)` or
`Timestamp(Microsecond)` value outside the nanosecond range (after 2262
or before 1677) therefore failed with `Timestamp ... out of range`.
`process_array` does not convert for the fine granularities: when
`granularity.is_fine_granularity()`, or the value has no timezone and
the granularity is `hour` or `day`, it truncates in the input's own unit
with plain arithmetic. So the same value succeeded in a column and
failed as a scalar:
```sql
SELECT date_trunc('second', to_timestamp_seconds(10000000000));
-- Execution error: Timestamp 10000000000 out of range
SELECT date_trunc('second', to_timestamp_seconds(ts)) FROM (VALUES (10000000000)) AS t(ts);
-- 2286-11-20T17:46:40
```
## What changes are included in this PR?
The scalar path now takes the same route as the array path:
- `truncates_in_input_unit` is the condition `process_array` already
used, now shared by both paths.
- `fine_granularity_unit` is the unit table from
`general_date_trunc_array_fine_granularity`, moved out so both paths
read it.
- `date_trunc_fine_granularity` truncates one value in its own unit,
with the checked subtraction and error message the array path already
used for values near `i64::MIN`. The array path's slow branch now calls
it too.
For those granularities, a scalar and a one-row column now run the same
function, so they accept the same values and give the same result.
Everything else (`week`, `month`, `quarter`, `year`, and `hour`/`day`
with a timezone) still goes through `general_date_trunc` on both paths,
so it is unchanged and still limited to the nanosecond range for both
scalars and columns.
One existing test changes. `date_trunc('hour',
arrow_cast(9223372036854775807, 'Timestamp(Second, None)'))` was
expected to fail with `Timestamp 9223372036854775807 out of range`. That
test comes from apache#22262, whose goal was an error instead of a panic, and
a column holding that value already succeeds. As a scalar it now
succeeds too, so the test uses `week`, which still converts to
nanoseconds on both paths and so still covers the error.
## What is the testing strategy for this PR?
- New unit test
`scalar_and_array_accept_timestamps_beyond_nanosecond_range` evaluates
`date_trunc` on a scalar and on a one-row column with the same value,
for all three coarser-than-nanosecond units, with and without a
timezone, and for values before 1677 and after 2262, and checks that
both succeed with the same result.
- New sqllogictest queries in `datetime/timestamps.slt` cover the
issue's scalar and column queries, plus millisecond and microsecond
scalars.
- With the scalar change disabled, the new unit test fails, and the
three new scalar queries fail with the error from the issue (`Timestamp
10000000000 out of range`, etc.), while the column query passes as it
does on `main`.
- `cargo test -p datafusion-functions` (384 passed), the full
sqllogictest suite (520 files), `cargo fmt` and `cargo clippy -p
datafusion-functions --all-targets --all-features -- -D warnings` pass.
## Are there any user-facing changes?
`date_trunc` on a scalar timestamp outside the nanosecond range now
returns the same result as on a column, instead of an out-of-range
error, for `microsecond`, `millisecond`, `second` and `minute`, and for
`hour` and `day` without a timezone. No API changes.
---------
Co-authored-by: Claude Sonnet 5 <[email protected]>
Co-authored-by: Gabriel <[email protected]>1 parent bb81cb8 commit c18a192
2 files changed
Lines changed: 204 additions & 44 deletions
File tree
- datafusion
- functions/src/datetime
- sqllogictest/test_files/datetime
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
| 275 | + | |
283 | 276 | | |
284 | 277 | | |
285 | 278 | | |
| |||
301 | 294 | | |
302 | 295 | | |
303 | 296 | | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
308 | 313 | | |
309 | 314 | | |
310 | 315 | | |
| |||
756 | 761 | | |
757 | 762 | | |
758 | 763 | | |
759 | | - | |
760 | | - | |
761 | | - | |
762 | | - | |
763 | | - | |
764 | | - | |
765 | | - | |
766 | | - | |
767 | | - | |
768 | | - | |
769 | | - | |
770 | | - | |
771 | | - | |
772 | | - | |
773 | | - | |
774 | | - | |
775 | | - | |
776 | | - | |
777 | | - | |
778 | | - | |
779 | | - | |
780 | | - | |
781 | | - | |
782 | | - | |
| 764 | + | |
783 | 765 | | |
784 | 766 | | |
785 | 767 | | |
| |||
797 | 779 | | |
798 | 780 | | |
799 | 781 | | |
800 | | - | |
801 | | - | |
802 | | - | |
803 | | - | |
804 | | - | |
805 | | - | |
806 | | - | |
| 782 | + | |
807 | 783 | | |
808 | 784 | | |
809 | 785 | | |
| |||
815 | 791 | | |
816 | 792 | | |
817 | 793 | | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
818 | 845 | | |
819 | 846 | | |
820 | 847 | | |
| |||
1434 | 1461 | | |
1435 | 1462 | | |
1436 | 1463 | | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
| 1488 | + | |
| 1489 | + | |
| 1490 | + | |
| 1491 | + | |
| 1492 | + | |
| 1493 | + | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
| 1497 | + | |
| 1498 | + | |
| 1499 | + | |
| 1500 | + | |
| 1501 | + | |
| 1502 | + | |
| 1503 | + | |
| 1504 | + | |
| 1505 | + | |
| 1506 | + | |
| 1507 | + | |
| 1508 | + | |
| 1509 | + | |
| 1510 | + | |
| 1511 | + | |
| 1512 | + | |
| 1513 | + | |
| 1514 | + | |
| 1515 | + | |
| 1516 | + | |
| 1517 | + | |
| 1518 | + | |
| 1519 | + | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
| 1527 | + | |
| 1528 | + | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
| 1550 | + | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
| 1571 | + | |
| 1572 | + | |
| 1573 | + | |
1437 | 1574 | | |
1438 | 1575 | | |
1439 | 1576 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2740 | 2740 | | |
2741 | 2741 | | |
2742 | 2742 | | |
| 2743 | + | |
| 2744 | + | |
| 2745 | + | |
| 2746 | + | |
| 2747 | + | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
| 2751 | + | |
| 2752 | + | |
| 2753 | + | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
| 2763 | + | |
| 2764 | + | |
| 2765 | + | |
2743 | 2766 | | |
2744 | 2767 | | |
2745 | | - | |
| 2768 | + | |
2746 | 2769 | | |
2747 | 2770 | | |
2748 | 2771 | | |
| |||
0 commit comments