Repository navigation
Commit c8e6553
authored
fix: count the days partition transform in UTC (#6348)
* fix: count the days partition transform in UTC
CometDays cast timestamps to dates in the session timezone, while CometHours and Iceberg's days transform count in UTC. Spark cannot evaluate the transform, so there is no Spark result to match. Use UTC so the transforms agree with each other and with Iceberg.
Closes #6333.
* test: pin UTC flooring of days for a pre-epoch timestamp
Check days one microsecond before the epoch in an Asia/Tokyo session.
That instant is day -1 in UTC but already 1970-01-01 in Tokyo, so the
session-timezone serde returned 0 and fails this check. The existing
timestamp literal checks pass either way: they run in the default
America/Los_Angeles session, away from a day boundary.
* docs: note where days differs from Iceberg for pre-1970 timestamps1 parent 46d91ef commit c8e6553
2 files changed
Lines changed: 20 additions & 8 deletions
File tree
- spark/src
- main/scala/org/apache/comet/serde
- test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | 25 | | |
27 | 26 | | |
28 | 27 | | |
| |||
822 | 821 | | |
823 | 822 | | |
824 | 823 | | |
825 | | - | |
826 | | - | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
827 | 832 | | |
828 | 833 | | |
829 | 834 | | |
| |||
846 | 851 | | |
847 | 852 | | |
848 | 853 | | |
849 | | - | |
850 | 854 | | |
851 | | - | |
| 855 | + | |
852 | 856 | | |
853 | 857 | | |
854 | 858 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
839 | 839 | | |
840 | 840 | | |
841 | 841 | | |
842 | | - | |
| 842 | + | |
843 | 843 | | |
844 | 844 | | |
845 | 845 | | |
| |||
885 | 885 | | |
886 | 886 | | |
887 | 887 | | |
888 | | - | |
889 | | - | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
890 | 898 | | |
891 | 899 | | |
892 | 900 | | |
| |||
0 commit comments