Repository navigation
Commit dbf9d05
[fix](arrow) encode DATE in the proleptic Gregorian calendar at format boundaries
### What problem does this PR solve?
Issue Number: close apache#67366
Problem Summary:
Doris follows MySQL's calendar, in which year 0 is not a leap year: 0000-02-29
does not exist and calc_daynr() therefore numbers 0000-01-01 .. 0000-02-28 one
day ahead of the proleptic Gregorian calendar. Arrow date32, Parquet DATE, ORC
DATE and the Iceberg spec all define their day ordinal in the proleptic
Gregorian calendar, where year 0 IS a leap year, and the two numberings coincide
only from 0000-03-01 onwards.
Every format boundary subtracted or added the epoch day number directly, so all
59 dates in 0000-01-01 .. 0000-02-28 were shipped one day late: 0000-01-01 left
Doris as -719527, which Arrow renders as 0000-01-02, while the MySQL protocol --
which carries year/month/day verbatim and never needs a calendar -- reported
0000-01-01 for the same value.
Add daynr_to_epoch_days()/epoch_days_to_daynr() and route the boundaries through
them: Arrow date32/date64 read and write (which Parquet export shares), the
Parquet DATE reader, and the ORC DATE reader and writer. Both helpers compile to
branchless code, so the write path costs two extra instructions and the read
path is absorbed by the following get_date_from_daynr(). The ORC reader now
passes the file's day value straight through instead of laundering it through
date_day_offset_dict, which also removes a silent fallback that decoded any
out-of-dictionary value as 1900-01-01. The Arrow date32 reader now checks the
result of get_date_from_daynr() instead of discarding it, and 0000-02-29 -- which
exists in the proleptic Gregorian calendar but not in Doris -- is rejected rather
than silently decoding as 0000-02-28.
The Iceberg partition transforms had a second, independent defect: they were
derived from datetime_diff(), which rounds towards zero, while Iceberg floors
(DateTimeUtil.convertDays/convertMicros evaluate one unit later for a negative
input and then subtract one). Every pre-1970 value that was not exactly on a unit
boundary got the wrong partition -- 1969-12-31 23:59:59 landed in the same day and
hour partition as 1970-01-01 00:00:00, and year()/month() reported 0 and -6 for
1969-06-15 where Iceberg requires -1 and -7. Rewrite year, month, day, hour and
bucket for both DATE and TIMESTAMP, and make human_hour() floor as well so a
negative hour ordinal renders as 1969-12-31-23 rather than 1970-01-01--1.
Verified against cctz over all 3,652,424 representable Doris dates, and against
the Apache Iceberg reference implementation (iceberg-api 1.10.1, after
reproducing the spec's own Appendix B test vector 2017-11-16 -> -653330422).
A new regression suite writes identical boundary rows from Doris and from Spark
into two identically partitioned Iceberg tables and compares the resulting
partition metadata, so the transforms are checked against the reference
implementation end to end rather than against Doris's own expectations.
### Release note
DATE values in 0000-01-01 .. 0000-02-28 now cross every file and wire boundary
with the proleptic Gregorian day ordinal the format specifications require:
- Arrow date32/date64, Parquet DATE and ORC DATE written by Doris encode
0000-01-01 as -719528 instead of -719527. Files written by an older Doris are
read one day later than before (0000-01-01 comes back as 0000-01-02), except
for an old-encoded 0000-02-28 (ordinal -719469), which is the proleptic-only
0000-02-29 and has no Doris DATE: an ORC scan fails, a Parquet scan fails in
strict mode and returns NULL otherwise, and Arrow input fails. Rewrite such
files from the source table to repair them.
- Arrow date32 input (stream load `format=arrow`, the ADBC catalog, the
remote-Doris catalog, the Paimon native reader, Python UDTFs) now reports an
out-of-range day instead of silently storing a zero date.
- Federating an older and a newer Doris through the remote-Doris catalog shifts
year-zero dates by one day in the mixed-version window, and fails on an
old-encoded 0000-02-28.
- An ORC DATE ordinal outside 0000-01-01 .. 9999-12-31, or equal to the
proleptic-only 0000-02-29, now fails the scan instead of silently decoding as
1900-01-01.
- Iceberg `year`, `month`, `day`, `hour` and `bucket` partition values now floor
towards negative infinity and use the proleptic ordinal, so pre-1970 and
year-zero rows land in the partitions the Iceberg specification defines
(1969-12-31 23:59:59 is hour -1, not 0). Data files and manifests written by an
older Doris keep the old values, so filtered scans in Doris and in Spark keep
skipping those rows until the files are rewritten; run `rewrite_data_files`
after every BE has been upgraded. During a rolling upgrade old and new BEs
write both variants into the same table.
### Check List (For Author)
- Test: Regression test + Unit Test
- BE UT: data_type_datev2_serde_calendar_test, vdatetime_value_test (all
3,652,424 representable dates against cctz), partition_transformers_test,
orc_reader_test
- Regression: export_p0/outfile/test_outfile_date_year_zero,
arrow_flight_sql_p0/test_date_year_zero,
external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary
- Behavior changed: Yes (see the Release note: the file/wire encoding of 59
year-zero DATE values and the Iceberg pre-1970 partition values change)
- Does this need documentation: No (no in-repo or website document describes the
DATE ordinal encoding or the Iceberg transform arithmetic)
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY1 parent ddbaaab commit dbf9d05
12 files changed
Lines changed: 1082 additions & 52 deletions
File tree
- be
- src
- core
- data_type_serde
- value
- exec/sink/writer/iceberg
- test
- core
- data_type_serde
- value
- exec/sink/writer/iceberg
- format_v2/orc
- fe/fe-core/src/main/java/org/apache/doris/nereids/stats
- regression-test
- data/external_table_p0/iceberg/write
- suites
- arrow_flight_sql_p0
- export_p0/outfile
- external_table_p0/iceberg/write
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
| 26 | + | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| |||
41 | 43 | | |
42 | 44 | | |
43 | 45 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | 46 | | |
48 | 47 | | |
49 | | - | |
50 | | - | |
51 | 48 | | |
52 | 49 | | |
53 | 50 | | |
| |||
64 | 61 | | |
65 | 62 | | |
66 | 63 | | |
67 | | - | |
68 | 64 | | |
69 | 65 | | |
70 | | - | |
71 | | - | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| |||
77 | 78 | | |
78 | 79 | | |
79 | 80 | | |
80 | | - | |
81 | | - | |
| 81 | + | |
| 82 | + | |
82 | 83 | | |
83 | 84 | | |
84 | 85 | | |
| |||
216 | 217 | | |
217 | 218 | | |
218 | 219 | | |
219 | | - | |
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
223 | | - | |
224 | | - | |
225 | | - | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
226 | 226 | | |
227 | 227 | | |
228 | 228 | | |
| |||
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
260 | | - | |
| 260 | + | |
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
| |||
280 | 280 | | |
281 | 281 | | |
282 | 282 | | |
| 283 | + | |
283 | 284 | | |
284 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
285 | 290 | | |
286 | 291 | | |
287 | 292 | | |
| |||
401 | 406 | | |
402 | 407 | | |
403 | 408 | | |
404 | | - | |
| 409 | + | |
405 | 410 | | |
406 | 411 | | |
407 | 412 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1770 | 1770 | | |
1771 | 1771 | | |
1772 | 1772 | | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
1773 | 1803 | | |
1774 | 1804 | | |
1775 | 1805 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
47 | 50 | | |
48 | 51 | | |
49 | 52 | | |
| |||
89 | 92 | | |
90 | 93 | | |
91 | 94 | | |
92 | | - | |
93 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
94 | 99 | | |
95 | 100 | | |
96 | 101 | | |
97 | 102 | | |
98 | | - | |
| 103 | + | |
99 | 104 | | |
100 | 105 | | |
101 | 106 | | |
| |||
615 | 620 | | |
616 | 621 | | |
617 | 622 | | |
618 | | - | |
| 623 | + | |
619 | 624 | | |
620 | 625 | | |
621 | 626 | | |
| |||
820 | 825 | | |
821 | 826 | | |
822 | 827 | | |
823 | | - | |
824 | | - | |
825 | | - | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
826 | 832 | | |
827 | 833 | | |
828 | 834 | | |
| |||
889 | 895 | | |
890 | 896 | | |
891 | 897 | | |
892 | | - | |
893 | | - | |
894 | | - | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
895 | 902 | | |
896 | 903 | | |
897 | 904 | | |
| |||
958 | 965 | | |
959 | 966 | | |
960 | 967 | | |
961 | | - | |
962 | | - | |
963 | | - | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
964 | 972 | | |
965 | 973 | | |
966 | 974 | | |
| |||
1027 | 1035 | | |
1028 | 1036 | | |
1029 | 1037 | | |
1030 | | - | |
1031 | | - | |
1032 | | - | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
1033 | 1042 | | |
1034 | 1043 | | |
1035 | 1044 | | |
| |||
1096 | 1105 | | |
1097 | 1106 | | |
1098 | 1107 | | |
1099 | | - | |
1100 | | - | |
1101 | | - | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
1102 | 1112 | | |
1103 | 1113 | | |
1104 | 1114 | | |
| |||
1170 | 1180 | | |
1171 | 1181 | | |
1172 | 1182 | | |
1173 | | - | |
1174 | | - | |
1175 | | - | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
1176 | 1187 | | |
1177 | 1188 | | |
1178 | 1189 | | |
| |||
1243 | 1254 | | |
1244 | 1255 | | |
1245 | 1256 | | |
1246 | | - | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
1247 | 1261 | | |
1248 | | - | |
| 1262 | + | |
1249 | 1263 | | |
1250 | 1264 | | |
1251 | 1265 | | |
| |||
0 commit comments