Skip to content

Commit dbf9d05

Browse files
morningmanclaude
andcommitted
[fix](arrow) encode DATE in the proleptic Gregorian calendar at format boundaries
### What problem does this PR solve? Issue Number: close apache#67366 Problem Summary: Doris follows MySQL's calendar, in which year 0 is not a leap year: 0000-02-29 does not exist and calc_daynr() therefore numbers 0000-01-01 .. 0000-02-28 one day ahead of the proleptic Gregorian calendar. Arrow date32, Parquet DATE, ORC DATE and the Iceberg spec all define their day ordinal in the proleptic Gregorian calendar, where year 0 IS a leap year, and the two numberings coincide only from 0000-03-01 onwards. Every format boundary subtracted or added the epoch day number directly, so all 59 dates in 0000-01-01 .. 0000-02-28 were shipped one day late: 0000-01-01 left Doris as -719527, which Arrow renders as 0000-01-02, while the MySQL protocol -- which carries year/month/day verbatim and never needs a calendar -- reported 0000-01-01 for the same value. Add daynr_to_epoch_days()/epoch_days_to_daynr() and route the boundaries through them: Arrow date32/date64 read and write (which Parquet export shares), the Parquet DATE reader, and the ORC DATE reader and writer. Both helpers compile to branchless code, so the write path costs two extra instructions and the read path is absorbed by the following get_date_from_daynr(). The ORC reader now passes the file's day value straight through instead of laundering it through date_day_offset_dict, which also removes a silent fallback that decoded any out-of-dictionary value as 1900-01-01. The Arrow date32 reader now checks the result of get_date_from_daynr() instead of discarding it, and 0000-02-29 -- which exists in the proleptic Gregorian calendar but not in Doris -- is rejected rather than silently decoding as 0000-02-28. The Iceberg partition transforms had a second, independent defect: they were derived from datetime_diff(), which rounds towards zero, while Iceberg floors (DateTimeUtil.convertDays/convertMicros evaluate one unit later for a negative input and then subtract one). Every pre-1970 value that was not exactly on a unit boundary got the wrong partition -- 1969-12-31 23:59:59 landed in the same day and hour partition as 1970-01-01 00:00:00, and year()/month() reported 0 and -6 for 1969-06-15 where Iceberg requires -1 and -7. Rewrite year, month, day, hour and bucket for both DATE and TIMESTAMP, and make human_hour() floor as well so a negative hour ordinal renders as 1969-12-31-23 rather than 1970-01-01--1. Verified against cctz over all 3,652,424 representable Doris dates, and against the Apache Iceberg reference implementation (iceberg-api 1.10.1, after reproducing the spec's own Appendix B test vector 2017-11-16 -> -653330422). A new regression suite writes identical boundary rows from Doris and from Spark into two identically partitioned Iceberg tables and compares the resulting partition metadata, so the transforms are checked against the reference implementation end to end rather than against Doris's own expectations. ### Release note DATE values in 0000-01-01 .. 0000-02-28 now cross every file and wire boundary with the proleptic Gregorian day ordinal the format specifications require: - Arrow date32/date64, Parquet DATE and ORC DATE written by Doris encode 0000-01-01 as -719528 instead of -719527. Files written by an older Doris are read one day later than before (0000-01-01 comes back as 0000-01-02), except for an old-encoded 0000-02-28 (ordinal -719469), which is the proleptic-only 0000-02-29 and has no Doris DATE: an ORC scan fails, a Parquet scan fails in strict mode and returns NULL otherwise, and Arrow input fails. Rewrite such files from the source table to repair them. - Arrow date32 input (stream load `format=arrow`, the ADBC catalog, the remote-Doris catalog, the Paimon native reader, Python UDTFs) now reports an out-of-range day instead of silently storing a zero date. - Federating an older and a newer Doris through the remote-Doris catalog shifts year-zero dates by one day in the mixed-version window, and fails on an old-encoded 0000-02-28. - An ORC DATE ordinal outside 0000-01-01 .. 9999-12-31, or equal to the proleptic-only 0000-02-29, now fails the scan instead of silently decoding as 1900-01-01. - Iceberg `year`, `month`, `day`, `hour` and `bucket` partition values now floor towards negative infinity and use the proleptic ordinal, so pre-1970 and year-zero rows land in the partitions the Iceberg specification defines (1969-12-31 23:59:59 is hour -1, not 0). Data files and manifests written by an older Doris keep the old values, so filtered scans in Doris and in Spark keep skipping those rows until the files are rewritten; run `rewrite_data_files` after every BE has been upgraded. During a rolling upgrade old and new BEs write both variants into the same table. ### Check List (For Author) - Test: Regression test + Unit Test - BE UT: data_type_datev2_serde_calendar_test, vdatetime_value_test (all 3,652,424 representable dates against cctz), partition_transformers_test, orc_reader_test - Regression: export_p0/outfile/test_outfile_date_year_zero, arrow_flight_sql_p0/test_date_year_zero, external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary - Behavior changed: Yes (see the Release note: the file/wire encoding of 59 year-zero DATE values and the Iceberg pre-1970 partition values change) - Does this need documentation: No (no in-repo or website document describes the DATE ordinal encoding or the Iceberg transform arithmetic) Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
1 parent ddbaaab commit dbf9d05

12 files changed

Lines changed: 1082 additions & 52 deletions

File tree

‎be/src/core/data_type_serde/data_type_datev2_serde.cpp‎

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,9 @@
2121
#include <cctz/time_zone.h>
2222
#include <fmt/core.h>
2323

24+
#include <algorithm>
2425
#include <cstdint>
26+
#include <limits>
2527
#include <vector>
2628

2729
#include "common/config.h"
@@ -41,13 +43,8 @@
4143

4244
namespace doris {
4345

44-
// This number represents the number of days from 0000-01-01 to 1970-01-01.
45-
static constexpr int32_t date_threshold = 719528;
46-
4746
namespace {
4847

49-
constexpr int32_t DORIS_DATE_EPOCH_DAYNR = 719528;
50-
5148
Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
5249
const OrcDecodedColumnView& orc_view) {
5350
const auto* orc_batch = dynamic_cast<const ::orc::LongVectorBatch*>(orc_view.batch);
@@ -64,11 +61,15 @@ Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
6461
orc_serde_utils::orc_decode_row_count(orc_view.rows, orc_view.selected_rows);
6562
std::vector<int32_t> date_values;
6663
date_values.resize(output_rows);
67-
auto& date_dict = date_day_offset_dict::get();
6864
for (size_t row = 0; row < output_rows; ++row) {
6965
const auto source_row = orc_serde_utils::orc_source_row_at(row, orc_view.selected_rows);
70-
const auto date = date_dict[cast_set<int>(orc_batch->data[source_row])];
71-
date_values[row] = cast_set<int32_t>(date.daynr() - DORIS_DATE_EPOCH_DAYNR);
66+
// ORC DATE is days since 1970-01-01 in the proleptic Gregorian calendar, the same encoding
67+
// `decode_date_orc_values`'s consumer expects. Clamp instead of narrowing blindly so a
68+
// corrupt file surfaces as a range error in `decode_parquet_date()` rather than wrapping.
69+
const int64_t file_days = orc_batch->data[source_row];
70+
date_values[row] = static_cast<int32_t>(
71+
std::clamp<int64_t>(file_days, std::numeric_limits<int32_t>::min(),
72+
std::numeric_limits<int32_t>::max()));
7273
}
7374
view.values = reinterpret_cast<const uint8_t*>(date_values.data());
7475
RETURN_IF_ERROR(orc_serde_utils::read_decoded_values(serde, column, &view));
@@ -77,8 +78,8 @@ Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
7778

7879
Status decode_parquet_date(int32_t encoded_date, DateV2Value<DateV2ValueType>* value) {
7980
DORIS_CHECK(value != nullptr);
80-
const int64_t day_number = static_cast<int64_t>(encoded_date) + date_threshold;
81-
if (day_number < 0 || !value->get_date_from_daynr(static_cast<uint64_t>(day_number))) {
81+
const int64_t day_number = epoch_days_to_daynr(encoded_date);
82+
if (day_number == 0 || !value->get_date_from_daynr(static_cast<uint64_t>(day_number))) {
8283
return Status::DataQualityError("Parquet DATE value is out of range");
8384
}
8485
return Status::OK();
@@ -216,13 +217,12 @@ Status DataTypeDateV2SerDe::write_column_to_arrow(const IColumn& column, const N
216217
const auto& col_data = static_cast<const ColumnDateV2&>(column).get_data();
217218
auto& date32_builder = assert_cast<arrow::Date32Builder&>(*array_builder);
218219
for (size_t i = start; i < end; ++i) {
219-
auto daynr = col_data[i].daynr() - date_threshold;
220220
if (null_map && (*null_map)[i]) {
221221
RETURN_IF_ERROR(checkArrowStatus(date32_builder.AppendNull(), column, *array_builder));
222222
} else {
223-
RETURN_IF_ERROR(
224-
checkArrowStatus(date32_builder.Append(cast_set<int, int64_t, false>(daynr)),
225-
column, *array_builder));
223+
RETURN_IF_ERROR(checkArrowStatus(
224+
date32_builder.Append(daynr_to_epoch_days(col_data[i].daynr())), column,
225+
*array_builder));
226226
}
227227
}
228228
return Status::OK();
@@ -257,7 +257,7 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
257257
"Arrow Date64 value must contain whole days: row={}, milliseconds={}",
258258
value_i, milliseconds);
259259
}
260-
const int64_t daynr = milliseconds / MILLISECONDS_PER_DAY + date_threshold;
260+
const int64_t daynr = epoch_days_to_daynr(milliseconds / MILLISECONDS_PER_DAY);
261261
DateV2Value<DateV2ValueType> value;
262262
if (daynr <= 0 || daynr > DATE_MAX_DAYNR ||
263263
!value.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
@@ -280,8 +280,13 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
280280
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
281281
auto date_value = unaligned_load<int32_t>(raw_byte_ptr);
282282

283+
const int64_t daynr = epoch_days_to_daynr(date_value);
283284
DateV2Value<DateV2ValueType> v;
284-
v.get_date_from_daynr(date_value + date_threshold);
285+
if (daynr == 0 || !v.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
286+
return Status::InvalidArgument(
287+
"Arrow Date32 value is outside the Doris DATE range: row={}, days={}",
288+
value_i, date_value);
289+
}
285290
col_data.emplace_back(v);
286291
}
287292
} else {
@@ -401,7 +406,7 @@ Status DataTypeDateV2SerDe::write_column_to_orc(const std::string& timezone, con
401406
if (cur_batch->notNull[row_id] == 0) {
402407
continue;
403408
}
404-
cur_batch->data[row_id] = col_data[row_id].daynr() - date_threshold;
409+
cur_batch->data[row_id] = daynr_to_epoch_days(col_data[row_id].daynr());
405410
}
406411
cur_batch->numElements = end - start;
407412
return Status::OK();

‎be/src/core/value/vdatetime_value.h‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1770,6 +1770,36 @@ inline uint32_t calc_daynr(uint16_t year, uint8_t month, uint8_t day) {
17701770
return delsum + y / 4 - y / 100 + y / 400;
17711771
}
17721772

1773+
// Doris follows MySQL's calendar, in which year 0 is NOT a leap year (see `is_leap()`): 0000-02-29
1774+
// does not exist and `calc_daynr()` therefore numbers 0000-01-01 .. 0000-02-28 one day ahead of the
1775+
// proleptic Gregorian calendar. Arrow `date32`, Parquet/ORC `DATE` and the Iceberg spec all define
1776+
// their day ordinal in the proleptic Gregorian calendar, where year 0 IS a leap year. The two
1777+
// numberings coincide from 0000-03-01 (daynr 60) onwards, so the whole difference is the missing
1778+
// 0000-02-29. Every conversion between a Doris DATE and an external "days since 1970-01-01" value
1779+
// must go through the two helpers below instead of adding/subtracting the epoch daynr directly.
1780+
inline constexpr int64_t DAYNR_OF_UNIX_EPOCH = 719528; // calc_daynr(1970, 1, 1)
1781+
inline constexpr int64_t DAYNR_OF_0000_03_01 = 60; // first daynr shared by both calendars
1782+
inline constexpr int64_t EPOCH_DAYS_MIN = -719528; // 0000-01-01, the smallest Doris DATE
1783+
inline constexpr int64_t EPOCH_DAYS_0000_02_29 = -719469; // exists in proleptic Gregorian only
1784+
inline constexpr int64_t EPOCH_DAYS_MAX = 2932896; // 9999-12-31, the largest Doris DATE
1785+
1786+
// Doris daynr -> days since 1970-01-01 in the proleptic Gregorian calendar.
1787+
inline constexpr int32_t daynr_to_epoch_days(int64_t daynr) {
1788+
return static_cast<int32_t>(daynr - DAYNR_OF_UNIX_EPOCH -
1789+
(daynr < DAYNR_OF_0000_03_01 ? 1 : 0));
1790+
}
1791+
1792+
// Inverse of `daynr_to_epoch_days()`. Returns 0, which is never a valid daynr, when the value has
1793+
// no Doris DATE representation: before 0000-01-01, after 9999-12-31, or the proleptic-only
1794+
// 0000-02-29. Callers must reject 0 instead of feeding it to `get_date_from_daynr()`.
1795+
inline constexpr int64_t epoch_days_to_daynr(int64_t epoch_days) {
1796+
if (epoch_days < EPOCH_DAYS_MIN || epoch_days > EPOCH_DAYS_MAX ||
1797+
epoch_days == EPOCH_DAYS_0000_02_29) {
1798+
return 0;
1799+
}
1800+
return epoch_days + DAYNR_OF_UNIX_EPOCH + (epoch_days < EPOCH_DAYS_0000_02_29 ? 1 : 0);
1801+
}
1802+
17731803
// DAY / WEEK fast path. Real workloads hold dates that cluster in a narrow range, so the two
17741804
// dictionary lookups below (`daynr(y, m, d)` and `daynr -> date`) are L1-resident; measured about
17751805
// 3x faster than the generic `date_add_interval<DAY>` (no TimeInterval, no second-level

‎be/src/exec/sink/writer/iceberg/partition_transformers.h‎

Lines changed: 38 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ class PartitionColumnTransforms {
4444
const doris::iceberg::PartitionField& field, const DataTypePtr& source_type);
4545
};
4646

47+
// Iceberg time transforms are all defined relative to 1970-01-01 (spec: Partition Transforms).
48+
inline constexpr int ICEBERG_EPOCH_YEAR = 1970;
49+
4750
class PartitionColumnTransformUtils {
4851
public:
4952
static DateV2Value<DateV2ValueType>& epoch_date() {
@@ -89,13 +92,15 @@ class PartitionColumnTransformUtils {
8992
}
9093

9194
static std::string human_hour(int hour_ordinal) {
92-
int day_value = hour_ordinal / 24;
93-
int housr_value = hour_ordinal % 24;
95+
// Hour ordinals are negative before 1970-01-01, so the split must floor rather than
96+
// truncate towards zero: -1 is 1969-12-31-23, not 1970-01-01 minus one hour.
97+
int day_value = (hour_ordinal >= 0 ? hour_ordinal : hour_ordinal - 23) / 24;
98+
int hour_value = hour_ordinal - day_value * 24;
9499
auto ymd = std::chrono::year_month_day(std::chrono::sys_days(
95100
std::chrono::floor<std::chrono::days>(EPOCH + std::chrono::days(day_value))));
96101
return fmt::format("{:04d}-{:02d}-{:02d}-{:02d}", static_cast<int>(ymd.year()),
97102
static_cast<unsigned>(ymd.month()), static_cast<unsigned>(ymd.day()),
98-
housr_value);
103+
hour_value);
99104
}
100105

101106
private:
@@ -615,7 +620,7 @@ class DateBucketPartitionColumnTransform : public PartitionColumnTransform {
615620
DateV2Value<DateV2ValueType> value =
616621
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
617622

618-
int64_t days_from_unix_epoch = value.daynr() - 719528;
623+
int64_t days_from_unix_epoch = daynr_to_epoch_days(value.daynr());
619624
uint32_t hash_value = HashUtil::murmur_hash3_32(&days_from_unix_epoch,
620625
sizeof(days_from_unix_epoch), 0);
621626

@@ -820,9 +825,10 @@ class DateYearPartitionColumnTransform : public PartitionColumnTransform {
820825
while (p_in < end_in) {
821826
DateV2Value<DateV2ValueType> value =
822827
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
823-
// datetime_diff<YEAR> actually returns int
824-
*p_out = cast_set<int, int64_t, false>(
825-
datetime_diff<YEAR>(PartitionColumnTransformUtils::epoch_date(), value));
828+
// Iceberg's `year` transform counts whole calendar years from 1970 and floors:
829+
// 1969-06-15 is -1, not the 0 that `datetime_diff<YEAR>` returns by rounding towards
830+
// zero. Taking the calendar year directly is exactly that floor.
831+
*p_out = value.year() - ICEBERG_EPOCH_YEAR;
826832
++p_in;
827833
++p_out;
828834
}
@@ -889,9 +895,10 @@ class TimestampYearPartitionColumnTransform : public PartitionColumnTransform {
889895
while (p_in < end_in) {
890896
DateV2Value<DateTimeV2ValueType> value =
891897
binary_cast<uint64_t, DateV2Value<DateTimeV2ValueType>>(*(UInt64*)p_in);
892-
// datetime_diff<YEAR> actually returns int
893-
*p_out = cast_set<int, int64_t, false>(
894-
datetime_diff<YEAR>(PartitionColumnTransformUtils::epoch_datetime(), value));
898+
// Iceberg's `year` transform counts whole calendar years from 1970 and floors:
899+
// 1969-06-15 is -1, not the 0 that `datetime_diff<YEAR>` returns by rounding towards
900+
// zero. Taking the calendar year directly is exactly that floor.
901+
*p_out = value.year() - ICEBERG_EPOCH_YEAR;
895902
++p_in;
896903
++p_out;
897904
}
@@ -958,9 +965,10 @@ class DateMonthPartitionColumnTransform : public PartitionColumnTransform {
958965
while (p_in < end_in) {
959966
DateV2Value<DateV2ValueType> value =
960967
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
961-
// datetime_diff<MONTH> actually returns int
962-
*p_out = cast_set<int, int64_t, false>(
963-
datetime_diff<MONTH>(PartitionColumnTransformUtils::epoch_date(), value));
968+
// Iceberg's `month` transform counts whole calendar months from 1970-01 and floors,
969+
// so 1969-06-15 is -7 rather than the -6 that `datetime_diff<MONTH>` produces by
970+
// rounding towards zero. The day of month never participates.
971+
*p_out = (value.year() - ICEBERG_EPOCH_YEAR) * 12 + (value.month() - 1);
964972
++p_in;
965973
++p_out;
966974
}
@@ -1027,9 +1035,10 @@ class TimestampMonthPartitionColumnTransform : public PartitionColumnTransform {
10271035
while (p_in < end_in) {
10281036
DateV2Value<DateTimeV2ValueType> value =
10291037
binary_cast<uint64_t, DateV2Value<DateTimeV2ValueType>>(*(UInt64*)p_in);
1030-
// datetime_diff<MONTH> actually returns int
1031-
*p_out = cast_set<int, int64_t, false>(
1032-
datetime_diff<MONTH>(PartitionColumnTransformUtils::epoch_datetime(), value));
1038+
// Iceberg's `month` transform counts whole calendar months from 1970-01 and floors,
1039+
// so 1969-06-15 is -7 rather than the -6 that `datetime_diff<MONTH>` produces by
1040+
// rounding towards zero. The day of month never participates.
1041+
*p_out = (value.year() - ICEBERG_EPOCH_YEAR) * 12 + (value.month() - 1);
10331042
++p_in;
10341043
++p_out;
10351044
}
@@ -1096,9 +1105,10 @@ class DateDayPartitionColumnTransform : public PartitionColumnTransform {
10961105
while (p_in < end_in) {
10971106
DateV2Value<DateV2ValueType> value =
10981107
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
1099-
// datetime_diff<DAY> actually returns int
1100-
*p_out = cast_set<int, int64_t, false>(
1101-
datetime_diff<DAY>(PartitionColumnTransformUtils::epoch_date(), value));
1108+
// Iceberg's `day` transform is "days from 1970-01-01" in the proleptic Gregorian
1109+
// calendar (spec: Partition Transforms). `datetime_diff<DAY>` cannot be used here: it
1110+
// counts in Doris's MySQL calendar, which is one day ahead for year-zero dates.
1111+
*p_out = daynr_to_epoch_days(value.daynr());
11021112
++p_in;
11031113
++p_out;
11041114
}
@@ -1170,9 +1180,10 @@ class TimestampDayPartitionColumnTransform : public PartitionColumnTransform {
11701180
while (p_in < end_in) {
11711181
DateV2Value<DateTimeV2ValueType> value =
11721182
binary_cast<uint64_t, DateV2Value<DateTimeV2ValueType>>(*(UInt64*)p_in);
1173-
// datetime_diff<DAY> actually returns int
1174-
*p_out = cast_set<int, int64_t, false>(
1175-
datetime_diff<DAY>(PartitionColumnTransformUtils::epoch_datetime(), value));
1183+
// Same as the DATE variant: the partition value is the proleptic-Gregorian ordinal
1184+
// of the calendar day. Note this is a floor, not a truncation towards the epoch, so
1185+
// `datetime_diff<DAY>` (which rounds towards zero by the time part) is not usable.
1186+
*p_out = daynr_to_epoch_days(value.daynr());
11761187
++p_in;
11771188
++p_out;
11781189
}
@@ -1243,9 +1254,12 @@ class TimestampHourPartitionColumnTransform : public PartitionColumnTransform {
12431254
while (p_in < end_in) {
12441255
DateV2Value<DateTimeV2ValueType> value =
12451256
binary_cast<uint64_t, DateV2Value<DateTimeV2ValueType>>(*(UInt64*)p_in);
1246-
// hour diff would't overflow int32
1257+
// Iceberg's `hour` transform floors to the hour boundary. Deriving it from the
1258+
// proleptic day ordinal plus the wall-clock hour is exact for every representable
1259+
// timestamp and, unlike `datetime_diff<HOUR>`, floors instead of truncating towards
1260+
// the epoch. The product cannot overflow int32: 2932896 * 24 + 23 == 70389527.
12471261
*p_out = cast_set<int, int64_t, false>(
1248-
datetime_diff<HOUR>(PartitionColumnTransformUtils::epoch_datetime(), value));
1262+
static_cast<int64_t>(daynr_to_epoch_days(value.daynr())) * 24 + value.hour());
12491263
++p_in;
12501264
++p_out;
12511265
}

0 commit comments

Comments
 (0)