Skip to content

Commit a5b8251

Browse files
morningmanclaude
andcommitted
[fix](parquet) accept year-zero DATETIME on the Parquet read path
Doris DATETIME starts at 0000-01-01 00:00:00, but the Parquet reader gated every timestamp at 0001-01-01, so a value Doris accepts, stores and exports could not be read back out of Doris's own file. Two independent defects sat behind that. MIN_DORIS_TIMESTAMP_MICROS was a whole year narrower than the type it materialises into: 0000-01-01 is 366 days below 0001-01-01, not 365, because year 0 IS a leap year in the proleptic Gregorian calendar that Parquet timestamps are defined in. The bound was also applied to the raw instant, before the timezone offset, so it lost representable values at both ends -- east of UTC a local 0001-01-01 00:00:00 is an instant below the civil minimum, and west of it a local 9999-12-31 23:59:59.999999 is one above the maximum. What stays in the shared helper is only a coarse guard (one day of slack, more than any real offset), keeping the unit-overflow and malformed-INT96 checks format-level; the exact civil range is now enforced per target type after the conversion. append_datetimev2_from_epoch_micros() then added calc_daynr(1970, 1, 1) straight to a proleptic Gregorian day ordinal. Doris follows MySQL's calendar, in which year 0 is not a leap year, so the two numberings disagree over 0000-01-01 .. 0000-02-28: 0000-01-01 computed daynr 0 and was rejected outright, and the rest of that window would have decoded one day early. Route it through epoch_days_to_daynr(), the helper the DATE reader already uses, which also rejects the proleptic-only 0000-02-29 rather than colliding it with 0000-02-28. TIMESTAMPTZ shares the same storage and the same helper, and both its error text and the type's own comment still advertised a year-one floor. The legacy scanner's zero-date compatibility for genuinely unrepresentable values is left alone: after this change year zero no longer reaches that path at all. A new SerDe test checks every representable day of year zero against cctz rather than against Doris's own day arithmetic, plus the two offset edges, INT96, the millis unit, plain, dictionary, decoded-value and raw-predicate materialization, and TIMESTAMPTZ. Reverting the three source files turns 12 of these red and leaves the two "still rejects out-of-range" cases green, so they cannot pass vacuously. End to end on a local cluster: the extended Iceberg suite now compares timestamp values, not just partition tuples, against Spark 4.0.0 in both directions (Doris-written read by both engines, Spark-written read by Doris) -- Iceberg `timestamp` is not adjusted to UTC, so it exercises the civil path. The OUTFILE round trip was verified over file:// plus local(), the reproduction from the report, across three session timezones including the two that straddle the range edges, plus the counts, the ORC control and INSERT ... SELECT. That run also showed the legacy scanner (enable_file_scanner_v2=false) returns NULL for any pre-epoch timestamp with a sub-second part, year zero or not, because it truncates a negative epoch value towards zero instead of flooring it. That is a separate defect in a path this change does not touch, so the new suite deliberately does not assert the two scanners agree. Fixes apache#67447 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
1 parent 186036c commit a5b8251

8 files changed

Lines changed: 830 additions & 30 deletions

File tree

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,6 @@ Status append_datetimev2_from_epoch_micros(ColumnDateTimeV2::Container& data,
104104
static constexpr int64_t MICROS_PER_MINUTE = MICROS_PER_SECOND * 60;
105105
static constexpr int64_t MICROS_PER_HOUR = MICROS_PER_MINUTE * 60;
106106
static constexpr int64_t MICROS_PER_DAY = MICROS_PER_HOUR * 24;
107-
static const int64_t EPOCH_DAYNR = calc_daynr(1970, 1, 1);
108107

109108
int64_t days_since_epoch = timestamp_micros / MICROS_PER_DAY;
110109
int64_t micros_of_day = timestamp_micros % MICROS_PER_DAY;
@@ -113,11 +112,16 @@ Status append_datetimev2_from_epoch_micros(ColumnDateTimeV2::Container& data,
113112
--days_since_epoch;
114113
}
115114

116-
const int64_t daynr = EPOCH_DAYNR + days_since_epoch;
117-
if (daynr <= 0) {
115+
// A local Parquet timestamp carries a civil value whose day ordinal is defined in the
116+
// proleptic Gregorian calendar, while Doris numbers days in MySQL's calendar, where year 0 is
117+
// not a leap year. Adding `calc_daynr(1970, 1, 1)` directly conflates the two and shifts
118+
// 0000-01-01 .. 0000-02-28 one day early. `epoch_days_to_daynr()` bridges them and returns 0
119+
// for everything Doris cannot represent, including the proleptic-only 0000-02-29.
120+
const int64_t daynr = epoch_days_to_daynr(days_since_epoch);
121+
if (daynr == 0) {
118122
return Status::DataQualityError(
119-
"Decoded DATETIMEV2 timestamp is out of range: micros={}, daynr={}",
120-
timestamp_micros, daynr);
123+
"Decoded DATETIMEV2 timestamp is out of range: micros={}, epoch_days={}",
124+
timestamp_micros, days_since_epoch);
121125
}
122126

123127
DateV2Value<DateTimeV2ValueType> datetime_value;
@@ -154,6 +158,11 @@ Status append_datetimev2_from_utc_epoch_micros(ColumnDateTimeV2::Container& data
154158
DateV2Value<DateTimeV2ValueType> datetime_value;
155159
datetime_value.from_unixtime(epoch_seconds, timezone);
156160
datetime_value.set_microsecond(static_cast<uint32_t>(micros_of_second));
161+
// The civil range has to be checked here, after the offset is applied, not on the raw instant:
162+
// a local 0001-01-01 00:00:00 east of UTC is an instant below the civil minimum, and a local
163+
// 9999-12-31 23:59:59 west of it is an instant above the maximum. cctz resolves year 0 (it
164+
// uses the proleptic Gregorian calendar), and a year outside 0..9999 fails the check below
165+
// because `from_unixtime()` narrows it into a `uint16_t`.
157166
if (!datetime_value.is_valid_date()) {
158167
return Status::DataQualityError(
159168
"Decoded DATETIMEV2 timestamp is outside the target timezone range: micros={}",

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,13 @@ Status append_timestamptz_from_utc_epoch_micros(ColumnTimeStampTz::Container& da
8989
TimestampTzValue timestamp_tz;
9090
timestamp_tz.from_unixtime(epoch_seconds, UTC);
9191
timestamp_tz.set_microsecond(static_cast<uint32_t>(micros_of_second));
92+
// `from_unixtime()` splits the instant with cctz (proleptic Gregorian, so year 0 exists) and
93+
// narrows the civil year into a `uint16_t`. This is the exact range check for the target
94+
// type: a year below 0 wraps above 9999 and a year of 10000 exceeds it, so both are rejected
95+
// here, as is the proleptic-only 0000-02-29, which Doris's calendar does not have.
9296
if (!timestamp_tz.is_valid_date()) {
9397
return Status::DataQualityError(
94-
"Decoded TIMESTAMPTZ is outside the Doris 0001-9999 range: micros={}",
98+
"Decoded TIMESTAMPTZ is outside the Doris 0000-9999 range: micros={}",
9599
timestamp_micros);
96100
}
97101
data.push_back(timestamp_tz);

‎be/src/core/data_type_serde/parquet_timestamp.h‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,14 +33,35 @@ struct ParquetInt96Timestamp {
3333
#pragma pack()
3434
static_assert(sizeof(ParquetInt96Timestamp) == 12);
3535

36-
inline constexpr int64_t MIN_DORIS_TIMESTAMP_MICROS = -62135596800000000LL;
37-
inline constexpr int64_t MAX_DORIS_TIMESTAMP_MICROS = 253402300799999999LL;
36+
// Doris DATETIMEV2 and TIMESTAMPTZ both start at 0000-01-01 00:00:00 (`MIN_DATE_V2`) and end at
37+
// 9999-12-31 23:59:59.999999. Parquet timestamps count from 1970-01-01 in the proleptic Gregorian
38+
// calendar, where year 0 IS a leap year, so 0000-01-01 is 366 days below 0001-01-01 rather than
39+
// 365. Pinning the lower bound at 0001-01-01 made the reader stricter than the type it
40+
// materializes into: every year-zero value Doris itself writes came back as a conversion failure.
41+
inline constexpr int64_t MIN_DORIS_TIMESTAMP_MICROS = -62167219200000000LL; // 0000-01-01 00:00:00
42+
inline constexpr int64_t MAX_DORIS_TIMESTAMP_MICROS =
43+
253402300799999999LL; // 9999-12-31 23:59:59.999999
44+
45+
// A UTC-adjusted Parquet timestamp is an instant, not a civil value: which Doris DATETIME it
46+
// becomes depends on the reader's timezone, so the raw instant must not be measured against the
47+
// civil range. A local 0001-01-01 00:00:00 in Asia/Shanghai is an instant *below* the civil
48+
// minimum, and a local 9999-12-31 23:59:59 in America/New_York is one *above* the maximum; both
49+
// are representable and must survive. What stays here is only a coarse guard keeping the value in
50+
// the domain where the cctz conversion and the narrowing to a `uint16_t` year behave. No timezone
51+
// offset has ever exceeded 16 hours, so one day of slack admits every instant that can still land
52+
// inside the civil range. The exact range is enforced per target type after the conversion:
53+
// `epoch_days_to_daynr()` for a civil timestamp, `is_valid_date()` for an instant.
54+
inline constexpr int64_t DORIS_TIMESTAMP_OFFSET_SLACK_MICROS = 86400000000LL;
55+
inline constexpr int64_t MIN_DORIS_INSTANT_MICROS =
56+
MIN_DORIS_TIMESTAMP_MICROS - DORIS_TIMESTAMP_OFFSET_SLACK_MICROS;
57+
inline constexpr int64_t MAX_DORIS_INSTANT_MICROS =
58+
MAX_DORIS_TIMESTAMP_MICROS + DORIS_TIMESTAMP_OFFSET_SLACK_MICROS;
3859

3960
inline Status validate_parquet_timestamp_micros(int64_t timestamp_micros) {
40-
if (timestamp_micros < MIN_DORIS_TIMESTAMP_MICROS ||
41-
timestamp_micros > MAX_DORIS_TIMESTAMP_MICROS) {
61+
if (timestamp_micros < MIN_DORIS_INSTANT_MICROS ||
62+
timestamp_micros > MAX_DORIS_INSTANT_MICROS) {
4263
return Status::DataQualityError(
43-
"Parquet timestamp is outside the Doris 0001-9999 range: micros={}",
64+
"Parquet timestamp is outside the Doris 0000-9999 range: micros={}",
4465
timestamp_micros);
4566
}
4667
return Status::OK();
@@ -83,10 +104,10 @@ inline Status parquet_int96_timestamp_micros(const ParquetInt96Timestamp& value,
83104
const __int128 days = static_cast<int64_t>(value.julian_day) - JULIAN_EPOCH_OFFSET_DAYS;
84105
const __int128 timestamp_micros =
85106
days * MICROS_IN_DAY + value.nanos_of_day / NANOS_PER_MICROSECOND;
86-
if (timestamp_micros < MIN_DORIS_TIMESTAMP_MICROS ||
87-
timestamp_micros > MAX_DORIS_TIMESTAMP_MICROS) {
107+
if (timestamp_micros < MIN_DORIS_INSTANT_MICROS ||
108+
timestamp_micros > MAX_DORIS_INSTANT_MICROS) {
88109
return Status::DataQualityError(
89-
"Parquet INT96 timestamp is outside the Doris 0001-9999 range");
110+
"Parquet INT96 timestamp is outside the Doris 0000-9999 range");
90111
}
91112
*result = static_cast<int64_t>(timestamp_micros);
92113
return Status::OK();

‎be/src/core/value/timestamptz_value.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ struct CastParameters;
3535

3636
// TIMESTAMPTZ can be understood as a DATETIME type with timezone conversion functionality.
3737
// Doris automatically handles timezone conversions internally.
38-
// The storage format of TIMESTAMPTZ is the same as DATETIMEV2, both are 8-byte integers
39-
// representing microseconds from 0001-01-01 00:00:00.000000 to 9999-12-31 23:59:59.999999.
38+
// The storage format of TIMESTAMPTZ is the same as DATETIMEV2: both are 8-byte packed civil
39+
// values spanning 0000-01-01 00:00:00.000000 (`MIN_DATETIME_V2`, also the default) to
40+
// 9999-12-31 23:59:59.999999. Year 0 is inside the domain, so readers must not floor at year 1.
4041
// TIMESTAMPTZ does not store timezone information; conversions are performed during read and write
4142
// operations according to the specified timezone.
4243
// This requires that both reading and writing operations need a timezone parameter.

0 commit comments

Comments
 (0)