Skip to content

Commit 6c1fd8f

Browse files
morningmanclaude
andcommitted
[fix](be) Keep a runtime calendar check after packing Arrow timestamps
### What problem does this PR solve? Issue Number: None Related PR: #69697 Problem Summary: The range check added before packing an Arrow timestamp does not make every instant inside the range a Doris date. cctz counts year 0 as a leap year and Doris does not (is_leap(0) in util/time_lut.h is false), so timestamp(us) -62162121600000000, 0000-02-29 00:00:00 UTC, is inside 0000-01-01Z..9999-12-31Z but packs into a value that is_valid_date() rejects. For TIMESTAMPTZ this PR had turned the runtime is_valid_date() check after packing into a DCHECK, so that input aborted an ASAN or debug BE and appended an invalid TIMESTAMPTZ in a release build, where master returns DataQualityError. The DATETIMEV2 Arrow path checks only the civil year, so the same date at the session time zone was stored as an invalid DATETIMEV2, as it was before this PR. Both paths now check is_valid_date() at run time after packing and return DataQualityError for such a value. The ORC path (orc_timestamp_to_datetime) has the same year-only check and is not changed here. ### Release note None ### Check List (For Author) - Test: Unit Test - BE UT RejectsArrowTimestampsOutsideTimestampTzRange and RejectsArrowTimestampsOutsideDateTimeV2Range read -62162121600000000 and expect DataQualityError; both fail when the new checks skip that value. The serde, Arrow round-trip, nested serde and Lance reader suites (168 tests) pass. - Behavior changed: Yes. An Arrow timestamp on 0000-02-29 is rejected instead of being stored as an invalid date. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 3f336d2 commit 6c1fd8f

4 files changed

Lines changed: 24 additions & 1 deletion

File tree

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -739,6 +739,13 @@ Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
739739
static_cast<uint16_t>(civil.year()), static_cast<uint8_t>(civil.month()),
740740
static_cast<uint8_t>(civil.day()), static_cast<uint8_t>(civil.hour()),
741741
static_cast<uint8_t>(civil.minute()), static_cast<uint16_t>(civil.second()), 0);
742+
// cctz counts year 0 as a leap year and Doris does not (is_leap(0) is false), so the
743+
// civil date 0000-02-29 passes the year check but is not a DATETIMEV2 value.
744+
if (!v.is_valid_date()) {
745+
return Status::DataQualityError(
746+
"Arrow timestamp {} of type {} is not a valid DATETIMEV2 value", date_value,
747+
type->ToString());
748+
}
742749
// Get the fractional part.
743750
// add 0 on the right to make it 6 digits. DateTimeV2Value microsecond is 6 digits,
744751
// the scale decides to keep the first few digits, so the valid digits should be kept at the front.

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,13 @@ Status append_timestamptz_from_utc_epoch_micros(ColumnTimeStampTz::Container& da
103103
TimestampTzValue timestamp_tz;
104104
timestamp_tz.from_unixtime(epoch_seconds, UTC);
105105
timestamp_tz.set_microsecond(static_cast<uint32_t>(micros_of_second));
106-
DCHECK(timestamp_tz.is_valid_date());
106+
// The range check does not make every instant inside it a Doris date: cctz counts year 0 as a
107+
// leap year and Doris does not (is_leap(0) is false), so 0000-02-29 UTC, timestamp(us)
108+
// -62162121600000000, passes the range and packs into an invalid value.
109+
if (!timestamp_tz.is_valid_date()) {
110+
return Status::DataQualityError("Decoded TIMESTAMPTZ is not a valid Doris date: micros={}",
111+
timestamp_micros);
112+
}
107113
data.push_back(timestamp_tz);
108114
return Status::OK();
109115
}

‎be/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -626,6 +626,11 @@ TEST_F(DataTypeDateTimeV2SerDeTest, RejectsArrowTimestampsOutsideDateTimeV2Range
626626
EXPECT_NE(std::string::npos, status.to_string().find("is outside the DATETIMEV2 range"))
627627
<< status.to_string();
628628
}
629+
// 0000-02-29 is inside the year range, but year 0 is not a leap year in Doris.
630+
const auto leap_day = read(naive, -62162121600000000, utc, &result);
631+
EXPECT_FALSE(leap_day.ok());
632+
EXPECT_NE(std::string::npos, leap_day.to_string().find("is not a valid DATETIMEV2 value"))
633+
<< leap_day.to_string();
629634

630635
// 9999-12-31 15:59:59.999999 UTC is 9999-12-31 23:59:59.999999 at +08:00, and one microsecond
631636
// later is 10000-01-01 there.

‎be/test/core/data_type_serde/data_type_serde_timestamptz_test.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -428,5 +428,10 @@ TEST_F(DataTypeTimeStampTzSerDeTest, RejectsArrowTimestampsOutsideTimestampTzRan
428428
EXPECT_NE(std::string::npos, status.to_string().find("outside the Doris 0000-9999 range"))
429429
<< status.to_string();
430430
}
431+
// 0000-02-29 UTC is inside the range, but year 0 is not a leap year in Doris.
432+
const auto leap_day = read(-62162121600000000, &result);
433+
EXPECT_FALSE(leap_day.ok());
434+
EXPECT_NE(std::string::npos, leap_day.to_string().find("not a valid Doris date"))
435+
<< leap_day.to_string();
431436
}
432437
} // namespace doris

0 commit comments

Comments
 (0)