Skip to content

Commit 3489284

Browse files
morningmanclaude
morningman
and
claude
committed
[fix](be) Hold zoned Flight timestamp bounds with pytz's rounded offsets
### What problem does this PR solve? Issue Number: None Related PR: #69771 Problem Summary: For an Arrow timestamp with a time zone, the Flight validator bounded values so that the local calendar in the zone, computed by cctz with the exact offsets, stays in 0001..9999. PyArrow resolves the zone name with pytz when pytz is installed, and pytz rounds every UTC offset to whole minutes. With time_zone 'America/Chicago', whose local mean time is -05:50:36, the lower bound 0001-01-01 05:50:36Z is local 0001-01-01 00:00:00 for cctz and zoneinfo, but 0000-12-31 23:59:36 for pytz (-05:51), so TimestampScalar.as_py() still raised OverflowError on a published batch. The local bounds are now also computed with the offsets rounded the way pytz does ((offset + 30) // 60 * 60, floor division), and the stricter bound wins: 05:51:00Z in that example. Bounds whose offsets are whole minutes (UTC, fixed offsets, the offsets at year 9999) do not change; for a zone whose local mean time rounds to a larger negative offset, up to 30 seconds at the start of year 1 are now rejected for every client. ### Release note None ### Check List (For Author) - Test: Unit Test - BE UT ArrowFlightTimestampTest.ZonedLowerBoundHoldsWithPytzRoundedOffsets: America/Chicago's exact-offset start is rejected and 05:51:00Z is accepted; the test fails when the bound uses the exact offset. - Behavior changed: Yes. The values above are rejected. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 3b3f2a6 commit 3489284

2 files changed

Lines changed: 53 additions & 6 deletions

File tree

‎be/src/format/arrow/arrow_block_convertor.cpp‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,16 @@ namespace doris {
6565

6666
namespace {
6767

68+
// PyArrow resolves an Arrow timezone name with pytz when pytz is installed, and pytz rounds every
69+
// UTC offset to whole minutes with Python's floor division, (offset + 30) // 60 * 60. Example:
70+
// America/Chicago's local mean time, -05:50:36 (-21036 s), becomes -05:51 (-21060 s).
71+
int64_t pytz_rounded_utc_offset(int64_t offset_seconds) {
72+
const int64_t shifted = offset_seconds + 30;
73+
// C++ division truncates toward zero; -21006 / 60 is -350, the floor is -351.
74+
const int64_t minutes = shifted / 60 - (shifted % 60 < 0 ? 1 : 0);
75+
return minutes * 60;
76+
}
77+
6878
class FlightTimestampValidator {
6979
public:
7080
Status init(std::shared_ptr<arrow::Array> array, std::string path,
@@ -89,17 +99,29 @@ class FlightTimestampValidator {
8999
units_per_second = 1000000000;
90100
break;
91101
}
92-
int64_t min_seconds = -62135596800LL;
93-
int64_t end_seconds = 253402300800LL;
102+
// 0001-01-01 00:00:00 and 10000-01-01 00:00:00 as seconds since the epoch.
103+
constexpr int64_t civil_start_seconds = -62135596800LL;
104+
constexpr int64_t civil_end_seconds = 253402300800LL;
105+
int64_t min_seconds = civil_start_seconds;
106+
int64_t end_seconds = civil_end_seconds;
94107
if (!type.timezone().empty()) {
95108
// The ordinary Arrow writer has already checked the schema/timezone binding.
96109
const auto local_start = cctz::convert(cctz::civil_second(1, 1, 1), timezone);
97110
const auto local_end = cctz::convert(cctz::civil_second(10000, 1, 1), timezone);
98111
// Python first constructs UTC and then applies the Arrow timezone. Both calendar
99-
// representations must fit, including offsets at the first and last supported day.
100-
min_seconds =
101-
std::max<int64_t>(min_seconds, local_start.time_since_epoch().count());
102-
end_seconds = std::min<int64_t>(end_seconds, local_end.time_since_epoch().count());
112+
// representations must fit, including offsets at the first and last supported day,
113+
// with the exact offsets (zoneinfo) and with pytz's rounded ones. Example for
114+
// America/Chicago: 0001-01-01 05:50:36Z is local 0001-01-01 00:00:00 with the
115+
// exact -05:50:36, but 0000-12-31 23:59:36 with pytz's -05:51, so the lower bound
116+
// is 05:51:00Z.
117+
const int64_t start_offset = timezone.lookup(local_start).offset;
118+
const int64_t end_offset = timezone.lookup(local_end).offset;
119+
min_seconds = std::max<int64_t>(
120+
{min_seconds, local_start.time_since_epoch().count(),
121+
civil_start_seconds - pytz_rounded_utc_offset(start_offset)});
122+
end_seconds = std::min<int64_t>(
123+
{end_seconds, local_end.time_since_epoch().count(),
124+
civil_end_seconds - pytz_rounded_utc_offset(end_offset)});
103125
}
104126
// Nanosecond bounds do not fit int64_t, even though every encoded value does.
105127
_min = static_cast<__int128>(min_seconds) * units_per_second;

‎be/test/format/arrow/arrow_flight_timestamp_test.cpp‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,31 @@ TEST_F(ArrowFlightTimestampTest, ChecksBothUtcAndZonedCalendarBounds) {
179179
}
180180
}
181181

182+
// PyArrow converts with pytz when it is installed, and pytz rounds America/Chicago's local mean
183+
// time, -05:50:36, to -05:51. 0001-01-01 05:50:36Z, the local start with the exact offset, is
184+
// 0000-12-31 23:59:36 in pytz and fails to convert; the first accepted instant is 05:51:00Z.
185+
TEST_F(ArrowFlightTimestampTest, ZonedLowerBoundHoldsWithPytzRoundedOffsets) {
186+
const std::string zone = "America/Chicago";
187+
cctz::time_zone timezone;
188+
ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone(zone, timezone));
189+
// Local 0001-01-01 00:00:00 with the exact offset is 05:50:36Z.
190+
auto block = timestamp_block(6, {make_datetime(1, 1, 1, 0, 0, 0, 0)});
191+
std::shared_ptr<arrow::Schema> schema;
192+
ASSERT_TRUE(DorisArrowSchemaConvertor(block, zone).get_arrow_schema(&schema).ok());
193+
ArrowFlightArrowBlockConvertor converter(schema, timezone);
194+
ASSERT_TRUE(converter.init().ok());
195+
std::shared_ptr<arrow::RecordBatch> batch;
196+
const auto status = converter.convert_to_arrow(block, arrow::default_memory_pool(), &batch);
197+
EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status;
198+
EXPECT_EQ(nullptr, batch);
199+
200+
// Local 00:00:24 is 05:51:00Z, which is 0001-01-01 00:00:00 with pytz's offset.
201+
auto valid = timestamp_block(6, {make_datetime(1, 1, 1, 0, 0, 24, 0)});
202+
ASSERT_TRUE(converter.convert_to_arrow(valid, arrow::default_memory_pool(), &batch).ok());
203+
const auto& values = static_cast<const arrow::TimestampArray&>(*batch->column(0));
204+
EXPECT_EQ(-62135575740000000LL, values.Value(0));
205+
}
206+
182207
TEST_F(ArrowFlightTimestampTest, NaiveBoundsDoNotDependOnSessionTimezone) {
183208
cctz::time_zone timezone;
184209
ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone("+08:00", timezone));

0 commit comments

Comments
 (0)