Skip to content

Commit 2166741

Browse files
adriangbclaudeJefffrey
authored
Resolve DST-ambiguous and nonexistent local times when casting timestamps to a named timezone (#11038)
# Which issue does this PR close? - Closes #11037. # Rationale for this change ## What a user hits today Cast a naive timestamp to a named timezone. Pick a wall clock reading that the zone skips or repeats. The cast fails. ```rust use arrow_array::TimestampSecondArray; use arrow_cast::{cast_with_options, CastOptions}; use arrow_schema::{DataType, TimeUnit}; // 1_710_037_800 is 2024-03-10T02:30:00, held as a naive wall clock reading. // America/New_York skips that hour: the clocks go forward at 02:00. let array = TimestampSecondArray::from(vec![1_710_037_800]); let to = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into())); let strict = CastOptions { safe: false, ..Default::default() }; let lenient = CastOptions { safe: true, ..Default::default() }; cast_with_options(&array, &to, &strict); cast_with_options(&array, &to, &lenient); ``` Today: ```text strict -> Err("Cast error: Cannot cast timezone to different timezone") lenient -> Ok, null_count = 1 ``` After this PR: ```text strict -> Ok, value = 1710055800 (2024-03-10T07:30:00Z, i.e. 03:30 EDT) lenient -> Ok, null_count = 0 ``` The `safe: true` result is the dangerous one. A whole column of DST-boundary readings turns to NULL, and nothing reports it. ## How this reaches a SQL user A DataFusion user reaches the same kernel through ordinary SQL. This is how the bug was found: ```sql SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York'; -- Cast error: Cannot cast timezone to different timezone ``` The engine plans that as `Timestamp(_, None)` to `Timestamp(_, Some("America/New_York"))`, so the error text above comes straight from this kernel. See apache/datafusion#25084. An upcoming type-coercion change inserts the same cast automatically for `timestamptz - timestamp`, so the failure gets more common, not less. ## Why the kernel fails `adjust_timestamp_to_timezone` reads each value as a wall clock time in the target zone. In an IANA timezone that reading is not always one instant: * A "fall back" transition repeats an hour, so the reading is **ambiguous**. * A "spring forward" transition skips an hour, so the reading is **nonexistent**. The kernel resolved the offset with `offset_from_local_datetime(..).single()`. That is `None` for both cases, so both cases fail. # What changes are included in this PR? A new private helper, `resolve_local_offset`, decides the two cases instead of rejecting them. It follows PostgreSQL and DuckDB: * An **ambiguous** reading takes the **later** instant. This is the offset in effect after the transition. `2024-11-03T01:30:00` in `America/New_York` becomes `01:30-05:00` (EST). * A **nonexistent** reading moves **forward** by the length of the gap. `2024-03-10T02:30:00` in `America/New_York` becomes `03:30-04:00` (EDT). `adjust_timestamp_to_timezone` calls the helper in place of `.single()`. That is the whole behavioural change. The forward shift needs the offset in effect *before* the transition. The code recovers it with one probe: it asks for the offset 24 hours earlier and takes the earliest result. `offset_from_utc_datetime` is not a valid shortcut, because it returns the post-transition offset. That is wrong for a southern hemisphere gap such as `Australia/Sydney` `2024-10-06T02:30:00`. Two properties of the timezone database make the probe safe: * No two transitions in any of the 597 zones are closer than **167 hours**. So the probe lands on the correct side of the transition. The smallest interval is `America/Boa_Vista` in 2000. * No local gap is longer than **24 hours**. So the probe lands outside the gap at all. Seven zones sit exactly at 24 hours — the dateline changes, such as `Pacific/Apia` in 2011. The next largest gap is 10 hours. If the probe still fails, the helper returns `None` and the caller keeps the old error or NULL. With current tzdata that branch is unreachable. Nothing else changes. Unit conversion, `safe` handling and the error message stay as they were. A fixed-offset timezone has no transitions, so it is unaffected. # Are these changes tested? Yes. Six new tests sit next to the existing `test_cast_timestamp_with_timezone_*` tests in `arrow-cast/src/cast/mod.rs`: * `test_cast_timestamp_to_named_timezone_dst` — `America/New_York`, with an unambiguous, an ambiguous and a nonexistent reading plus a null, under `safe: false`. * `test_cast_timestamp_to_named_timezone_dst_safe` — the same input under `safe: true`, which no longer produces nulls. * `test_cast_timestamp_to_named_timezone_dst_southern_hemisphere` — `Australia/Sydney`, where the transitions run the other way round. * `test_cast_timestamp_to_named_timezone_dst_nanosecond` and `..._dst_changing_unit` — the resolution composes with the unit paths. * `test_cast_timestamp_to_fixed_offset_timezone_unaffected` — `+08:00` on the same readings is unchanged. Expected values come from `chrono`, built from the wall clock reading and the expected offset. They are not hardcoded. Revert the one-line kernel change and the five DST tests fail, while the fixed-offset test still passes. ## Validation The change was checked against 97,162 cases. These cover every transition of every IANA zone in 1890–1995 and 2023–2025: 15,871 in a gap and 14,370 ambiguous. There were zero disagreements with an independent implementation on the same tzdata. Offsets resolve **per row**. One array that straddles both 2024 `America/New_York` transitions yields three distinct offsets, and every element matches PostgreSQL 17.11. One unrelated discrepancy came up. chrono-tz 0.10.4 bundles tzdata **2025b**, which puts the `Europe/Chisinau` and `Europe/Tiraspol` transitions an hour away from PostgreSQL, ICU and CPython. That reproduces on `main` without this PR. It needs a chrono-tz bump, not a change here. ## A note on the test manifest arrow-cast had no way to name an IANA zone in its own tests. `Tz` parses IANA names only when `arrow-array` is built with `chrono-tz`, and arrow-cast never enabled it. This PR enables it on the **dev-dependency only**: `arrow-array = { workspace = true, features = ["chrono-tz"] }` under `[dev-dependencies]`. The new tests then run in every arrow-cast CI job. No public feature appears, and nothing changes for downstream crates. One test needed an update. `test_cast_string_to_timestamp_invalid_tz` asserted the exact error tail `only offset based timezones supported without chrono-tz feature`, which is now `failed to parse timezone`. It asserts the stable prefix `Parser error: Invalid timezone \"ZZTOP\":` instead. Local runs that pass: `cargo test -p arrow-cast` (default and `--all-features`, debug and release), `cargo test -p arrow --features chrono-tz,prettyprint --test array_cast --test timezone`, clippy with `-D warnings` on the CI feature combinations, and `cargo doc --all-features` with `-D warnings`. # Field research: PostgreSQL 17 and DuckDB The resolution policy is the one arbitrary decision in this change, so it needs more than one reference. Both engines were measured directly: PostgreSQL 17.11 in `postgres:17`, and the DuckDB 1.5.2 CLI with the ICU extension. All times below are UTC. | Case | PostgreSQL 17.11 | DuckDB 1.5.2 | This PR | | --- | --- | --- | --- | | `America/New_York` `2024-11-03 01:30` — **ambiguous** | `2024-11-03 06:30:00` | `2024-11-03 06:30:00` | `2024-11-03 06:30:00` | | `America/Havana` `2024-11-03 00:00` — **ambiguous** local midnight | `2024-11-03 05:00:00` | `2024-11-03 05:00:00` | `2024-11-03 05:00:00` | | `America/New_York` `2024-03-10 02:30` — gap | `2024-03-10 07:30:00` | `2024-03-10 07:30:00` | `2024-03-10 07:30:00` | | `America/Sao_Paulo` `2018-11-04 00:00` — local midnight does not exist | `2018-11-04 03:00:00` | `2018-11-04 03:00:00` | `2018-11-04 03:00:00` | | `Australia/Sydney` `2024-10-06 02:30` — southern hemisphere gap | `2024-10-05 16:30:00` | `2024-10-05 16:30:00` | `2024-10-05 16:30:00` | | `Australia/Lord_Howe` `2024-10-06 02:15` — 30 minute DST step | `2024-10-05 15:45:00` | `2024-10-05 15:45:00` | `2024-10-05 15:45:00` | | `Pacific/Chatham` `2024-09-29 03:00` — +12:45 / +13:45 | `2024-09-28 14:15:00` | `2024-09-28 14:15:00` | `2024-09-28 14:15:00` | **The two ambiguous rows carry the argument.** A gap has only one sensible answer, so it discriminates nothing. An ambiguous reading has two real answers, and the choice between them is the policy. `2024-11-03 01:30` in New York is either `05:30Z` (EDT) or `06:30Z` (EST). Both engines give `06:30Z`. Havana's ambiguous midnight is either `04:00Z` (CDT) or `05:00Z` (CST). Both engines give `05:00Z`. In both cases the answer is **the later instant**, which is what this PR does. ## Spark takes the earlier instant Spark 4.2.0 was measured on the same cases. `CAST(TIMESTAMP_NTZ ... AS TIMESTAMP)`, `to_utc_timestamp`, `convert_timezone` and a string cast all agree, with ANSI mode on and off. Spark matches every gap row above. It takes the **earlier** instant for every ambiguous reading: New York `01:30` gives `05:30Z`, Havana midnight gives `04:00Z`, and `Australia/Sydney` `2024-04-07 02:30` gives `2024-04-06 15:30Z`. Spark follows Java here: the cast calls [`convertTz`](https://github.com/apache/spark/blob/b1b2685058d6359e00842cbd4c723f4ce3810709/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L901), which uses `LocalDateTime.atZone` (`ZonedDateTime.of`). This PR keeps the PostgreSQL and DuckDB rule. ## The case neither engine can arbitrate A fixed-offset string such as `'+05:30'` has no agreed reading, so it is recorded here rather than hidden. Measured: | Spelling | PostgreSQL 17.11 | Convention | | --- | --- | --- | | `SET TimeZone='+05:30'` then `'2024-01-01 12:00:00'::timestamptz` | `17:30Z` | POSIX, west-positive | | `'2024-01-01 12:00:00'::timestamp AT TIME ZONE '+05:30'` | `17:30Z` | POSIX, west-positive | | `timestamptz '2024-01-01 12:00:00 +05:30'` | `06:30Z` | ISO, east-positive | PostgreSQL agrees with arrow-rs when the offset sits inside the literal. It disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright: `Not implemented Error: Unknown TimeZone '+05:30'`. So neither engine settles it. This PR does not change that behaviour. It is tracked downstream in apache/datafusion#25170. # Are there any user-facing changes? Yes. A cast that previously raised an error, or returned NULL under `safe: true`, now returns the instants described above. There is no API change. ## This PR does not close DataFusion #25084 on its own There are two independent paths to a DST-boundary failure. This PR fixes one of them: ```sql -- cast kernel, fixed here SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York'; -- string parser, NOT fixed here (see #11039) SELECT '2024-03-10 02:30:00' AT TIME ZONE 'America/New_York'; SET datafusion.execution.time_zone = 'America/New_York'; SELECT '2024-03-10 02:30:00'::timestamptz; -- Parser error: Error parsing timestamp from '2024-03-10 02:30:00': error computing timezone offset ``` To a user those are the same query, and the last spelling is the most natural one. Merge this alone and an explicit `::timestamp` in the middle of an expression makes a query start to work. That is harder to explain than the current uniform failure. #11039 must land with this PR or close behind it, and must use the same policy so the two paths cannot drift. This PR **is** sufficient for apache/datafusion#10308, which is entirely on the cast path. ## Follow-up: `arrow-array` resolves ambiguity the opposite way After this PR, arrow resolves an ambiguous local time two different ways in two crates: | | Ambiguous | In a gap | | --- | --- | --- | | `arrow-cast::resolve_local_offset` (this PR) | the **later** instant | offset probed 24 hours earlier | | `arrow-array::types::from_naive_datetime` ([`types.rs:347`](https://github.com/apache/arrow-rs/blob/main/arrow-array/src/types.rs#L347)) | the **earlier** instant (`Ambiguous(dt1, _)`) | `None` | That function is deliberately untouched here. This matters downstream. DataFusion calls the `arrow-array` form directly with a `Some(tz)` in `datafusion/functions/src/datetime/date_part.rs` (`date_to_scalar`, four call sites). It builds local-midnight bounds for the `date_part(YEAR, col) = <year>` rewrite. Where that local midnight is ambiguous, the earlier instant becomes an **upper** bound, so valid rows can drop out. The behaviour predates this PR, but this PR widens the gap between the two policies. The two want reconciling, or the difference wants documenting on both functions. Changing `from_naive_datetime` affects every caller of `arrow-array`, so it deserves its own review rather than a ride along with a cast fix. ## Follow-up: a loud error becomes a quiet non-identity `to_local_time(ts) AT TIME ZONE tz` is a documented DataFusion idiom. Across an ambiguous hour it is no longer the identity, and it no longer says so. Before this PR the second leg raised `Cannot cast timezone to different timezone`: ```text orig local roundtrip identity 2021-10-31T02:00:00+02:00 2021-10-31T02:00:00 2021-10-31T02:00:00+01:00 false 2021-10-31T02:30:00+02:00 2021-10-31T02:30:00 2021-10-31T02:30:00+01:00 false 2021-10-31T02:00:00+01:00 2021-10-31T02:00:00 2021-10-31T02:00:00+01:00 true ``` PostgreSQL behaves the same way. `t = ((t AT TIME ZONE 'Europe/Brussels') AT TIME ZONE 'Europe/Brussels')` is `f` for those same two instants. So this is the intended trade, not an argument against the change. But it is a real behaviour change, and downstream docs want to say so. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Jeffrey Vo <[email protected]>
1 parent fdeada1 commit 2166741

2 files changed

Lines changed: 265 additions & 8 deletions

File tree

‎arrow-cast/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ base64 = "0.23"
5555
ryu = "1.0.16"
5656

5757
[dev-dependencies]
58+
# Enable `chrono-tz` so that tests can exercise IANA (named) timezones such as
59+
# `America/New_York`, which is where DST transitions can be observed.
60+
arrow-array = { workspace = true, features = ["chrono-tz"] }
5861
criterion = { workspace = true, default-features = false }
5962
half = { version = "2.1", default-features = false }
6063
insta = { workspace = true }

‎arrow-cast/src/cast/mod.rs‎

Lines changed: 262 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,9 @@ pub use crate::cast::union::*;
5757

5858
use arrow_buffer::IntervalMonthDayNano;
5959
use arrow_data::ByteView;
60-
use chrono::{NaiveTime, Offset, TimeZone, Utc};
60+
use chrono::{
61+
FixedOffset, LocalResult, NaiveDateTime, NaiveTime, Offset, TimeDelta, TimeZone, Utc,
62+
};
6163
use std::cmp::Ordering;
6264
use std::sync::Arc;
6365

@@ -1921,6 +1923,11 @@ pub fn cast_with_options(
19211923
// unchanged.
19221924
//
19231925
// i.e. Timestamp('2001-01-01T00:00', None) -> Timestamp('2001-01-01T00:00', '+0700')
1926+
//
1927+
// For IANA timezones a wall clock reading is not always a unique
1928+
// instant: a "fall back" DST transition makes an hour occur twice
1929+
// and a "spring forward" transition skips an hour entirely. See
1930+
// `resolve_local_offset` for how those readings are resolved.
19241931
(None, Some(to_tz)) => {
19251932
let to_tz: Tz = to_tz.parse()?;
19261933
match to_unit {
@@ -2600,15 +2607,74 @@ fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>(
26002607
)?))
26012608
}
26022609

2610+
/// Returns the offset to use when interpreting `local` as a wall clock reading
2611+
/// in `tz`, or `None` if it cannot be resolved.
2612+
///
2613+
/// `None` is not expected in practice. With the current timezone database no
2614+
/// reading reaches it, because every ambiguous or nonexistent reading resolves
2615+
/// as described below. The `None` path is a safeguard against a future
2616+
/// timezone database that breaks the assumptions of the gap handling. Callers
2617+
/// then apply their usual error or null handling.
2618+
///
2619+
/// In an IANA timezone a wall clock reading does not always identify a unique
2620+
/// instant, and this function picks one following the same rules as PostgreSQL
2621+
/// and DuckDB:
2622+
///
2623+
/// * **Ambiguous** -- when the clocks go back ("fall back") the same reading
2624+
/// occurs twice. The *later* instant is chosen, i.e. the offset in effect
2625+
/// after the transition. For example `2024-11-03T01:30:00` in
2626+
/// `America/New_York` is read as `-05:00` (EST), not `-04:00` (EDT).
2627+
/// * **Nonexistent** -- when the clocks go forward ("spring forward") the
2628+
/// reading never occurs. It is shifted forward by the length of the gap,
2629+
/// which is the same as reading it with the offset in effect *before* the
2630+
/// transition. For example `2024-03-10T02:30:00` in `America/New_York` is
2631+
/// read as `-05:00` (EST) and therefore denotes `2024-03-10T03:30:00-04:00`.
2632+
///
2633+
/// Timezones with a fixed offset are never ambiguous and have no gaps.
2634+
///
2635+
/// See <https://github.com/apache/arrow-rs/issues/11037> for the PostgreSQL and
2636+
/// ICU (DuckDB) sources these rules are taken from.
2637+
fn resolve_local_offset(tz: &Tz, local: &NaiveDateTime) -> Option<FixedOffset> {
2638+
match tz.offset_from_local_datetime(local) {
2639+
LocalResult::Single(offset) => Some(offset.fix()),
2640+
LocalResult::Ambiguous(_earlier, later) => Some(later.fix()),
2641+
LocalResult::None => {
2642+
// The reading falls in a gap. Recover the offset in effect before
2643+
// the transition by probing 24 hours earlier.
2644+
//
2645+
// Two separate properties of the timezone database make this sound:
2646+
//
2647+
// 1. No local gap is longer than 24 hours, so the probe lands
2648+
// outside this gap and is itself resolvable. Seven zones have a
2649+
// gap of exactly 24 hours -- the dateline changes, such as
2650+
// `Pacific/Apia` in 2011 and `Pacific/Kiritimati` in 1994. At the
2651+
// last second of one of those the probe lands one second before
2652+
// the gap starts, so the true margin here is one second, not a
2653+
// comfortable one.
2654+
// 2. No two transitions are closer together than 24 hours, so the
2655+
// offset the probe finds is the one in effect immediately before
2656+
// this transition, and not some older offset. The smallest
2657+
// observed interval is 167 hours (`America/Boa_Vista`, 2000).
2658+
//
2659+
// Property 1 is what makes the probe resolvable; property 2 is what
2660+
// makes the answer correct. If the probe is still unresolvable, give
2661+
// up and let the caller apply the usual error / null handling.
2662+
tz.offset_from_local_datetime(&(*local - TimeDelta::hours(24)))
2663+
.earliest()
2664+
.map(|offset| offset.fix())
2665+
}
2666+
}
2667+
}
2668+
26032669
fn adjust_timestamp_to_timezone<T: ArrowTimestampType>(
26042670
array: PrimitiveArray<Int64Type>,
26052671
to_tz: &Tz,
26062672
cast_options: &CastOptions,
26072673
) -> Result<PrimitiveArray<Int64Type>, ArrowError> {
26082674
let adjust = |o| {
26092675
let local = as_datetime::<T>(o)?;
2610-
let offset = to_tz.offset_from_local_datetime(&local).single()?;
2611-
T::from_naive_datetime(local - offset.fix(), None)
2676+
let offset = resolve_local_offset(to_tz, &local)?;
2677+
T::from_naive_datetime(local - offset, None)
26122678
};
26132679
let adjusted = if cast_options.safe {
26142680
array.unary_opt::<_, Int64Type>(adjust)
@@ -6956,6 +7022,191 @@ mod tests {
69567022
assert!(result.is_null(2));
69577023
}
69587024

7025+
/// The i64 value of a `Timestamp(Second, None)` holding the given wall clock
7026+
/// reading (a naive timestamp is stored as if it were UTC).
7027+
fn naive_seconds(y: i32, m: u32, d: u32, h: u32, min: u32) -> i64 {
7028+
NaiveDate::from_ymd_opt(y, m, d)
7029+
.unwrap()
7030+
.and_hms_opt(h, min, 0)
7031+
.unwrap()
7032+
.and_utc()
7033+
.timestamp()
7034+
}
7035+
7036+
/// The instant, in seconds since the epoch, denoted by the given wall clock
7037+
/// reading at a fixed offset of `offset_hours`.
7038+
fn instant_seconds(y: i32, m: u32, d: u32, h: u32, min: u32, offset_hours: i32) -> i64 {
7039+
let offset = FixedOffset::east_opt(offset_hours * 3600).unwrap();
7040+
NaiveDate::from_ymd_opt(y, m, d)
7041+
.unwrap()
7042+
.and_hms_opt(h, min, 0)
7043+
.unwrap()
7044+
.and_local_timezone(offset)
7045+
.unwrap()
7046+
.timestamp()
7047+
}
7048+
7049+
// Cast Timestamp(_, None) -> Timestamp(_, Some(IANA timezone)) across DST
7050+
// transitions. See `resolve_local_offset`.
7051+
#[test]
7052+
fn test_cast_timestamp_to_named_timezone_dst() {
7053+
// Unambiguous, EDT (-04:00) is in effect.
7054+
let unambiguous = naive_seconds(2024, 11, 1, 0, 0);
7055+
// Ambiguous: the clocks go back at 2024-11-03T02:00 EDT, so 01:30
7056+
// happens twice, first at -04:00 and then at -05:00.
7057+
let ambiguous = naive_seconds(2024, 11, 3, 1, 30);
7058+
// Nonexistent: the clocks go forward at 2024-03-10T02:00 EST, so 02:30
7059+
// never happens.
7060+
let nonexistent = naive_seconds(2024, 3, 10, 2, 30);
7061+
assert_eq!(
7062+
[unambiguous, ambiguous, nonexistent],
7063+
[1_730_419_200, 1_730_597_400, 1_710_037_800]
7064+
);
7065+
7066+
let array = TimestampSecondArray::from(vec![
7067+
Some(unambiguous),
7068+
Some(ambiguous),
7069+
Some(nonexistent),
7070+
None,
7071+
]);
7072+
let to_type = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into()));
7073+
let options = CastOptions {
7074+
safe: false,
7075+
..Default::default()
7076+
};
7077+
7078+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7079+
assert_eq!(b.data_type(), &to_type);
7080+
let c = b.as_primitive::<TimestampSecondType>();
7081+
7082+
assert_eq!(c.value(0), instant_seconds(2024, 11, 1, 0, 0, -4));
7083+
assert_eq!(c.value(0), 1_730_433_600);
7084+
// The later of the two candidates, i.e. EST rather than EDT.
7085+
assert_eq!(c.value(1), instant_seconds(2024, 11, 3, 1, 30, -5));
7086+
assert_eq!(c.value(1), 1_730_615_400);
7087+
// Shifted forward by the one hour gap: 02:30 EST is 03:30 EDT.
7088+
assert_eq!(c.value(2), instant_seconds(2024, 3, 10, 2, 30, -5));
7089+
assert_eq!(c.value(2), instant_seconds(2024, 3, 10, 3, 30, -4));
7090+
assert_eq!(c.value(2), 1_710_055_800);
7091+
assert!(c.is_null(3));
7092+
}
7093+
7094+
// The same values must not become null when `safe` casting is requested.
7095+
#[test]
7096+
fn test_cast_timestamp_to_named_timezone_dst_safe() {
7097+
let array = TimestampSecondArray::from(vec![
7098+
Some(naive_seconds(2024, 11, 1, 0, 0)),
7099+
Some(naive_seconds(2024, 11, 3, 1, 30)),
7100+
Some(naive_seconds(2024, 3, 10, 2, 30)),
7101+
None,
7102+
]);
7103+
let to_type = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into()));
7104+
let options = CastOptions {
7105+
safe: true,
7106+
..Default::default()
7107+
};
7108+
7109+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7110+
assert_eq!(b.data_type(), &to_type);
7111+
let c = b.as_primitive::<TimestampSecondType>();
7112+
assert_eq!(c.null_count(), 1);
7113+
assert_eq!(c.value(0), 1_730_433_600);
7114+
assert_eq!(c.value(1), 1_730_615_400);
7115+
assert_eq!(c.value(2), 1_710_055_800);
7116+
assert!(c.is_null(3));
7117+
}
7118+
7119+
// Southern hemisphere: the transitions run the other way around.
7120+
#[test]
7121+
fn test_cast_timestamp_to_named_timezone_dst_southern_hemisphere() {
7122+
// Ambiguous: clocks go back at 2024-04-07T03:00 AEDT (+11:00 -> +10:00).
7123+
let ambiguous = naive_seconds(2024, 4, 7, 2, 30);
7124+
// Nonexistent: clocks go forward at 2024-10-06T02:00 AEST (+10:00 -> +11:00).
7125+
let nonexistent = naive_seconds(2024, 10, 6, 2, 30);
7126+
7127+
let array = TimestampSecondArray::from(vec![Some(ambiguous), Some(nonexistent)]);
7128+
let to_type = DataType::Timestamp(TimeUnit::Second, Some("Australia/Sydney".into()));
7129+
let options = CastOptions {
7130+
safe: false,
7131+
..Default::default()
7132+
};
7133+
7134+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7135+
assert_eq!(b.data_type(), &to_type);
7136+
let c = b.as_primitive::<TimestampSecondType>();
7137+
7138+
// The later of the two candidates, i.e. AEST (+10:00).
7139+
assert_eq!(c.value(0), instant_seconds(2024, 4, 7, 2, 30, 10));
7140+
assert_eq!(c.value(0), 1_712_421_000);
7141+
// Shifted forward by the one hour gap: 02:30 AEST is 03:30 AEDT.
7142+
assert_eq!(c.value(1), instant_seconds(2024, 10, 6, 2, 30, 10));
7143+
assert_eq!(c.value(1), instant_seconds(2024, 10, 6, 3, 30, 11));
7144+
assert_eq!(c.value(1), 1_728_145_800);
7145+
}
7146+
7147+
// The resolution is independent of the time unit.
7148+
#[test]
7149+
fn test_cast_timestamp_to_named_timezone_dst_nanosecond() {
7150+
let ambiguous = naive_seconds(2024, 11, 3, 1, 30) * 1_000_000_000 + 123_456_789;
7151+
let nonexistent = naive_seconds(2024, 3, 10, 2, 30) * 1_000_000_000 + 123_456_789;
7152+
let array = TimestampNanosecondArray::from(vec![Some(ambiguous), Some(nonexistent)]);
7153+
let to_type = DataType::Timestamp(TimeUnit::Nanosecond, Some("America/New_York".into()));
7154+
let options = CastOptions {
7155+
safe: false,
7156+
..Default::default()
7157+
};
7158+
7159+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7160+
assert_eq!(b.data_type(), &to_type);
7161+
let c = b.as_primitive::<TimestampNanosecondType>();
7162+
assert_eq!(c.value(0), 1_730_615_400 * 1_000_000_000 + 123_456_789);
7163+
assert_eq!(c.value(1), 1_710_055_800 * 1_000_000_000 + 123_456_789);
7164+
}
7165+
7166+
// Unit conversion still composes with the timezone adjustment.
7167+
#[test]
7168+
fn test_cast_timestamp_to_named_timezone_dst_changing_unit() {
7169+
let array = TimestampSecondArray::from(vec![
7170+
Some(naive_seconds(2024, 11, 3, 1, 30)),
7171+
Some(naive_seconds(2024, 3, 10, 2, 30)),
7172+
]);
7173+
let to_type = DataType::Timestamp(TimeUnit::Millisecond, Some("America/New_York".into()));
7174+
let options = CastOptions {
7175+
safe: false,
7176+
..Default::default()
7177+
};
7178+
7179+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7180+
assert_eq!(b.data_type(), &to_type);
7181+
let c = b.as_primitive::<TimestampMillisecondType>();
7182+
assert_eq!(c.value(0), 1_730_615_400_000);
7183+
assert_eq!(c.value(1), 1_710_055_800_000);
7184+
}
7185+
7186+
// A fixed offset has no transitions, so the same readings are unaffected.
7187+
#[test]
7188+
fn test_cast_timestamp_to_fixed_offset_timezone_unaffected() {
7189+
let array = TimestampSecondArray::from(vec![
7190+
Some(naive_seconds(2024, 11, 1, 0, 0)),
7191+
Some(naive_seconds(2024, 11, 3, 1, 30)),
7192+
Some(naive_seconds(2024, 3, 10, 2, 30)),
7193+
None,
7194+
]);
7195+
let to_type = DataType::Timestamp(TimeUnit::Second, Some("+08:00".into()));
7196+
let options = CastOptions {
7197+
safe: false,
7198+
..Default::default()
7199+
};
7200+
7201+
let b = cast_with_options(&array, &to_type, &options).unwrap();
7202+
assert_eq!(b.data_type(), &to_type);
7203+
let c = b.as_primitive::<TimestampSecondType>();
7204+
assert_eq!(c.value(0), instant_seconds(2024, 11, 1, 0, 0, 8));
7205+
assert_eq!(c.value(1), instant_seconds(2024, 11, 3, 1, 30, 8));
7206+
assert_eq!(c.value(2), instant_seconds(2024, 3, 10, 2, 30, 8));
7207+
assert!(c.is_null(3));
7208+
}
7209+
69597210
#[test]
69607211
fn test_cast_date64_to_timestamp() {
69617212
let array = Date64Array::from(vec![Some(864000000005), Some(1545696000001), None]);
@@ -12598,11 +12849,14 @@ mod tests {
1259812849
};
1259912850

1260012851
for dt in data_types {
12601-
assert_eq!(
12602-
cast_with_options(&array, &dt, &cast_options)
12603-
.unwrap_err()
12604-
.to_string(),
12605-
"Parser error: Invalid timezone \"ZZTOP\": only offset based timezones supported without chrono-tz feature"
12852+
// The trailing detail of the message differs depending on whether
12853+
// `chrono-tz` is enabled, so only the common prefix is asserted.
12854+
let err = cast_with_options(&array, &dt, &cast_options)
12855+
.unwrap_err()
12856+
.to_string();
12857+
assert!(
12858+
err.starts_with("Parser error: Invalid timezone \"ZZTOP\":"),
12859+
"{err}"
1260612860
);
1260712861
}
1260812862
}

0 commit comments

Comments
 (0)