Which issue does this PR close?

• Closes string_to_datetime (and Utf8 → Timestamp(tz) casts) fail on DST-ambiguous and nonexistent local times #11039 and Cast from Timestamp(_, None) to a named timezone errors on DST boundaries datafusion#25084

Rationale for this change

What a user hits today

Parse a timestamp string into a named timezone. Pick a wall clock reading that the zone skips or repeats. The parse fails.

use arrow_array::timezone::Tz;
use arrow_cast::parse::string_to_datetime;

let tz: Tz = "America/New_York".parse().unwrap();

// 02:30 does not exist on 2024-03-10: the clocks go forward at 02:00.
string_to_datetime(&tz, "2024-03-10 02:30:00");
// 01:30 happens twice on 2024-11-03: the clocks go back at 02:00.
string_to_datetime(&tz, "2024-11-03 01:30:00");

Today:

gap       -> Err("Parser error: Error parsing timestamp from '2024-03-10 02:30:00': error computing timezone offset")
ambiguous -> Err("Parser error: Error parsing timestamp from '2024-11-03 01:30:00': error computing timezone offset")

After this PR:

gap       -> Ok("2024-03-10T07:30:00+00:00")
ambiguous -> Ok("2024-11-03T06:30:00+00:00")

Every Utf8 / LargeUtf8 / Utf8View cast to Timestamp(_, Some(tz)) goes through that function. Under CastOptions { safe: true } the error becomes a silent NULL for the whole row.

How this reaches a SQL user

A DataFusion user reaches the parser through ordinary SQL. #11038 fixed the cast kernel, so the first line below now works and the other two still fail:

SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York';  -- cast kernel: fixed by #11038
SELECT '2024-03-10 02:30:00'            AT TIME ZONE 'America/New_York';  -- parser: this PR
SET datafusion.execution.time_zone = 'America/New_York';
SELECT '2024-03-10 02:30:00'::timestamptz;                                -- parser: this PR

To a user those are the same query, and the last spelling is the most natural one. With only #11038 merged, an explicit ::timestamp in the middle of an expression makes a query start to work. That is harder to explain than the current uniform failure. So this PR completes #11038, it is not an optional follow-up.

Why the parser fails

string_to_datetime in arrow-cast/src/parse.rs resolves a wall clock reading with LocalResult::single(). That is None in two distinct cases:

• the reading is ambiguous — the repeated hour when the clocks go back;
• the reading is nonexistent — the skipped hour when the clocks go forward.

There are three such call sites, one per shape of input:

What changes are included in this PR?

Overview

One shared function now decides every wall clock reading in arrow-cast. The three parser sites and the cast kernel all call it. The policy itself does not change.

Detail

The resolution policy is deliberately shared with #11038, not restated. Two copies of this policy that drift apart is exactly the bug this PR avoids. One such divergence already exists in the tree, and this PR leaves it alone. See the follow-up section below.

So #11038's resolve_local_offset moves out of cast/temporal.rs into a new private arrow-cast/src/local_time.rs module and becomes pub(crate). It generalises from &Tz to any chrono::TimeZone, because string_to_datetime is generic over its target timezone. A thin resolve_local_datetime wrapper returns a DateTime<T>, which is the shape the three parser sites want. All four call sites now go through the one function.

The policy is unchanged from #11038:

• An ambiguous reading takes the later instant. This is the offset in effect after the transition.
• A nonexistent reading moves forward by the length of the gap. The code recovers the pre-transition offset with one probe 24 hours earlier, and takes the earliest result.

Two properties of the timezone database make the probe safe:

• No two transitions in any of the 597 zones are closer than 167 hours. So the probe lands on the correct side of the transition.
• No local gap is longer than 24 hours. So the probe lands outside the gap at all.

Both were verified exhaustively for #11038. This PR does not re-derive them.

On site 3, the zone named in the string

Site 3 resolves against a zone the user spelled out, rather than one taken from the target type. So it is fair to ask whether it deserves stricter treatment. It gets the same policy, for two reasons:

1. A named zone makes the zone unambiguous. It does nothing to make the wall clock reading unambiguous. '2024-03-10 02:30:00 America/New_York' is exactly as unresolvable as '2024-03-10 02:30:00' read in America/New_York, and the user has no other spelling for what they meant.
2. PostgreSQL 17 makes no distinction between the two spellings. SET TimeZone='America/New_York'; SELECT '2024-03-10 02:30:00'::timestamptz; and SELECT timestamptz '2024-03-10 02:30:00 America/New_York'; both return 2024-03-10 07:30:00+00. This was measured, not assumed.

A different rule for site 3 would put the "same query, different spelling, different outcome" split back inside a single function. That split is what this PR exists to remove.

Are these changes tested?

Yes. Every expected instant in the new tests came from a real PostgreSQL 17.11 (postgres:17). None were derived by hand. All three call sites are covered for both an ambiguous and a nonexistent reading:

• America/New_York and America/Los_Angeles — a whole-hour DST step.
• Australia/Lord_Howe — a thirty minute step, so a gap shifts 02:15 to 02:45, not 03:15. This rules out any implementation that assumes a one-hour gap.
• Pacific/Chatham — +12:45 / +13:45, an offset that is not a whole number of hours on either side.
• Australia/Sydney — southern hemisphere, so the gap falls in October and the repeated hour in April. This catches implementations that reach for offset_from_utc_datetime as a shortcut, which returns the post-transition offset and is wrong here.
• The date-only site, through the two zones whose transitions land on midnight and are therefore unreachable from any datetime input:
  • America/Sao_Paulo 2018-11-04 — local midnight does not exist, because Brazil started DST at 00:00. Result: 2018-11-04T03:00:00Z.
  • America/Havana 2024-11-03 — local midnight happens twice, because Cuba ends DST at 00:00. Result: the later instant, 2024-11-03T05:00:00Z.
• A fixed offset (+05:30), which has no transitions and must be unaffected.
• CastOptions { safe: true } against safe: false at the array level. The former produced a silent NULL and the latter an error. Both now produce the resolved instant, and genuinely unparseable input still nulls or errors as before.
• An array that straddles both of a zone's 2024 transitions, across StringArray, LargeStringArray and StringViewArray. This confirms the offset resolves per row, not once for the array.
• Sub-second precision survives the shift applied to a nonexistent reading.

Counterfactually validated. Revert the three call sites and keep the tests, and 9 of the new tests fail. The safe: true case fails with left: 0, which is the silent-NULL mode.

cargo test -p arrow-cast            # and --all-features, --doc, --doc --no-default-features
cargo clippy -p arrow-cast --all-targets --all-features -- -D warnings
cargo test -p arrow --features chrono-tz
cargo test -p arrow-csv -p arrow-json -p arrow-array
cargo fmt --all

arrow-cast already gained arrow-array/chrono-tz as a dev-dependency in #11038, so no manifest change is needed here.

Field research: PostgreSQL 17 and DuckDB

The resolution policy is the one arbitrary decision in this stack, so it needs more than one reference. Both engines were measured directly: PostgreSQL 17.11 in postgres:17, and the DuckDB 1.5.2 CLI with the ICU extension. All times below are UTC.

The two ambiguous rows carry the argument. A gap has only one sensible answer, so it discriminates nothing. An ambiguous reading has two real answers, and the choice between them is the policy.

2024-11-03 01:30 in New York is either 05:30Z (EDT) or 06:30Z (EST). Both engines give 06:30Z. Havana's ambiguous midnight is either 04:00Z (CDT) or 05:00Z (CST). Both engines give 05:00Z. In both cases the answer is the later instant, which is what this PR does.

The case neither engine can arbitrate

A fixed-offset string such as '+05:30' has no agreed reading, so it is recorded here rather than hidden. Measured:

PostgreSQL agrees with arrow-rs when the offset sits inside the literal. It disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright: Not implemented Error: Unknown TimeZone '+05:30'.

So neither engine settles it. This PR does not change that behaviour. It is tracked downstream in apache/datafusion#25170.

Are there any user-facing changes?

Yes, and the change is intentional. A string that previously failed to parse, or silently became NULL, now parses to the instant PostgreSQL and DuckDB produce. There is no API change. resolve_local_offset stays crate-private.

Together with #11038 this closes DataFusion #25084

Neither PR closes it alone. #11038 covers the cast kernel. This PR covers the parser. A user hits both through the same SQL, so both must land.

One existing test changes expectation

arrow/tests/timezone.rs asserted the old error for two America/Los_Angeles cases. They move from test_parse_timezone_invalid to test_parse_timezone:

Downstream

DataFusion has one matching expectation, at datafusion/sqllogictest/test_files/datetime/timestamps.slt:2360-2366. It asserts that SELECT TIMESTAMPTZ '2023-03-12 02:00:00 America/Los_Angeles' errors, and its own comment already notes # postgresql: accepts. On the next arrow upgrade it becomes a query P that returns 2023-03-12T10:00:00Z, which is what PostgreSQL returns. This PR includes no DataFusion change.

Follow-up: arrow-array resolves ambiguity the opposite way

This PR creates one shared helper for arrow-cast, but one divergence remains in the tree. It is deliberately untouched here:

Measured in America/New_York for 2024-11-03 01:30:00: from_naive_datetime gives 05:30Z, and this stack gives 06:30Z.

This matters downstream. DataFusion calls the arrow-array form directly with a Some(tz) in datafusion/functions/src/datetime/date_part.rs (date_to_scalar, four call sites). Where the local midnight it builds is ambiguous, the earlier instant becomes an upper bound, so valid rows can drop out. The behaviour predates this stack.

Changing from_naive_datetime affects every caller of arrow-array, so it deserves its own review rather than a ride along with a parser fix.

🤖 Generated with Claude Code