Skip to content

Extract a fixed offset from Tz once and apply it to array's timestamps #10829

Description

@mzabaluev

Problem statement

Computations dealing with arrays of date/time values in a time zone go by obtaining the Tz value and then calling its methods per element.
This is inefficient for two reasons:

  1. A per-element dispatch on the TzInner enum, which may or may not be optimized out of the loop;
  2. For named time zones, a lookup into the chrono-tz generated dispatch and method call(s), which are again subject to optimization. Some of the time zones, most importantly UTC, amount to applying a fixed offset invariantly to all values.

Candidates for optimization

Extracted and summarized with Claude:

1. date_part extraction (year/month/day/hour/etc.)

arrow-arith/src/temporal.rs:418-488

Four near-identical impls for TimestampSecondType / MillisecondType / MicrosecondType / NanosecondType. tz is resolved once via get_tz(...), then each element does:

timestamp_s_to_datetime(d)
    .map(|c| Utc.from_utc_datetime(&c).with_timezone(&tz))
    .map(map_func)

with_timezone invokes Tz::offset_from_utc_datetime per row.

2. Timestamp ± interval arithmetic

arrow-arith/src/numeric.rs:437 resolves l_tz once, then per-element ops (e.g. Op::Add at numeric.rs:456-465) call T::add_year_month(l, r, l_tz), which forwards to add_year_months in arrow-array/src/types.rs:412-419 → as_datetime_with_timezone (candidate 6) → .with_timezone(&tz).

3. Timestamp display / pretty-print

arrow-cast/src/display.rs:755-763, write_timestamp, invoked once per row by the timestamp_display! macro:

let date = Utc.from_utc_datetime(&naive).with_timezone(&tz);

4. Timestamp → Time32/Time64 cast

arrow-cast/src/cast/mod.rs:615-631 (as_time_res_with_timezone), called per element from many (Timestamp(_, tz), Time32/Time64(_)) match arms, e.g. cast/mod.rs:1974-1985. tz is parsed once per cast call, then as_datetime_with_timezone (candidate 6) is called inside try_unary for every element.

5. Timestamp → Date32 cast

arrow-cast/src/cast/mod.rs:633-657 (timestamp_to_date32): tz parsed once, then array.try_unary(|x| as_datetime_with_timezone::<T>(x, tz)...) per element.

6. Timestamp → Timestamp with different timezone

arrow-cast/src/cast/mod.rs:2629-2648 (adjust_timestamp_to_timezone):

let adjust = |o| {
    let local = as_datetime::<T>(o)?;
    let offset = to_tz.offset_from_local_datetime(&local).single()?;
    T::from_naive_datetime(local - offset.fix(), None)
};

to_tz is loop-invariant across the whole unary_opt/try_unary call. This is the cleanest optimization target: if to_tz is a fixed offset, offset is also loop-invariant and can be computed once outside the closure.

Proposed solution

Add a public accessor on Tz, e.g. pub fn fixed_offset(&self) -> Option<FixedOffset>, in arrow-array/src/timezone.rs:

  • Returns Some immediately for TzInner::Offset.
  • Can also return Some on recognized named time zone variants, primarily UTC and its aliases, Etc/GMT* etc..

Each candidate site would then check this once outside its per-element loop and, when Some(offset), apply the offset via plain arithmetic (NaiveDateTime + Duration / integer add) instead of dispatching through chrono::TimeZone per element.

Other aliternatives considered

Replace chrono with jiff as proposed in #9183. Jiff has the requisite method on its TimeZone type.
That is a much wider change and requires API breaks.

Additional context

In testing of a similar optimization done outsize of arrow-rs, elimination of the chrono-tz dispatch from offsetting a timestamp for a timestamp-to-string array cast resulted in 13-18% improvement.

Activity

  1. tustvold commented on Aug 26, 2026

    @tustvold
    Contributor

    This is necessary to handle daylight savings time correctly, the offset depends on the values

  2. mzabaluev commented on Aug 26, 2026

    @mzabaluev
    ContributorAuthor

    This is necessary to handle daylight savings time correctly, the offset depends on the values

    Yes, in the case when the tz does not amount to a fixed offset, the computation still has to go through a TimeZone method call per value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions