PERF: improve performance with ZoneInfo timezones (get internal transitions/deltas to avoid slow python path) - #64379

Couple of test failures to figure out, but this looks like the right approach.

@jbrockmendel Thank you for your review! I've applied your feedback.
The tests failed because ZoneInfo and dateutil interpret offsets differently before the first transition (LMT era). For example, Africa/Lusaka shows a 2-second difference between the two libraries.
To address this, I modified the logic to use the original approach before the first transition, and the optimized approach after. (I've tried several approaches, but it seems impossible to fully resolve the performance issue for historical dates (before ~1900) where discrepancies occur between ZoneInfo and dateutil.)

ZoneInfo and dateutil interpret offsets differently before the first transition (LMT era)

This surprises me. Is there a minimal example?

ZoneInfo and dateutil interpret offsets differently before the first transition (LMT era)

This surprises me. Is there a minimal example?

@jbrockmendel
Here's a minimal example from my environment:

from zoneinfo import ZoneInfo
from dateutil.tz import gettz
from datetime import datetime

dt = datetime(1900, 1, 1)

# Africa/Lusaka: 2-second difference
print(ZoneInfo("Africa/Lusaka").utcoffset(dt))  # -> 2:10:18
print(gettz("Africa/Lusaka").utcoffset(dt))     # -> 2:10:20

# Africa/Sao_Tome: much larger difference
print(ZoneInfo("Africa/Sao_Tome").utcoffset(dt))  # -> -1 day, 23:23:15
print(gettz("Africa/Sao_Tome").utcoffset(dt))     # -> 0:26:56

Note that this discrepancy may not occur in all environments. Looking at the CI failure results, macOS passes while Ubuntu and Windows fail.
The tests pass when using the original logic (ZoneInfo.utcoffset()) for timestamps before the first transition.

@pganssle thoughts here on how to explain/handle the discrepancy?

I suspect this would also close #58962

@kjmin622 see also the discussion on the issue #64363 for a possible alternative approach

@pganssle thoughts here on how to explain/handle the discrepancy?

I haven't looked into the details, but I suspect that it might be down to the fact that dateutil still only uses the V1 portion of the TZif data, so it doesn't properly handle transitions that occur outside of ±2**32 seconds of the unix epoch. If there's a discrepancy between the two outside of that range, zoneinfo is almost certainly correct.

That will be fixed in dateutil 3.0.0 when this PR lands. That PR has been in flight for several years, so the conservative bet is that that will ~never happen but in the past month or so I managed to get everything checked off. I think a few more tests started failing and I need some reviews, but I'm optimistic about landing it before May this year.

FWIW, my preferred mechanism to handle all of this stuff (not that I would recommend blocking PRs like this one on it) would be for someone to write a new "vectorized time zone" library for pandas to use instead of zoneinfo, pytz or dateutil. Under the hood pandas is already doing a lot of this, but just with a bunch of weird stuff on top of it, it would be nice to move it into its own library for use across the ecosystem.

I'd be happy to champion whatever changes are necessary in zoneinfo to expose an API for using the same time zone data search path that CPython uses so that it's at least standardized across all libraries where to look for the data.

@kjmin622 small ping here (I would like to get this in for 3.0.3). Do you have time to take a look at my last comments?

@jorisvandenbossche I've had a lot on my schedule lately and couldn’t check. Would it be okay to apply it by May 3?

@jorisvandenbossche I applied your review! Thank you.

@kjmin622 thanks for the update! I just pushed one small additional simplication (the two small helper functions were essentially only used once now, so I inlined them)