Better timezone support for astropy.Time objects.

Attempts to support Python 3.12 in #14784 have revealed that Python is getting rid of the datetime.utcnow method; however, the attempts to fix it as part of #15298 indicate that Time objects need to be more timezone aware. That is if one passes a datatime object which is timezone aware to astropy.Time, one should expect that the .datetime value output by the astropy.Time object should also be timezone aware with a matching the original timezone.

The solution proposed for dropping datetime.utcnow in #15298 (comment) will change any non-UTC timezones to assuming the time is UTC. That is

from astropy.time import Time
import datetime

t = datetime.datetime.now(tz=datetime.timezone(datetime.timedelta(seconds=-14400)))
a_t = Time(t)

assert (a_t.datetime - t).total_seconds() == 0  # Passes meaning the time values are identical
assert a_t.datetime.tzinfo != t.tzinfo  # Passes because the are different tzinfo (tzinfo has a proper `==` defined).

While this does not change any of the stored values or computations among the different times, it does represent a loss of information and can create confusion when changing among times, and incompatibilities between times which "should be compatible". E.G.

from astropy.time import Time
import datetime

t = datetime.datetime.now()  # not timezone aweare
a_t = Time(t)


a_t.datetime - t  # Raises error

I think there are subtleties which need to be accounted for here.

Oops, hadn't seen that you already opened an issue when I opened mine at #15311. Let me just copy my text from there, which describes what my desired outcome would be. I'll also replace "bug" with "feature request" since I think in #15298 you have suggested a good way that solves the "bug" part of the problem.

What is the problem this feature will solve?

Python in trying to move away from datetime that have not explicitly set a timezone, raising warning for, e.g., datetime.utcnow() which caused problems for our tests for python 3.12 (see #14784 and the discussion in #15298). This may mean that output from time.to_datetime() by default produces output that cannot easily be calculated with.

Describe the desired outcome

To at least communicate that our scale="utc" times have timezone="utc", i.e., make TimeDatetime as informative as possible. For scale="utc", it could set timezone="utc" (if no explicit timezone is given by calling to_datetime(timezone), for other scales it still only makes sense to use timezone=None.

EDIT: in principle, we could also store any timezone the user has given on Time, but this would seem to add a lot of complexity for little benefit for anything astronomers actually need Time for.

Additional context

The actual change would be rather minimal. From #15298 (comment), it would require changes at

astropy/astropy/time/formats.py

Lines 1126 to 1130 in bde1b85

    
           if timezone is not None: 
        
               if self._scale != "utc": 
        
                   raise ScaleValueError( 
        
                       f"scale is {self._scale}, must be 'utc' when timezone is supplied." 
        
                   )

Essentially, one would change those lines to,

        if self._scale == "utc":
            if timezone is None:
                timezone = timezone.utc
        else:
            if timezone is not None:
                raise ScaleValueError(
                    f"scale is {self._scale}, must be 'utc' when timezone is supplied."
                )

@mhvk

I've have explored how one might attempt to solve this problem inspired by #15298 (comment) and have essentially concluded the same thing as @mhvk. However, I have some additional and (possibly) more concrete thoughts.

Current Behavior

First, lets considered Time's current (astropy 5.3.3 in Python 3.11.5) behavior with respect to timezone-aware (TZA) and timezone-unaware (TZU) datetimes. Note that for convenience I will be using the .now method to construct these, but the same applies to any TZA or TZU time and that I will use timezone.utc as a convenient timezone object which I don't have to construct (which is rather annoying to do in general, see below).

from datetime import datetime, timezone
from astropy.time import Time

t_tza = datetime.now(tz=timezone.utc)
t_tzu = datetime.now(tz=timezone.utc).replace(tzinfo=None)  # Should be semantically equivalent to datetime.utcnow()

astropy_tza = Time(t_tza)
astropy_tzu = Time(t_tzu)

# Check timezone awareness of the datetime object returned from the Time object
assert astropy_tza.datetime.tzinfo is None  # tza is lost by astropy
assert astropy_tzu.datetime.tzinfo is None

# This breaks astropy's interactions with tza datetimes
astropy_tza.datetime - t_tza # Raises TypeError due to working between tza and tzu times
astropy_tzu.datetime - t_tzu # Succeeds

This means that currently Time is always loosing timezone awareness. However, if we make the relatively simple change suggested to astropy.time.formats

        if self._scale == "utc":
            if timezone is None:
                timezone = timezone.utc
        else:
            if timezone is not None:
                raise ScaleValueError(
                    f"scale is {self._scale}, must be 'utc' when timezone is supplied."
                )

we instead get

# This breaks astropy's interactions with tzu datetimes
astropy_tza.datetime - t_tza # Succeeds
astropy_tzu.datetime - t_tzu # Raises TypeError due to working between tza and tzu times

which represents a reversal of the Time.datetime behavior. That is Time.datetime goes from always TZU to always TZA. So we need to be more careful than just doing this.

Expected Behavior

Thus I think we should consider what a user's expectations of the behavior would naively be:

Passing TZA datetime to Time should not loose TZA for Time.datetime.
Passing TZU datetime to Time should not introduce TZA to Time.datetime.
The default behavior of Time.now should not change; namely, it should remain TZU by default.
In general Time should be TZU unless a timezone is directly specified (matching the Python datetime behavior).

Which are completely reasonable expectations, which I think are achievable. However, this does not match what Time is currently doing.

Thoughts on a Solution

As @mhvk has alluded to, Time needs to start storing the timezone behavior, say in a Time.timezone property (I think it needs to be something like a property, see below), where:

By default Time.timezone = None.
If a TZA object (Time or datetime) is passed to Time then the new object's Time.timezone gets the timezone information from the input TZA object.
The appropriate Time construction interfaces (e.g. the __init__, now, etc) should be able to accept reasonable ways to add a timezone awareness (under the appropriate circumstances).

What this effectively amounts to is adding the TIme.timezone property, optional arguments in appropriate places, and passing the timezone out if it exists for the Time.datetime output. For the last one, the modification suggested by @mhvk should instead be something like:

        # Use the time's current timezone (which can be None itself) as the default timezone
        timezone = self.timezone if timezone is None else timezone

        if timezone is not None:
            if self._scale != "utc":
                raise ScaleValueError(
                    f"scale is {self._scale}, must be 'utc' when timezone is supplied."
                )

Note that due to my unfamiliarity with the internals of astropy.time, I am not sure if/how the formatting object might get the value of the appropriate time's .timezone value, so I used self.timezone to represent this.

Why `.timezone` should be a property

As @mhvk has stated, the .timezone is really only useful when scale=utc so a setter should be used to protect against this and raise a useful error. Note that @nstarman pointed out in #15298 (comment) it might be useful to include some "convenience" timezone setters like the ability to set the timezone to the user's current timezone as there is not an easy/simple way to construct a tzinfo using the datetime module. In datetime you either construct your own tzinfo subclass, use timezone.utc, run .astimezone() on your datetime to add the system's local timezone, or figure out your UTC time offset and pass that to timezone(offset); meaning that, only the UTC and local timezones are "simple" to use. Some digging in the Python standard library uncovered the zoneinfo module, which constructs a tzinfo object from any IANA timezone code.

However, zoneinfo does not always work as you would expect. For example when asking the time module about my local timezone, I get time.tzname = ('EST', 'EDT') (US Eastern Standard Time, US Eastern Daylight Savings Time) for me currently. If I run zoneinfo.ZoneInfo("EST") returns a totally valid tzinfo object but zoneinfo.ZoneInfo("EDT") raises an error because EDT is not an IANA timezone code. So users (if they follow old guides/stackoverflow answers) to get a timezone code may end up with an error.

This being said, I think its reasonable to have a timezone setter which:

Protects against setting a timezone when scale != utc.
Accepts any arbitrary tzinfo object.
Accepts any IANA timezone code (using zoneinfo for construction). Note that "UTC" does return a "UTC" tzinfo object.
Accepts "local" as an input to select the user's system local time. Note that "local" is not a valid "IANA" timezone code so its reasonable value to accept to represent this.

Something like:

@property
def timezone(self):
    return self._timezone

@timezone.setter
def timezone(self, timezone):
    if self._scale != "utc":
        raise ScaleValueError(f"scale is {self._scale}, which must be 'utc' when timezone is supplied.")

    if timezone == "local":
        timezone = datetime.datetime.now().astimezone().tzinfo

    elif isinstance(timezone, str):
        timezone = zoneinfo.ZoneInfo(timezone)


    if not isinstance(timezone, datetime.tzinfo):
        raise ValueError(f"Got timezone of type {type(timezone)}, it must be a `tzinfo` instance, valid IANA timezone code, or local")

    self._timezone = timezone

Conculsions

I am willing to assist with doing this work or something similar if the maintainers agree with this path. However, I do not know how much time I have personally to dedicate to this especially given the fact that I don't know some of the details of the internal implementation which will need to be adjusted so the information gets passed around correctly

@WilliamJamieson

@WilliamJamieson - thanks for the detailed write-up! One thing I think is missing is that, typically, one does not just get back the Time instance itself, but does some calculations with it, which may involve scale changes (including non-obvious ones; e.g., for t2=t1+dt, with t1 in UTC, t1 is transferred to TAI, then the delta is added, and then transferred back to UTC). Should the timezone then be kept? Furthermore, should it be used for the iso and isot formats? I fear this would become a major hassle, for very little benefits: really, astronomers always work in UTC, and Time was never meant to have to worry about local times.

Hence, my personal preference would be to keep the behaviour that Time has no timezone awareness: it is always in UTC (or something convertible to it). But I think I am in favour of making the change that clarifies that, by adding timezone.utc explicitly (for scale="utc") since that seems to be how python datetime now likes it. If users do want to use a different timezone, they already can, by passing in datetime with a timezone set, and asking for a given timezone in time.to_datetime().

pinging @taldcroft too since his (correct) aversion for API changes should weigh heavily. But noting that this is slightly forced upon us by python itself changing datetime.

@WilliamJamieson

@WilliamJamieson - thanks for the detailed write-up! One thing I think is missing is that, typically, one does not just get back the Time instance itself, but does some calculations with it, which may involve scale changes (including non-obvious ones; e.g., for t2=t1+dt, with t1 in UTC, t1 is transferred to TAI, then the delta is added, and then transferred back to UTC). Should the timezone then be kept? Furthermore, should it be used for the iso and isot formats? I fear this would become a major hassle, for very little benefits: really, astronomers always work in UTC, and Time was never meant to have to worry about local times.

I agree, there are lots of gotchas/edge cases when it comes to timezones especially, which is why I think astropy should discuss it attempts to do anything. Working with Python's datetime gets even more convoluted when adding TZA times, and as you have pointed out astronomers typically don't care about timezones in a meaningful sense for the types of things they are working.

Hence, my personal preference would be to keep the behaviour that Time has no timezone awareness: it is always in UTC (or something convertible to it). But I think I am in favour of making the change that clarifies that, by adding timezone.utc explicitly (for scale="utc") since that seems to be how python datetime now likes it.

My read on why Python is removing the datetime.utcnow option (note that I haven't gone through the discussion at Python about the reasoning) isn't that it necessarily prefers TZA times, but rather utcnow in theory should be timezone aware since it is requesting the UTC time (which is a timezone in and of itself) and so that fact should be in coded. This essentially makes it very easy to mistake something as using datetime.utcnow() when it actually used datetime.now(). For example in Python 3.11

from datetime import datetime

t1 = datetime.utcnow()
t2 = datetime.now()

print(t1 - t2)
# Returns something around `3:59:59.999693` on my computer

which is a big difference. If programmers were not aware of which was being used can result in large and unexpected differences. It is my impression that it is more pythonic for things to be explicit about themselves; meaning that, this potential for ambiguity is not desirable. Thus by eventually removing datetime.utcnow, this forces programmers to explicitly encode the fact that the time was drawn with respect to UTC in the resulting object by making them specify the timezone as in datetime.now(tz=timezone.utc).

If users do want to use a different timezone, they already can, by passing in datetime with a timezone set, and asking for a given timezone in time.to_datetime().

Currently, time.to_datetime(timezone=<timezone object>) just assumes whatever the values of the time are, are there values in the timezone assigned. This is how datetime also treats TZU times, where calling datetime.replace(tzinfo=<timezone info object>) does not actually do anything to the values themselves relative to the timezone. E.G.

from datetime import datetime, timezone

t_tzu = datetime.now()  # A timezone unaware time
print(t_tzu)
# 2023-09-13 13:07:44.896722 when I ran this

t_utc = t_tzu.replace(tzinfo=timezone.utc)
print(t_utc)
# 2023-09-13 13:07:44.896722+00:00

t_loc = t_tzu.astimezone()  # set timezone to local system timezone
print(t_loc)
# 2023-09-13 13:07:44.896722-04:00 # on my local system which is currently reporting EDT

As you can see it assumes whatever the TZU time is as the local time in the timezone when it is assigned. The same behavior is evidenced by astropy currently:

from datetime import datetime, timezone
from astropy.time import Time

t_tzu = Time.now()
print(t_tzu)
# 2023-09-13 17:16:41.286123 when I ran this

t_utc = t_tzu.to_datetime(timezone=timezone.utc)
print(t_utc)
2023-09-13 17:16:41.286123+00:00

tzloc = datetime.now().astimezone().tzinfo # find local timezone
t_loc = t_tzu.to_datetime(timezone=tzloc)
print(t_loc)
# 2023-09-13 17:16:41.286123-04:00 # on my local system which is currently reporting EDT

Note that the change in Python 3.12 does not change any of this behavior, it simply deprecates datetime.utcnow() which creates the ambiguity I alluded to above.

pinging @taldcroft too since his (correct) aversion for API changes should weigh heavily.

My main concern is also has to do with an aversion to making API changes without due consideration.

But noting that this is slightly forced upon us by python itself changing datetime.

I would not consider this as Python fundamentally changing datetime, but rather it forcing programmers to be more explicit about what is going on. As such, there is a way to resolve most of the ambiguities in an explicit manner which does not change the API as it currently is but rather adds additional things.

That is Time() should have an additional optional argument convert_to_utc which defaults to False (to preserve the current behavior). When convert_to_utc = False and a TZA object is passed a warning to the effect of

astropy.time.Time is purposely not aware of timezone information. The timezone information will be lost. It is assumed that the times involved are all in the UTC timezone, if not it is up to the user to handle this information on their own. It is suggested that you set convert_to_utc = True

Then the datetime object is processed in the following way (which doesn't matter if the datetime object is TZU):

time = time.astimezone(timezone.utc) if convert_to_utc else time
time = time.replace(tzinfo=None)

before being further processed so that we explicitly remove this information. By default, the resulting Time object will be the same, but a warning will be issued during its creation. However, one can then be explicit about following the nominal assumption that Time is always in the UTC timezone (when timezones make sense).

Then the time.to_datetime method should have an additional optional argument like convert_from_utc which defaults to False (to preserve the current behavior), which if timezone is specified that the time represented in the object is UTC and then performs the conversion to the specified timezone. Replace

astropy/astropy/time/formats.py

Lines 1156 to 1158 in c095956

    
           dt = datetime.datetime( 
        
               iy, im, id, ihr, imin, isec, ifracsec, tzinfo=TimezoneInfo() 
        
           ).astimezone(timezone)

with something like

                dt = datetime.datetime(
                    iy, im, id, ihr, imin, isec, ifracsec, tzinfo=TimezoneInfo()
                )
                dt = dt.astimezone(datetime.timezone.utc) if convert_from_utc else dt
                dt = dt.astimezone(timezone)

So by default (convert_from_utc = False) one gets equivalent behavior to the current behavior to the current to_datetime method, but one can also explicitly choose to assume the time is UTC and that you are making a conversion from that to the timezone of your choosing.

Finally, the docs and docstring for Time should make it very explicit that there is an implicit assumption that the timezone (when the notion of timezone makes sense) for Time a time object is in the UTC timezone; otherwise, it is up to the end user to keep track the timezone information accurately on their own.

Ah, I had not realized that our to_datetime just set the timezone. That is actually rather misleading (but just goes to show I never use datetime). I think your suggestions above with a convert_from_utc or so argument make a lot of sense!

Navigation Menu

Repository navigation

Better timezone support for `astropy.Time` objects. #15306

What is the problem this feature will solve?

Describe the desired outcome

Additional context

Current Behavior

Expected Behavior

Thoughts on a Solution

Why `.timezone` should be a property

Conculsions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if timezone is not None:
	if self._scale != "utc":
	raise ScaleValueError(
	f"scale is {self._scale}, must be 'utc' when timezone is supplied."
	)

	dt = datetime.datetime(
	iy, im, id, ihr, imin, isec, ifracsec, tzinfo=TimezoneInfo()
	).astimezone(timezone)

Better timezone support for astropy.Time objects. #15306

Description

Activity

mhvk commented on Sep 13, 2023

What is the problem this feature will solve?

Describe the desired outcome

Additional context

WilliamJamieson commented on Sep 13, 2023

Current Behavior

Expected Behavior

Thoughts on a Solution

Why .timezone should be a property

Conculsions

mhvk commented on Sep 13, 2023

WilliamJamieson commented on Sep 13, 2023

mhvk commented on Sep 14, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Better timezone support for `astropy.Time` objects. #15306

Why `.timezone` should be a property