Please support TAI time expressed as unix seconds

Description

I would like astropy.time.Time to directly support expressing TAI date in unix seconds. This would allow manipulating the value returned by TAI_CLOCK on linux. It is also a standard used by the LSST for timestamps for telescope control.

At present astropy.time.Time seems to think that any time in unix seconds is always UTC. Thus the two asserts in the following code pass, but if this feature was implemented then they would not -- the two times would differ by 37 seconds (on 2020-03-19):

import time
import astropy.time
unix_time = time.time()
date1 = astropy.time.Time(unix_time, scale="tai", format="unix")
date2 = astropy.time.Time(unix_time, scale="utc", format="unix")
assert date1 == date2
now = astropy.time.Time.now()
assert now.tai.unix == now.utc.unix

It is currently quite painful to obtain TAI time in unix seconds from astropy.time,
and even more painful to go in the other direction.

Additional context

Here is the code I am using to obtain TAI unix seconds:

def tai_from_utc(utc_unix):
    """Return TAI in unix seconds, given UTC in unix seconds."""
    astropy_utc = astropy.time.Time(utc_unix, scale="utc", format="unix")
    try:
        dt_utc = astropy_utc.utc.to_datetime()
        dt_tai = astropy_utc.tai.to_datetime()
    except ValueError:
        # datetime cannot have 60 in the seconds field; back off a second
        astropy_utc = astropy_utc - 1 * u.second
        dt_utc = astropy_utc.utc.to_datetime()
        dt_tai = astropy_utc.tai.to_datetime()
    tai_minus_utc = (dt_tai - dt_utc).total_seconds()
    return utc_unix + tai_minus_utc

If this feature was implemented this code would become:

def tai_from_utc(utc_unix):
    """Return TAI in unix seconds, given UTC in unix seconds."""
    return astropy.time.Time(utc_unix, scale="utc", format="unix").tai.unix

I have not yet figured out tolerable code to go in the other direction though I have cobbled something together that works.

@r-owen

@r-owen - I may not be understanding completely right, but this does not seem a good idea. POSIX unix is strictly defined based on UTC (with strange duplication of leap seconds) [1], so Time indeed always does the calculation with unix second input taking it as UTC (including omitting leap seconds), and only then converts to TAI. Similarly for output, but in reverse. I think we have to stick to that standard rather than add to the confusion by making up our own timescales. Indeed, it horrifies me that LSST would be doing just that... Might it not be better to use cxcsec, which is used also by XMM, and is not based on UTC? (but TT).

cc @taldcroft since he has thought about timescales even more than I have (and was responsible for the TimeUnix implementation).

p.s. I'm all too aware that these choices matter as I lost several ours of Keck time while observing over a New Year, finding that the telescope pointing was off because it was one year in UTC and another at Mauna Kea (we only recognized it when we checked sidereal time with other telescopes).

[1] https://en.wikipedia.org/wiki/Unix_time

@taldcroft

p.s. Putting my @taldcroft hat on, I don't think we can change the behaviour even if we thought it was a good idea, if only for backwards compatibility. Quite likely people are already pointing telescopes based on unix time, and are implicitly relying on the current behaviour.

I'm not sure I understand the issue with backwards compatibility. I doubt anyone is trying to use Time with scale="tai" and format="unix" precisely because it gives UTC. If we wanted UTC we would ask for it. I was not proposing any change to any existing usage except that one combination.

It's not a new time standard. It's the value of the CLOCK_TAI clock on linux.

I can imagine that internally astropy might want to keep the time as UTC, or however it presently does it.

Presently, it turns the seconds given into days by dividing by 86400, then adds those to the jd1, jd2 for the epoch to give a UTC jd1, jd2 (as is the definition), and finally does a scale change if someone requested that. Conversely, to give .unix, it converts to UTC, subtracts the epoch to give a difference in days, and multiplies that by 86400. Note that this behaviour is shared by all TimeFromEpoch formats (but with their own scales, obviously).

I remain very much unconvinced that it is a good idea to change this, but if you want, it is trivial to make your own format that is based on TAI: just defining something like

class TimeUnixTAI(TimeUnix):
    """Private LSST scale"""
    name = 'unix_tai'
    epoch_scale = 'tai'

After that, you can to Time.now().unix_tai, etc.

One more point. I find this behavior quite surprising and I would think other users would, as well:

import time
import astropy.time
unix_time = time.time()
date1 = astropy.time.Time(unix_time, scale="tai", format="unix")
date2 = astropy.time.Time(unix_time, scale="utc", format="unix")
assert date1 == date2  # passes
(date1 - date2).sec # 0
date1.jd - date2.jd # 0.00042824074625968933 (37 seconds)

So the two time objects claim to be equal, yet the internal time is different. If tai.unix gave the time I am requesting (TAI using the unix epoch with scale=TAI) the assertion would fail and the time differences would both be 37 seconds.

What do you expect to happen on the day of a leap second? It appears that astropy is gradually applying the leap second, e.g. halfway through the day the time difference is halfway between the before and after leap second values. My original code (which is much messier) gives the answer I would expect on the day of a leap second: the old difference until the new day, then the new difference.

Here is a simplified example halfway through the day of the most recent leap second. Note that tai_minus_utc1 is not general; it needs extra code to handle dates during the leap second itself:

import astropy.time

class TimeUnixTAI(astropy.time.TimeUnix):
    name = 'unix_tai'
    epoch_scale = 'tai'

def tai_minus_utc1(ap_time):
    dt_utc = ap_time.utc.to_datetime()
    dt_tai = ap_time.tai.to_datetime()
    return (dt_tai - dt_utc).total_seconds()

def tai_minus_utc2(ap_time):
    return (ap_time.tai.mjd - ap_time.utc.mjd) * 24 * 60 * 60

def tai_minus_utc3(ap_time):
    return ap_time.tai.unix_tai - ap_time.utc.unix

ap_time = astropy.time.Time("2016-12-31 12:00:00", scale="utc", format="iso")
print(f"tai_minus_utc1: {tai_minus_utc1(ap_time)}")
print(f"tai_minus_utc2: {tai_minus_utc2(ap_time)}")
print(f"tai_minus_utc3: {tai_minus_utc3(ap_time)}")

produces:
tai_minus_utc1: 36.0 # the value I expected
tai_minus_utc2: 36.499994387850165 # wrong
tai_minus_utc3: 36.49999403953552 # wrong

What's going on? Is this some form of compensation for system clocks that might spread the leap second out over the whole day? Is there a way to turn it off/avoid it? I was hoping for true UTC to TAI time conversion.

@mhvk

So just to be clear, the time unix format is an attempt to faithfully implement this definition (from wikipedia):

Unix time (also known as Epoch time, POSIX time,[1] seconds since the Epoch,[2] or UNIX Epoch time[3]) is a system for describing a point in time. It is the number of seconds that have elapsed since the Unix epoch, that is the time 00:00:00 UTC on 1 January 1970, minus leap seconds. Leap seconds are ignored,[4] with a leap second having the same Unix time as the second before it, and every day is treated as if it contains exactly 86400 seconds.

This definition itself is entirely independent of time scale, so it is an oddball and that is why you can find tests that give non-intuitive results like above. However the assert statements given in the original description are consistent with the fundamental definition of unix time being independent of time scale.

The behavior on the day with a leap second is documented to deviate by up to 1 second on a leap day. The rationale for not trying harder here is that unix time is not something that astronomers should be using in production work, in large part because standard unix time is not mappable to a unique absolute time during a leap second.

As said by @mhvk, we are very unlikely to change the existing behavior of the unix format to maintain compatibility.

However, the question of how to get from a clock time on the computer to the corresponding (and correct) absolute Time object is a good one. I took a quick look at this CLOCK_UTC specifier. Is that available on most modern posix systems? Are there linux utilities that provide the current time in UTC format that are correct during a leap second? What about Windows?

Regarding my code sample above that shows 1/2 leap second in the middle of a day: tai_minus_utc2 is an attempt to convert between UTC and TAI with no reliance on unix time. Both UTC and TAI are well defined and always differ by an integer number of seconds.

I filed issue 10055 about it.

At this point my plan is to go from CLOCK_TAI to astropy time by converting CLOCK_TAI to MJD (a trivial linear transformation). Whether the result will be safe to use on the day of a leap second is issue 10055.

Hi everyone,

I'm facing the same problem, we would like to be able to take into account the leap seconds when working with unix format. In the astropy docs it is suggested to define this new class to deal with them. It would be great if it were included by default in astropy already.

>>> from astropy.time.formats import erfa, TimeFromEpoch
>>> class TimeUnixLeap(TimeFromEpoch):
...    """
...    Seconds from 1970-01-01 00:00:00 TAI.  Similar to Unix time
...    but this includes leap seconds.
...    """
...    name = 'unix_leap'
...    unit = 1.0 / erfa.DAYSEC  # in days (1 day == 86400 seconds)
...    epoch_val = '1970-01-01 00:00:00'
...    epoch_val2 = None
...    epoch_scale = 'tai'  # Scale for epoch_val class attribute
...    epoch_format = 'iso'  # Format for epoch_val class attribute

The CLOCK_TAI constant will be added to the time module in Python 3.9. At this point it will be readily available to Python users on Linux platforms. I hope this might encourage you to add support for this standard to astropy.time.

@r-owen

@r-owen - do you have a reference for TAI_CLOCK in python? (I tried but a quick google failed.) If it is exposed in python, that certainly gives more reason to try to support it, but I'd like to see the planned API.

https://bugs.python.org/issue40017 (python/cpython#19096)

The API is: time.CLOCK_TAI will be another https://docs.python.org/3/library/time.html#time-clock-id-constants
that will be available on operating systems that support it. It's was very small pull request that the Python maintainers accepted and merged earlier this week.

Linux supports it and macOS does not. I doubt Windows supports it, because it does not support any of the other constants.

Thanks, both. It may mean we now move into the larger problem of a good name...

I suggest unix_tai or tai_unix. The former is less likely to be mis-typed as tai.unix (which would be my first choice, but seems unlikely).

Nothing is ever easy. I went to just put this new time format into astropy and discovered something annoying (or something I don't understand). From 1972 onward, the offset between UTC and TAI is an exact multiple of 1.0 sec, but before that the diffs were fractional. At the point where the unix time epoch is defined, the offset appears to be 8.000082 sec.

In [9]: (Time('1970-01-01 00:00:00', scale='utc') - Time('1970-01-01 00:00:00', scale='tai')).sec                                                  
Out[9]: 8.00008200000164

So assuming that the definition of tai_unix is "seconds since the Unix time epoch 1970-01-01 UTC" then the class definition is:

class TimeUnixTai(TimeUnix):
    """
    Total seconds from 1970-01-01 00:00:00 UTC.
    For example, 946684800.0 in Unix time is midnight on January 1, 2000.
    """
    name = 'unix_tai'
    epoch_val = '1970-01-01 00:00:08.000082'
    epoch_scale = 'tai'

That gives:

In [16]: t = Time('1970-01-01 00:00:00') 
    ...: print(t.unix) 
    ...: print(t.unix_tai) 
    ...:                                                                                                                                           
0.0
0.0

But then in the modern epoch:

>>> t = Time('2020-01-01 00:00:01') 
>>> t.unix - t.unix_tai 
-28.99991798400879

In other words this 0.000082 is always there, which will probably cause no end of confusion.

So the real question is how is CLOCK_TAI defined? The page https://www.cl.cam.ac.uk/~mgk25/posix-clocks.html says "therefore TAI is a few whole seconds ahead of UTC", but if we can trust SOFA then if you are counting seconds from 1970-01-01, there is another 82 microseconds.

It's clear that the Linux API requires integer second offsets between the UTC and TAI clocks.

SOFA says:

In the 1960-1972 era there were smaller jumps (in either direction) each time the linear UTC(TAI) expression was changed, and these "mini-leaps" are also included in the SOFA convention.

I think this means that Linux is ignoring the mini-leaps and astropy has to ignore them to.

@timj

@timj - that works for me!

See #10081.

@r-owen

@r-owen - thanks for persisting! Still not entirely sure that I like the format, but you definitely convinced me that given that it exists, we should support it. And it is good to see that the solution was relatively painless.

Navigation Menu

Repository navigation

Please support TAI time expressed as unix seconds #10052

Description

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Please support TAI time expressed as unix seconds #10052

Description

Description

Additional context

Activity

mhvk commented on Mar 19, 2020

mhvk commented on Mar 19, 2020

r-owen commented on Mar 19, 2020

mhvk commented on Mar 19, 2020

r-owen commented on Mar 19, 2020

r-owen commented on Mar 20, 2020

taldcroft commented on Mar 20, 2020

r-owen commented on Mar 20, 2020

morcuended commented on Mar 25, 2020

r-owen commented on Mar 25, 2020

mhvk commented on Mar 25, 2020

timj commented on Mar 25, 2020

r-owen commented on Mar 25, 2020

mhvk commented on Mar 25, 2020

r-owen commented on Mar 25, 2020

taldcroft commented on Mar 29, 2020

timj commented on Mar 29, 2020

taldcroft commented on Mar 29, 2020

mhvk commented on Mar 29, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions