The unix_tai time format appears to be off by 8 seconds

Description

The zero point for the new time format unix_tai appears to be off by 8 seconds.

Expected behavior

The new time format unix_tai is defined to match the CLOCK_TAI Linux real-time clock (and I was looking forward to using it, for that reason).

For a current date I expect CLOCK_TAI - CLOCK_UTC = 37 seconds and that is what I observe on a linux system whose leap second table is properly maintained by ntp.
When I try the same computation using Astropy I get 29 seconds.

Actual behavior

Astropy.time seems to believe that CLOCK_TAI - CLOCK_UTC is 29 seconds for current dates. That is what is shown as an example in your docs and is coded in the unit test, so you clearly have a reason for this. But I don't see how it is compatible with CLOCK_TAI.

Steps to Reproduce

1: Find a linux system that has a current leap second table (linux is the only OS I know of that supports the CLOCK_TAI real time clock).
2: Use a reasonably recent Python 3 (I use 3.7)
3: Use AstroPy 4.1 or the current dev version

import time
import astropy.time
clock_tai = getattr(time, "CLOCK_TAI", 11)
now_utc_unix = time.time()
now_tai_unix = time.clock_gettime(clock_tai)
astropy_now_utc = astropy.time.Time(now_utc_unix, format="unix")
print(now_tai_unix - now_utc_unix)
print(astropy_now_utc.unix_tai - now_utc_unix)

System Details

import platform; print(platform.platform())
Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
import sys; print("Python", sys.version)
Python 3.7.8 | packaged by conda-forge | (default, Jul 23 2020, 03:54:19)
[GCC 7.5.0]
import numpy; print("Numpy", numpy.version)
Numpy 1.19.1
import astropy; print("astropy", astropy.version)
astropy 4.3.dev452+gc3132e70a
import scipy; print("Scipy", scipy.version)
Scipy 1.5.2
import matplotlib; print("Matplotlib", matplotlib.version)
Matplotlib 3.0.3

Here is the output I see:

37.00000762939453
29.0

The 29 is expected at least according to the documentation - see https://docs.astropy.org/en/latest/api/astropy.time.TimeUnixTai.html#astropy.time.TimeUnixTai

The difference of 8 may be because the zero point is 1970-01-01 00:00:08 TAI? I.e., the difference in number of seconds should be the number of leap seconds since 1970.

p.s. Your code gives a different result on my Debian system for the built-in package (I have python 3.9.1):

2.956390380859375e-05
29.0

@mhvk

@mhvk the reason you see a different result is that your CLOCK_TAI is not configured. You are seeing the classic issue that CLOCK_TAI is returning the same value as CLOCK_UTC. It takes some care to configure ntp to set the leap second table (we are motivated to go to the trouble because we use TAI to run our telescopes at Vera Rubin Observatory). If you run ptp instead of ntp I think it will always set the leap second table, but thats a much less popular option (one we are considering).

I understand that the 8 second difference probably comes from the (surprising to me) documented zero point. But tai_unix is also documented to match CLOCK_TAI and in my own tests it does not. My own tests show that CLOCK_TAI has the value CLOCK_TAI = CLOCK_UTC + 37 seconds. I have not found a formal specification for CLOCK_TAI, but the 37 second difference is certainly what we expected and what we rely on for our telescope control systems.

@r-owen

@r-owen - the definition currently in astropy is that both unix and unix_tai count seconds since the standard unix Epoch time of 1970-01-01 00:00:00 UTC, but unix_tai includes all seconds whereas unix does not include leap seconds. There have been 29 leap seconds since that Epoch time, hence the difference of 29.

In [2]: t = Time('1970-01-01 00:00:00', scale='utc')

In [3]: t.unix
Out[3]: 0.0

In [4]: t.unix_tai
Out[4]: 8.200000198854696e-05

See #10052 (comment) for context about that 82 us.

Now it sounds like you are saying that unix_tai should count seconds since 1970-01-01 00:00:00 TAI?

@r-owen - If this is the case, can you figure out what went wrong in the discussion of your #10052 issue and #10081 which you and @timj looked at?

This document https://www.cl.cam.ac.uk/~mgk25/posix-clocks.html is the only reference I know about, and it is basically ambiguous about the definition.

CLOCK_TAI
This clock is only available when the system knows International Atomic Time (TAI) with at least an accuracy of 1 s. The only difference between TAI and UTC is that TAI is never corrected by leap seconds, therefore TAI is a few whole seconds ahead of UTC (one second more after every UTC leap second). Some time broadcasting services such as GPS provide both TAI and UTC (e.g., by publishing a scale linked to TAI plus the difference to UTC). TAI is needed for instance to control processes (e.g., astronomical observations, navigation, etc.) where leap seconds are undesirable. CLOCK_TAI is handled very similarly to CLOCK_UTC in that it becomes unavailable when the clock filter algorithm estimates the accuracy of its output to be worse than 1 s.

Basically it doesn't explicitly say if that whole number difference at 1970-01-01 is 0 (makes sense since this is the start of UNIX time) or 8 (total number of leap seconds at that point).

If this needs to be fixed, then we should talk about strategy. Maybe we just change the epoch time to be 1970-01-01 00:00:00 TAI and backport, advertising this as a breaking change due to a conceptual mistake in the original implementation. The other plan would be introducing yet another unix_tai time with that epoch.

In the end, I think we have to match CLOCK_TAI however that is actually implemented in current systems. Hopefully, those are consistent...

I fear we need to go the "advertise as a breaking change route" - having two implementations is just going to be super-confusing.

@r-owen

@r-owen - are you confident that most / all appropriately configured POSIX machines will give this same answer? Is there an easy way to know?

I am reasonably confident because:

It is true for all our computers running ntp. These include a mix of linux, "compact RIO" and PXI computers, all running their own different flavors of linux.
I can't imagine what else unix is going to do. It uses the standard leap second table and that has a value of 37 for today's date.

But I can't prove it's true for all linux systems. It be nice to hear from somebody synching time with ptp.

Here is a non-authoritative answer: https://superuser.com/a/1163770 that explains how to configure ntp to maintain the leap second table and which states that the offset is 37 seconds.

OK, I had entirely forgotten about PR #11143 that got merged only a month again, but it feels like a century ago at this point. I think this does what we agreed we shouldn't do because it would be too confusing. Gahh.

@r-owen

@r-owen - thanks for the feedback and reference. That's good enough for me. Are you OK using unix_ptp? Should we change unix_tai still?

Well, I had completely missed that one! But I did look at the unix_tai documentation and did not see it mentioned, so a minimal follow-up would be to add a cross-reference in the documentation of the two formats... (and probably note there that unix_ptp is likely what one wants...)

@eteq

Re: 4.1 vs 4.2 -- As per discussions on Slack, there is no more planned 4.1.x release, so this has to go into 4.2.x, but we can put a note in Known Issues to state that this bug exists in 4.1 (as part of #11249). And then remove this Known Issues entry at 5.0 release (the next LTS). cc @eteq

Change epoch of TimeUnixTAI to 1970-01-01T00:00:00.0 TAI, fixes astr…

Monkey patching this is possible by the way:

from astropy import version
from astropy.time import TimeUnixTai

if version.major == 4 and version.minor <= 2 and version.bugfix <= 0:
    # fix for astropy #11245
    TimeUnixTai.epoch_val = '1970-01-01 00:00:00.0'
    TimeUnixTai.epoch_scale = 'tai'

Change epoch of TimeUnixTAI to 1970-01-01T00:00:00.0 TAI, fixes astr…

Change epoch of TimeUnixTAI to 1970-01-01T00:00:00.0 TAI, fixes astro…

Merge pull request #11249 from maxnoe/fix_unix_tai

Change epoch of TimeUnixTAI to 1970-01-01T00:00:00.0 TAI, fixes astro…

Merge pull request #11249 from maxnoe/fix_unix_tai

Navigation Menu

Repository navigation

The unix_tai time format appears to be off by 8 seconds #11245

Description

Expected behavior

Actual behavior

Steps to Reproduce

System Details

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

The unix_tai time format appears to be off by 8 seconds #11245

Description

Description

Expected behavior

Actual behavior

Steps to Reproduce

System Details

Activity

r-owen commented on Jan 13, 2021

mhvk commented on Jan 13, 2021

r-owen commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

mhvk commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

r-owen commented on Jan 13, 2021

r-owen commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

taldcroft commented on Jan 13, 2021

mhvk commented on Jan 13, 2021

14 remaining items

pllim commented on Jan 15, 2021

maxnoe commented on Jan 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions