Scalar missing / invalid values and NaN handling for astropy.time.Time

Is there an equivalent of a NaN for astropy.time.Time? Like pandas NaT (Not a Time)?

not isinstance(var, astropy.time.Time)?

I think you are missing the point.

It should be an instance of astropy.time.Time! Just like np.nan is an instance of float.
An instance conveying the meaning that the time is not valid.

Not a direct answer to your question but currently NaT becomes a masked entry in Table (you probably already knows this but I am stating it here for others' future reference):

>>> import pandas as pd
>>> from astropy.table import Table
>>> s = pd.to_datetime(pd.Series(['Jul 31, 2009', '2010-01-10', None]))
>>> df = pd.DataFrame(data={'time': s})
>>> df
        time
0 2009-07-31
1 2010-01-10
2        NaT
>>> t = Table.from_pandas(df)
>>> t
<Table masked=True length=3>
             time            
        datetime64[ns]       
-----------------------------
2009-07-31T00:00:00.000000000
2010-01-10T00:00:00.000000000
                           --

p.s. Series values taken from https://pandas.pydata.org/pandas-docs/stable/timeseries.html

@maxnoe

@maxnoe - there is no NaT-equivalent in astropy Time at this moment, but there is a mostly-complete PR #6028 which adds support. This allows for masked Time arrays much like MaskedColumn in Table or MaskedArray in numpy. It also lets you do all the table operations like join or stacking that can generate missing values.

So unfortunately there is no good news for the current 2.0 or even master, but this should definitely be in the 3.0 release at the end of this year.

@taldcroft

@taldcroft While the PR you mentioned enables support for masking array time values, there is still no support for a scalar time value that signals missing data or invalid values.

We just ran into this again. Why can we not simply allow np.nan as value for the time attributes?
Why explicitly fail?

Missing values occur all the time (no pun intended) and everywhere in the real world. Prohibiting this seems very unreasonable to me.

@maxnoe

@maxnoe - can you give some examples of how you would like to use this? I'm not entirely clear on your use case so examples would help.

We have hardware failures of a certain telescope component sometimes that results in missing timestamps for certain events.

The usecase is simply being able to mark missing/invalid data.

numpy has np.NaT for this in its datetime64 datatype. pandas has something similar.

In [1]: import numpy as np                                                                                                                    

In [2]: np.array([np.nan]).astype('datetime64[s]')                                                                                            
Out[2]: array(['NaT'], dtype='datetime64[s]')

In [3]: np.isfinite(np.array(np.nan).astype('datetime64[s]'))                                                                                 
Out[3]: False

Your example just shows arrays, which are supported now and can give (I think) the same functionality you just showed.

In [5]: t = Time([1, 2], format='cxcsec')                                       

In [6]: t[0] = np.nan                                                           

In [7]: t.mask                                                                  
Out[7]: array([ True, False])

But you said something about a scalar Time object that is effectively "missing". I want to know how you want to use that.

At the simplest level do you want to do Time(np.nan)? What other operations would be useful?

At the simplest level do you want to do Time(np.nan)?
Yes.

What other operations would be useful?

Normal nan propagation. E.g.:

Time(np.nan) + 5 * u.s ⇒ Time(np.nan)
Time.now() + np.nan * u.s ⇒ Time(np.nan)

The second case errors right now, which is why this came up again yesterday.

In my example, the second case is actually a scalar (np.array(np.nan) is a scalar).

With your nan array example, there is a pretty big problem, as there is no scalar nan Time value,
this happens when accessing the value set to nan:

In [1]: from astropy.time import Time
In [2]: import numpy as np
In [3]: t = Time([1, 2], format='unix')
In [4]: t.format = 'iso'
In [5]: t
Out[5]: <Time object: scale='utc' format='iso' value=['1970-01-01 00:00:01.000' '1970-01-01 00:00:02.000']>
In [6]: t[1] = np.nan
In [7]: t
Out[7]: <Time object: scale='utc' format='iso' value=['1970-01-01 00:00:01.000' --]>
In [8]: t[1]
Out[8]: <Time object: scale='utc' format='iso' value=1970-01-01 12:00:00.000>

That deserves a wat?, don't you think?

So what exactly is the motivation to allow setting an element of an array Time to nan but not a single Time.

That is highly confusing to me.

Also, the error one get's for the second case is very unhelpful:

In [20]: Time.now() + np.nan * u.s
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/.local/anaconda3/lib/python3.7/site-packages/astropy/units/quantity_helper/converters.py in converters_and_unit(function, method, *args)
    176             try:
--> 177                 if can_have_arbitrary_unit(maybe_arbitrary_arg):
    178                     converters = [None, None]

~/.local/anaconda3/lib/python3.7/site-packages/astropy/units/quantity_helper/converters.py in can_have_arbitrary_unit(value)
    121     """
--> 122     return np.all(np.logical_or(np.equal(value, 0.), ~np.isfinite(value)))
    123

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-20-285622984270> in <module>
----> 1 Time.now() + np.nan * u.s

~/.local/anaconda3/lib/python3.7/site-packages/astropy/units/quantity.py in __array_ufunc__(self, function, method, *inputs, **kwargs)
    442         # consistent units between two inputs (e.g., in np.add) --
    443         # and the unit of the result (or tuple of units for nout > 1).
--> 444         converters, unit = converters_and_unit(function, method, *inputs)
    445
    446         out = kwargs.get('out', None)

~/.local/anaconda3/lib/python3.7/site-packages/astropy/units/quantity_helper/converters.py in converters_and_unit(function, method, *args)
    190                                 .format(function.__name__,
    191                                         args[0].__class__.__name__,
--> 192                                         args[1].__class__.__name__))
    193
    194         # In the case of np.power and np.float_power, the unit itself needs to

TypeError: Unsupported operand type(s) for ufunc add: 'Time' and 'Quantity'

@maxnoe

@maxnoe - arriving here via #10272, and thinking we should keep the general discussion here. I like the idea of being able to handle nan graciously. One possibility is to make the time immediately masked (which is what is currently done by setting jd2 to nan, and the effects of which you see already a bit above, though also with some problematic examples).

The other would be to have an unmasked nan (say in jd1). An issue with this is that it will cause complaints any time one does, say, a scale transformation and thus call an erfa routine. I'm a little wary of adding more checks around it, unless one can be sure it doesn't slow down regular stuff.

but I did note that t = Time(np.nan, format="mjd")

That's a bit surprising to me, why not accept the nan and mask it?

This is also ... surprising:

In [16]: t = Time(Masked(np.nan, mask=True), format="mjd")

In [17]: t.unmasked.iso
Out[17]: '2000-01-01 12:00:00.000'

In [18]: t.mjd.unmasked
Out[18]: array(51544.5)

@maxnoe

@maxnoe - the erfa functions cannot deal with np.nan, which is why on construction of a Time object any nan get replaced with J2000 by a call to _fill_masked_values. If we want to preserve nan to convert them to NaT later, we could repurpose @taldcroft's original scheme where he used jd2=nan to indicate masked values to know indicate NaT. But it means that on every call into an erfa routine we'd have to replace nan with 0. I think this doesn't have to affect performance too badly if we cache whether or not any nan is present at all. But sadly it means it is not quite trivial to implement either.

p.s. An alternative might be to store nan, inf, and ninf as bits in the mask array, treating it as uint8 - unless special care is taken, though, that would mean every one of those would be seen as masked, which may or may not be good. So, probably too much hackery...

@mvhk

I just discovered that np.isnat is a ufunc, not a numpy function.

@mvhk knows all this of course. I just asked Claude Fable to have a go at supporting np.isnat for Time but adding __array_ufunc__ broke a whole bunch of stuff (initially 28 unrelated failing tests). Claude was able to whittle this down to just 2 failures, but it was looking ugly. Maybe a human expert could do better, but I now have a better understanding of what is going on:

__array_function__ is a safe, additive hook — nothing in the language dispatches through it, and a class that doesn't implement a given function just doesn't support it.
__array_ufunc__ is the opposite: defining it puts your class in the middle of every arithmetic and comparison expression it appears in.

@taldcroft

@taldcroft Yes, I started to had a go at this in #10272, but closed it as I could not get it right at the time

About the np.nan initialization, I have a vague recall of purposely wanting to move away from np.nan as a sentinel or proxy for missing values. Again this relates to NaN not being universally apropos as a missing value in a Time object. I will say that np.ma.masked would not suffer that problem, but it is explicitly excluded as well for reasons I don't recall:

File ~/git/astropy/astropy/utils/masked/core.py:92, in get_data_and_mask(array)
     86         raise AttributeError(
     87             f"'{type(array).__name__}' object has a 'mask' attribute but not an "
     88             "'unmasked' attribute (and is not an np.ma.MaskedArray instance)."
     89         ) from None
     91 if array is np.ma.masked:
---> 92     raise ValueError("cannot handle np.ma.masked.")

Note that np.ma.masked is just a masked 0; the reason that it nevertheless is explicitly set to fail in get_data_and_mask is that conceptually it carries no data, so what should the function return?

On __array_function__ vs __array_ufunc__ - the former is easier in part because Time already carries it from ShapedLikeNDArray, which does the proper forwarding for any unrecognized function. The same can be done for __array_ufunc__ but as one starts from scratch, it is messier. Anyway, I'm fairly sure I could get it to work, but I think we need a plan on how to keep track of nan beforehand, and decide if that's worth it. In principle, the jd2=nan route seems easiest (could even extent to infinities).

I'm very much not enthusiastic about returning to injecting jd2=nan within the internals of TimeFormat. It substantially increases code complexity by having two ways of representing missing values. In fact the nan case might introduce ambiguity about whether nan is missing or just nan and the user can interpret it however they want.

If I understand, the intent of "preserving nan" would be something like:

>>> tm = Time(np.nan, format="jd")
>>> tm.value
np.nan  # True NaN
>>> tm.iso
---  # ?? but probably now missing.

Allowing for np.nan on input for numerical formats seems possible in principle, where it would convert to a masked value just like providing Masked(0, masked=True) (which replaces the data with a fixed value that keeps ERFA happy).

Overall I would prefer to support only np.ma.masked as the sentinel since it is not numeric (conceptually). Then there is just one way to do it, and inputs like Time(["2000:001", np.ma.masked]) would make sense. Even if Time(["2000:001", np.nan]) could be handled, it just looks wrong to me. But I could be swayed on that.

Time(["2000:001", np.nan])

Here I completely agree, but there are many time formats which are represented by floats, for which nan is the most natural, of the top of my head jd, mjd, unix, unix_tai, probably more.

or which nan is the most natural

If you mean for the purpose of indicating a missing value on input, I can certainly be convinced. This is in line with the principle of being liberal on input since other codes may well use nan to represent missing values for numerical time formats.

On output, there can only be one way of representing missing values, and that is Masked.

@taldcroft

Thanks, @taldcroft, for clarifying the discussion. Trying to follow up on that...

For a masked float one can distinguish between "missing" (masked)- and "messed up" (nan), where in normal usage "messed up" means that one tried to do an operation that is not possible, like dividing 0/0 or the log of a negative number.

I guess a primary question is whether it is useful in any way to distinguish between these cases in Time. If we decide it is not, i.e., we want nan and NaT just to signal something is missing or messed up, then we indeed need to be careful on input only, and if one wants nan one can convert masked elements on output (currently not super conveniently, but could be made better).

Distinguishing between missing and messed up has a real cost in code complexity, so the argument for it should be strong.

For Time itself, I think it is weak, since, there are no operations that mess it up. However, TimeDelta can be messed up and this can propagate:

t = Time("J2001")
dt = (t-t)/0
# /usr/lib/python3/dist-packages/astropy/time/utils.py:55: RuntimeWarning: invalid value encountered in scalar divide q1 = sum12 / divisor
dt
# <TimeDelta object: scale='tt' format='jd' value=nan>

At present, Time does not guard against such a nan entering:

t2 = t + dt
t2
# <Time object: scale='tt' format='jyear_str' value=Jnan>
t2.tai
# /usr/lib/python3/dist-packages/erfa/core.py:17280: RuntimeWarning: invalid value encountered in tttai
  tai1, tai2, c_retval = ufunc.tttai(tt1, tt2)
# <Time object: scale='tai' format='jyear_str' value=Jnan>
t2.isot
# /usr/lib/python3/dist-packages/erfa/core.py:16223: RuntimeWarning: invalid value encountered in d2dtf
  iy, im, id, ihmsf, c_retval = ufunc.d2dtf(scale, ndp, d1, d2)
# '-4713-11-24T-2147483648:-2147483648:-2147483648.-2147483648'
t2.datetime
# /usr/lib/python3/dist-packages/erfa/core.py:16223: RuntimeWarning: invalid value encountered in d2dtf
  iy, im, id, ihmsf, c_retval = ufunc.d2dtf(scale, ndp, d1, d2)
# ValueError: year must be in 1..9999, not -4713

I've been thinking a bit more about implications, trying to keep in mind that whatever we do should not lead to a performance hit for the common case.

On input, I actually think the current behaviour is correct: if we are fed something like Time([50000.0, np.nan], format="mjd"), we should "refuse the temptation to guess" and raise an error, not mask the element. It seems fine to me to leave it up to the user to pass in a Masked instance instead (they could use, e.g., Time(np.ma.fix_invalid([50000.0, np.nan]), format="mjd") -- which works as expected (we could document that, provide a similar function for Masked, or even introduce a new keyword argument).

For operations, I'd argue that instituting a post-operation check for nan inside an unmasked Time carries too high a cost for the common case to be worth it: let the warnings suffice. If the Time is masked already, then perhaps it is different. For Masked proper, it was a conscious choice not to do any checks, partially because of the performance hit but also because it is actually hard to do something beforehand that prevents the warnings (the checks in np.ma.MaskedArray are not great). But perhaps Masked itself, or the masked time, could have an option to convert any NaN to masked after each operation. That seems mostly a question of whether it is worth the performance hit and coding effort.

On output, I fear we do need to do something. For the masked case, I think just setting the mask on output is fine. For the unmasked case, changing to masked would seem surprising, and I think I'd prefer to just raise if the NaN cannot be represented (i.e., the non-float output formats). Getting '-4713-11-24T-2147483648:-2147483648:-2147483648.-2147483648' is really no good. And for string outputs, the perfomance hit will be minimal.

For a masked float one can distinguish between "missing" (masked)- and "messed up" (nan), where in normal usage "messed up" means that one tried to do an operation that is not possible, like dividing 0/0 or the log of a negative number.

nan is commonly used for both "messed up" and "missing". The latter "missing" use is actually how I more frequently use it or encounter nan. For instance, Pandas missing data for float, or the FITS standard defines the use of nan for missing or invalid data for floats. Likewise Matplotlib essentially interprets nan as missing and I use this as a feature frequently in plotting.

So basically it is ambiguous which is why I'm not a huge fan of nan. But that doesn't mean we can't try to come up with some handling, it's just that defining something that makes everyone happy may be tricky.

What about saying that for astropy, nan is unambiguously "missing", meaning that it always converts to a masked item. That gives a simple story and would (I think) be relatively straightforward to implement in the current code framework. (though I'm not sure about dt = (t-t)/0).

Navigation Menu

Repository navigation

Scalar missing / invalid values and `NaN` handling for `astropy.time.Time` #6509

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scalar missing / invalid values and NaN handling for astropy.time.Time #6509

Description

Activity

pllim commented on Sep 5, 2017

maxnoe commented on Sep 5, 2017

pllim commented on Sep 5, 2017

taldcroft commented on Sep 5, 2017

maxnoe commented on Apr 30, 2020

taldcroft commented on Apr 30, 2020

maxnoe commented on Apr 30, 2020

taldcroft commented on Apr 30, 2020

taldcroft commented on Apr 30, 2020

maxnoe commented on May 1, 2020

maxnoe commented on May 1, 2020

maxnoe commented on May 1, 2020

mhvk commented on May 2, 2020

33 remaining items

maxnoe commented on Aug 11, 2026

maxnoe commented on Aug 11, 2026

mhvk commented on Aug 11, 2026

taldcroft commented on Aug 11, 2026

maxnoe commented on Aug 11, 2026

taldcroft commented on Aug 11, 2026

mhvk commented on Aug 11, 2026

mhvk commented on Aug 11, 2026

taldcroft commented on Aug 12, 2026

maxnoe commented on Aug 12, 2026

taldcroft commented on Aug 12, 2026

mhvk commented on Aug 12, 2026

mhvk commented on Aug 12, 2026

taldcroft commented on Aug 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Scalar missing / invalid values and `NaN` handling for `astropy.time.Time` #6509