Repository navigation
BUG: dt64[s].astype("M8[m]") overflow #22346
Description
Describe the issue:
I'm used to getting silent overflows when converting to higher-resolution units. This one is new to me.
Reproduce the code example:
import numpy as np
dt = np.datetime64(np.iinfo(np.int64).min + 1, "s")
expected = np.datetime64('-292277022657-01-27T08:29')
dt.astype("M8[m]")
>>> dt.astype("M8[m]")
numpy.datetime64('292277026596-12-04T15:29')Error message:
No response
NumPy/Python version information:
numpy 1.23.1
py3.9.13 Mac
Context for the issue:
No response
Activity
seberg commentedon Sep 28, 2022
Interesting, the reason is that we are using this to do the calculation when the value is
< 0:(missing first bracket so that grepping for all occurrences works). In this case
numis 1 anddenomis 60.This is supposed to get the rounding right, which I guess is probably right? But the temporary overflows. Maybe
(dt * num + 1) / denom - 1would help?jbrockmendel commentedon Sep 29, 2022
That's plausible. If viable my solution would be to upstream astype_overflowsafe
seberg commentedon Sep 29, 2022
NumPy does have a code path that goes via the datetimestruct and back (for complicated non-linear time casts), but it is only used for "nonlinear" translation.
It might be nice to try and see if we can just adjust the fast path to fix these, rather do the full, slow path?
Plus, timedelta's have the same problem, and those never use the datetimestruct!
seberg commentedon Sep 30, 2022
OK, so for datetimes, we know that
num == 1 || denom == 1, because if that is not the case we have a "nonlinear" transform (months or years are involved, which we do not define a clear duration for).For that path, I think simply reordering the calculation using that knowledge seems an easy fix for
datetime.For timedelta, going via the datetimestruct would seem to subtly change behavior, because currently it uses the 400 year average duration for years and months?
@jbrockmendel am I understanding it right that the pandas code effectively pretends the timedelta is a datetime for the conversion?
In general, we could throw in overflow detection for the multiplication and return NaT with a warning...
(I will note that I suspect the whole thing is more far more complicated than NumPy is making it to begin with... If we are aiming for perfect time handling.)
jbrockmendel commentedon Oct 3, 2022
Yes.
Y and M are kind of a mess in pandas, not sure how much of that is shared with numpy vs introduced in precision_from_unit.
seberg commentedon Oct 4, 2022
I think that the last part is shared. Did pandas remove support for those timedelta's completely (my pandas version complains:
ValueError: Units 'M', 'Y', and 'y' are no longer supported, as they do not represent unambiguous timedelta values durations.)?Maybe NumPy should consider deprecating these "nonlinear" casts for timedelta?
frbelotto commentedon Oct 4, 2022
Where did you read about such changes on pandas?
I've noticed a weird behavior on a old script after updating some libs but couldn't confirm such change.
seberg commentedon Oct 4, 2022
TST: add targeted tests for numpygh-16352 and numpygh-22346