BUG: astype(object) downcasts for datetime-dtype

The dtype np.object_ is the most general catch-all type to contain arbitrary python objects. As such, .astype(object) should preferably not upcast, but certainly never downcast:

Reproducing code example:

>>> import numpy as np
>>> arr = np.array([10 ** 18], dtype='M8[ns]')
>>> arr
array(['2001-09-09T01:46:40.000000000'], dtype='datetime64[ns]')
>>> arr[0]
numpy.datetime64('2001-09-09T01:46:40.000000000')
>>> arr.astype(object)
array([1000000000000000000], dtype=object)
>>> arr.astype(object)[0]
1000000000000000000

The expected outcome for the last two lines is:

>>> arr.astype(object)
array([numpy.datetime64('2001-09-09T01:46:40.000000000')], dtype=object)
>>> arr.astype(object)[0]
numpy.datetime64('2001-09-09T01:46:40.000000000')

as well as

>>> arr.astype(object)[0] == arr[0]
True

Numpy/Python version information:

numpy: 1.15.4
python: 3.7.1

Note that the result is usually a datetime.timedelta, if one is possible to create:

>>> np.array([10 ** 10], dtype='M8[us]').astype(object)
array([datetime.datetime(1970, 1, 1, 2, 46, 40)], dtype=object)

Note that the result is usually a datetime.~~timedelta~~datetime, if one is possible to create:

Fair enough, as long as it's not an int... ;-)

Well in this case, it doesn't fit in a date time, so we don't have that option.

Is it better to return:

Only datetime64
Only rounded / truncated datetime
A mixture of datetime and int
A mixture of datetime64 and datetime

Right now, we do 3.

To me the answer is clearly 1 (should be clear that a mixture is bound to inconsistencies...).

TBH, I was really surprised that (from your example):

>>> np.array([10 ** 10], dtype='M8[us]').astype(object)
array([datetime.datetime(1970, 1, 1, 2, 46, 40)], dtype=object)

rather than

>>> np.array([10 ** 10], dtype='M8[us]').astype(object)
array([numpy.datetime64('1970-01-01T02:46:40.000000')], dtype=object)

This is also inconsistent because passing an np.datetime64 to a constructor with dtype=object does not change it to datetime.datetime, so I was really surprised that .astype(object) does something different.

>>> np.array([np.datetime64('1970-01-01T02:46:40.000000')], dtype=object)
array([numpy.datetime64('1970-01-01T02:46:40.000000')], dtype=object)
>>>
>>> np.array([np.datetime64('1970-01-01T02:46:40.000000')]).astype(object)
array([datetime.datetime(1970, 1, 1, 2, 46, 40)], dtype=object)

Well, the current state does seem strange. But I am not sure I like to change the behaviour because of something that on first sight seems rather hypothetical? (I mean changing the behaviour to returning datetime64, which seems pretty different.)

Do you actually work with dates past 10000 years? If that is a real use case, maybe we should see that python allows it rather?

@seberg

@seberg

Well, the current state does seem strange. But I am not sure I like to change the behaviour because of something that on first sight seems rather hypothetical? (I mean changing the behaviour to returning datetime64, which seems pretty different.)

I don't really get that point. The underlying values were datetime64 to begin with, why should astype(object) change the values (rather than just the dtype of the array)?
I understand of course that other astype-cases deliberately change the values (indeed, that's often the whole point of the method call), but object is special as the most general container, and should never need to cast at all.

Do you actually work with dates past 10000 years? If that is a real use case, maybe we should see that python allows it rather?

This is not an issue of date range limitations (to me). I'm writing lots of parametrized tests for a larger pandas-PR, and this discrepancy between dtype=object in the constructor and .astype(object) caught me out.

This is annoying, but... If you look closer, you will notice that also other types cast to the python version. For datetime this is a bigger step/change and also annoying.

What I mean is that we use .item()'s "convert to a python object" logic for datetime:

This assumes that numpy datetimes can be safely cast to python datetimes (which is not quite true)
But: Changing this is a pretty large change that needs to be thoroughly discussed (at least as far as I can see). For example, what will happen with matplotlib, and all the other users down there that (unwittingly) are using it to get datetime objects.

So, yes, this is not a safe cast. And yes it is utterly broken. But, I need a a lot more to be convinced that changing the cast from returning datetime objects to returning numpy datetime64 objects will not have a huge impact on downstream.

If such a fix would actually fix bugs for downstream in the long run, it would be more compelling. But it seems to me that for most users the bug will be very mild...

Note that this discrepancy occurs also for all of the numerical types in numpy, even if it may be just as surprising there.

@seberg

@seberg
We got side-tracked a bit with the datetime vs datetime64 - I see your points, even though I believe downstream effort is not a good reason to keep something in an "utterly broken" state. That's why there's version numbers. ;-)

I can work around the datetime/datetime64 thing, but the ints that get returned in the OP are really a killer.

I noticed that years ago I once glanced at changing the default behaviour for all types here (so that the object cast would retain the numpy type). But I am not quite sure we should do it.

EDIT: or well, aim for it maybe. First, there is no way to warn about it. Second, we probably have no clue about how disruptive that would be.

.astype(np.generic) might be an interesting way to allow spelling that, @seberg

…

On Sun, Dec 30, 2018, 3:13 PM Sebastian Berg ***@***.*** wrote: I noticed that years ago I once glanced at changing the default behaviour for all types here (so that the object cast would retain the numpy type). But I am not quite sure we should do it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12550 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZ9LErr9lbXUlPGXTUYlxAzhIqfediNks5u-MnugaJpZM4ZUOsf> .

There seem to be some similar odd casts noted in gh-5180, will close that one in favor if this one.

This is a duplicate of #8546 (closed) and #7619

Is it better to return:

1. Only datetime64

2. Only rounded / truncated datetime

3. A mixture of datetime and int

4. A mixture of datetime64 and datetime

Right now, we do 3.

How about adding a keyword argument to .astype and related methods, such that the user can choose between those four options? I usually want a truncated datetime, but maybe sometimes a datetime64. I never want int.

I don't know if anyone has the cycles/enthusiasm to dive into this one again. But I fixed some infinite recursions here recently which may make this more feasible.
(Of course it is still a breaking change for things that know how to do deal with Python datetimes but not NumPy ones and currently can work via datetime_arr + other_obj.)

BUG: return numpy scalars instead of ints from datetime64/timedelta64…

Navigation Menu

Repository navigation

BUG: astype(object) downcasts for datetime-dtype #12550

Reproducing code example:

Numpy/Python version information:

50 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: astype(object) downcasts for datetime-dtype #12550

Description

Reproducing code example:

Numpy/Python version information:

Activity

eric-wieser commented on Dec 14, 2018

h-vetinari commented on Dec 14, 2018

eric-wieser commented on Dec 14, 2018

h-vetinari commented on Dec 15, 2018

seberg commented on Dec 15, 2018

h-vetinari commented on Dec 17, 2018

seberg commented on Dec 17, 2018

seberg commented on Dec 17, 2018

h-vetinari commented on Dec 18, 2018

seberg commented on Dec 30, 2018

eric-wieser commented on Dec 30, 2018

seberg commented on Jan 3, 2019

eric-wieser commented on Feb 18, 2019

gerritholl commented on Feb 18, 2019

50 remaining items

seberg commented on Aug 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions