Skip to content

Commit 8fc855b

Browse files
jbrockmendelclaude
andcommitted
BUG: return numpy scalars instead of ints from datetime64/timedelta64 astype(object)
convert_datetime_to_pyobject and convert_timedelta_to_pyobject returned raw Python ints for values that could not be represented as Python datetime.datetime or datetime.timedelta (sub-microsecond units, out-of-range years, leap seconds, overflowing timedelta days). This caused arr.astype(object) to silently produce integers, breaking roundtrips and surprising users. Replace those PyLong_FromLongLong fallbacks with PyArray_Scalar calls that return proper numpy.datetime64 / numpy.timedelta64 scalars. Because getitem now returns numpy scalars in those cases, int() and float() on datetime64/timedelta64 scalars would infinitely recurse through gentype_int -> array_int -> getitem -> gentype_int. Break the recursion by giving the two scalar types their own tp_as_number with nb_int/nb_float that raise TypeError, consistent with how int(datetime.datetime(...)) and int(datetime.timedelta(...)) already behave. Closes numpy#12550 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
1 parent 758d986 commit 8fc855b

10 files changed

Lines changed: 169 additions & 150 deletions

File tree

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
``datetime64`` and ``timedelta64`` now always return numpy scalars from ``.item()`` and ``.astype(object)``
2+
---------------------------------------------------------------------------------------------------------
3+
4+
Previously, ``.item()`` and ``.astype(object)`` on ``datetime64`` and
5+
``timedelta64`` arrays returned a mix of Python types depending on the unit
6+
and value: ``datetime.datetime``, ``datetime.date``, ``datetime.timedelta``,
7+
``None`` (for NaT), or raw ``int`` (for sub-microsecond units or out-of-range
8+
values). They now always return ``numpy.datetime64`` or ``numpy.timedelta64``
9+
scalars.
10+
11+
As a consequence, ``int()`` and ``float()`` on ``datetime64`` and
12+
``timedelta64`` scalars now raise ``TypeError`` for all units. Previously
13+
they returned the underlying integer for sub-microsecond units.

‎doc/source/reference/arrays.datetime.rst‎

Lines changed: 15 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -330,58 +330,43 @@ us / μs microsecond +/- 2.9e5 years [290301 BC, 294241 AD]
330330
Converting datetime and timedelta to Python Object
331331
==================================================
332332

333-
NumPy follows a strict protocol when converting `datetime64` and/or `timedelta64` to Python Objects (e.g., ``tuple``, ``list``, `datetime.datetime`).
334-
335-
The protocol is described in the following table:
336-
337-
================================ ================================= ==================================
338-
Input Type for `datetime64` for `timedelta64`
339-
================================ ================================= ==================================
340-
``NaT`` ``None`` ``None``
341-
ns/ps/fs/as ``int`` ``int``
342-
μs/ms/s/m/h `datetime.datetime` `datetime.timedelta`
343-
D/W (Linear units) `datetime.date` `datetime.timedelta`
344-
Y/M (Non-linear units) `datetime.date` ``int``
345-
Generic units `datetime.date` ``int``
346-
================================ ================================= ==================================
333+
When converting `datetime64` and `timedelta64` to Python objects (e.g., via
334+
``.item()``, ``.astype(object)``, or ``list()``), NumPy always returns
335+
`numpy.datetime64` or `numpy.timedelta64` scalars, regardless of the unit or
336+
value.
347337

348338
.. admonition:: Example
349339

350340
.. try_examples::
351341

352342
>>> import numpy as np
353343

354-
>>> type(np.datetime64('NaT').item())
355-
<class 'NoneType'>
356-
357-
>>> type(np.timedelta64('NaT').item())
358-
<class 'NoneType'>
359-
360-
>>> type(np.timedelta64(123, 'ns').item())
361-
<class 'int'>
362-
363344
>>> type(np.datetime64('2025-01-01T12:00:00.123456').item())
364-
<class 'datetime.datetime'>
345+
<class 'numpy.datetime64'>
365346

366347
>>> type(np.timedelta64(10, 'D').item())
367-
<class 'datetime.timedelta'>
348+
<class 'numpy.timedelta64'>
368349

350+
>>> type(np.datetime64('NaT').item())
351+
<class 'numpy.datetime64'>
352+
353+
>>> type(np.timedelta64(123, 'ns').item())
354+
<class 'numpy.timedelta64'>
369355

370-
In the case where conversion of `datetime64` and/or `timedelta64` is done against Python types like ``int``, ``float``, and ``str`` the corresponding return types will be ``np.str_``, ``np.int64`` and ``np.float64``.
371356

357+
When converting `datetime64` or `timedelta64` to ``int``, ``float``, or
358+
``str`` via ``.astype()``, the corresponding return types will be
359+
``np.int64``, ``np.float64``, and ``np.str_``.
372360

373361
.. admonition:: Example
374362

375363
.. try_examples::
376364

377365
>>> import numpy as np
378-
366+
379367
>>> type(np.timedelta64(1, 'D').astype(int))
380368
<class 'numpy.int64'>
381369

382-
>>> type(np.datetime64('2025-01-01T12:00:00.123456').astype(float))
383-
<class 'numpy.float64'>
384-
385370
>>> type(np.timedelta64(123, 'ns').astype(str))
386371
<class 'numpy.str_'>
387372

‎numpy/_core/src/multiarray/_datetime.h‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -242,20 +242,15 @@ convert_pyobject_to_timedelta(PyArray_DatetimeMetaData *meta, PyObject *obj,
242242
/*
243243
* Converts a datetime into a PyObject *.
244244
*
245-
* NaT (Not-a-time) is returned as None.
246-
* For D/W/Y/M (days or coarser), returns a datetime.date.
247-
* For μs/ms/s/m/h/D/W (microseconds or coarser), returns a datetime.datetime.
248-
* For ns/ps/fs/as (units shorter than microseconds), returns an integer.
245+
* Always returns a numpy.datetime64 scalar (including NaT).
249246
*/
250247
NPY_NO_EXPORT PyObject *
251248
convert_datetime_to_pyobject(npy_datetime dt, PyArray_DatetimeMetaData *meta);
252249

253250
/*
254251
* Converts a timedelta into a PyObject *.
255252
*
256-
* NaT (Not-a-time) is returned as None.
257-
* For μs/ms/s/m/h/D/W (microseconds or coarser), returns a datetime.timedelta.
258-
* For Y/M (non-linear units), generic units and ns/ps/fs/as (units shorter than microseconds), returns an integer.
253+
* Always returns a numpy.timedelta64 scalar (including NaT).
259254
*/
260255
NPY_NO_EXPORT PyObject *
261256
convert_timedelta_to_pyobject(npy_timedelta td, PyArray_DatetimeMetaData *meta);

‎numpy/_core/src/multiarray/datetime.c‎

Lines changed: 22 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -2774,58 +2774,33 @@ convert_pyobject_to_timedelta(PyArray_DatetimeMetaData *meta, PyObject *obj,
27742774
}
27752775
}
27762776

2777+
/*
2778+
* Creates a numpy datetime64 or timedelta64 scalar from a raw value
2779+
* and metadata. Used as a fallback when the value cannot be represented
2780+
* as a Python datetime.datetime or datetime.timedelta.
2781+
*/
2782+
static PyObject *
2783+
_make_datetime_or_timedelta_scalar(
2784+
int type_num, npy_int64 val, PyArray_DatetimeMetaData *meta)
2785+
{
2786+
PyArray_Descr *dtype = create_datetime_dtype(type_num, meta);
2787+
if (dtype == NULL) {
2788+
return NULL;
2789+
}
2790+
PyObject *ret = PyArray_Scalar(&val, dtype, NULL);
2791+
Py_DECREF(dtype);
2792+
return ret;
2793+
}
2794+
27772795
/*
27782796
* Converts a datetime into a PyObject *.
27792797
*
2780-
* NaT (Not-a-time) is returned as None.
2781-
* For D/W/Y/M (days or coarser), returns a datetime.date.
2782-
* For μs/ms/s/m/h/D/W (microseconds or coarser), returns a datetime.datetime.
2783-
* For ns/ps/fs/as (units shorter than microseconds), returns an integer.
2798+
* Always returns a numpy.datetime64 scalar (including NaT).
27842799
*/
27852800
NPY_NO_EXPORT PyObject *
27862801
convert_datetime_to_pyobject(npy_datetime dt, PyArray_DatetimeMetaData *meta)
27872802
{
2788-
PyObject *ret = NULL;
2789-
npy_datetimestruct dts;
2790-
2791-
/*
2792-
* Convert NaT (not-a-time) and any value with generic units
2793-
* into None.
2794-
*/
2795-
if (dt == NPY_DATETIME_NAT || meta->base == NPY_FR_GENERIC) {
2796-
Py_RETURN_NONE;
2797-
}
2798-
2799-
/* If the type's precision is greater than microseconds, return an int */
2800-
if (meta->base > NPY_FR_us) {
2801-
return PyLong_FromLongLong(dt);
2802-
}
2803-
2804-
/* Convert to a datetimestruct */
2805-
if (NpyDatetime_ConvertDatetime64ToDatetimeStruct(meta, dt, &dts) < 0) {
2806-
return NULL;
2807-
}
2808-
2809-
/*
2810-
* If the year is outside the range of years supported by Python's
2811-
* datetime, or the datetime64 falls on a leap second,
2812-
* return a raw int.
2813-
*/
2814-
if (dts.year < 1 || dts.year > 9999 || dts.sec == 60) {
2815-
return PyLong_FromLongLong(dt);
2816-
}
2817-
2818-
/* If the type's precision is greater than days, return a datetime */
2819-
if (meta->base > NPY_FR_D) {
2820-
ret = PyDateTime_FromDateAndTime(dts.year, dts.month, dts.day,
2821-
dts.hour, dts.min, dts.sec, dts.us);
2822-
}
2823-
/* Otherwise return a date */
2824-
else {
2825-
ret = PyDate_FromDate(dts.year, dts.month, dts.day);
2826-
}
2827-
2828-
return ret;
2803+
return _make_datetime_or_timedelta_scalar(NPY_DATETIME, dt, meta);
28292804
}
28302805

28312806
/*
@@ -2961,46 +2936,12 @@ convert_timedelta_to_timedeltastruct(PyArray_DatetimeMetaData *meta,
29612936
/*
29622937
* Converts a timedelta into a PyObject *.
29632938
*
2964-
* NaT (Not-a-time) is returned as None.
2965-
* For μs/ms/s/m/h/D/W (microseconds or coarser), returns a datetime.timedelta.
2966-
* For Y/M (non-linear units), generic units and ns/ps/fs/as (units shorter than microseconds), returns an integer.
2939+
* Always returns a numpy.timedelta64 scalar (including NaT).
29672940
*/
29682941
NPY_NO_EXPORT PyObject *
29692942
convert_timedelta_to_pyobject(npy_timedelta td, PyArray_DatetimeMetaData *meta)
29702943
{
2971-
npy_timedeltastruct tds;
2972-
2973-
/*
2974-
* Convert NaT (not-a-time) into None.
2975-
*/
2976-
if (td == NPY_DATETIME_NAT) {
2977-
Py_RETURN_NONE;
2978-
}
2979-
2980-
/*
2981-
* If the type's precision is greater than microseconds, is
2982-
* Y/M (nonlinear units), or is generic units, return an int
2983-
*/
2984-
if (meta->base > NPY_FR_us ||
2985-
meta->base == NPY_FR_Y ||
2986-
meta->base == NPY_FR_M ||
2987-
meta->base == NPY_FR_GENERIC) {
2988-
return PyLong_FromLongLong(td);
2989-
}
2990-
2991-
if (convert_timedelta_to_timedeltastruct(meta, td, &tds) < 0) {
2992-
return NULL;
2993-
}
2994-
2995-
/*
2996-
* If it would overflow the datetime.timedelta days, return a raw int
2997-
*/
2998-
if (tds.day < -999999999 || tds.day > 999999999) {
2999-
return PyLong_FromLongLong(td);
3000-
}
3001-
else {
3002-
return PyDelta_FromDSU(tds.day, tds.sec, tds.us);
3003-
}
2944+
return _make_datetime_or_timedelta_scalar(NPY_TIMEDELTA, td, meta);
30042945
}
30052946

30062947
/*

‎numpy/_core/src/multiarray/scalartypes.c.src‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3786,6 +3786,29 @@ static PyObject *
37863786
static PyNumberMethods @name@_arrtype_as_number;
37873787
/**end repeat**/
37883788

3789+
static PyNumberMethods datetime_arrtype_as_number;
3790+
static PyNumberMethods timedelta_arrtype_as_number;
3791+
3792+
static PyObject *
3793+
datetimetype_int(PyObject *self)
3794+
{
3795+
PyErr_Format(PyExc_TypeError,
3796+
"int() argument must be a string, a bytes-like object "
3797+
"or a real number, not '%.200s'",
3798+
Py_TYPE(self)->tp_name);
3799+
return NULL;
3800+
}
3801+
3802+
static PyObject *
3803+
datetimetype_float(PyObject *self)
3804+
{
3805+
PyErr_Format(PyExc_TypeError,
3806+
"float() argument must be a string or a real number, "
3807+
"not '%.200s'",
3808+
Py_TYPE(self)->tp_name);
3809+
return NULL;
3810+
}
3811+
37893812
/* Arithmetic methods -- only so we can override &, |, ^. */
37903813
NPY_NO_EXPORT PyNumberMethods bool_arrtype_as_number = {
37913814
.nb_bool = (inquiry)bool_arrtype_nonzero,
@@ -4476,6 +4499,21 @@ initialize_numeric_types(void)
44764499

44774500
/**end repeat**/
44784501

4502+
/*
4503+
* Datetime and timedelta types should not be convertible to int/float.
4504+
* Their getitem returns numpy scalars (datetime64/timedelta64), not
4505+
* Python scalars, so gentype_int/gentype_float would infinitely recurse.
4506+
*/
4507+
datetime_arrtype_as_number = gentype_as_number;
4508+
datetime_arrtype_as_number.nb_int = (unaryfunc)datetimetype_int;
4509+
datetime_arrtype_as_number.nb_float = (unaryfunc)datetimetype_float;
4510+
PyDatetimeArrType_Type.tp_as_number = &datetime_arrtype_as_number;
4511+
4512+
timedelta_arrtype_as_number = gentype_as_number;
4513+
timedelta_arrtype_as_number.nb_int = (unaryfunc)datetimetype_int;
4514+
timedelta_arrtype_as_number.nb_float = (unaryfunc)datetimetype_float;
4515+
PyTimedeltaArrType_Type.tp_as_number = &timedelta_arrtype_as_number;
4516+
44794517
PyStringArrType_Type.tp_alloc = NULL;
44804518
PyStringArrType_Type.tp_free = NULL;
44814519

‎numpy/_core/tests/test_array_coercion.py‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -425,15 +425,20 @@ def test_coercion_basic(self, dtype, scalar):
425425
[param(np.timedelta64(123, "ns"), id="timedelta64[ns]"),
426426
param(np.timedelta64(12, "generic"), id="timedelta64[generic]")])
427427
def test_coercion_timedelta_convert_to_number(self, dtype, scalar):
428-
# Only "ns" and "generic" timedeltas can be converted to numbers
429-
# so these are slightly special.
428+
# "ns" and "generic" timedeltas can be converted to numeric dtypes
429+
# via np.array(scalar, dtype=...) and .astype(), but integer item
430+
# assignment raises TypeError because it goes through int().
430431
arr = np.array(scalar, dtype=dtype)
431432
cast = np.array(scalar).astype(dtype)
432-
ass = np.ones((), dtype=dtype)
433-
ass[()] = scalar # raises, as would np.array([scalar], dtype=dtype)
434-
435433
assert_array_equal(arr, cast)
436-
assert_array_equal(cast, cast)
434+
435+
ass = np.ones((), dtype=dtype)
436+
if np.issubdtype(dtype, np.integer):
437+
with pytest.raises(TypeError):
438+
ass[()] = scalar
439+
else:
440+
ass[()] = scalar
441+
assert_array_equal(ass, cast)
437442

438443
@pytest.mark.parametrize("dtype", ["S6", "U6"])
439444
@pytest.mark.parametrize(["val", "unit"],

0 commit comments

Comments
 (0)