datetime64 arrays don't support buffer protocol

In [4] m = memoryview(np.array(['2001-01-01'], dtype='M8[D]'))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-033a8bae908a> in <module>()
----> 1 m = memoryview(np.array(['2001-01-01'], dtype='M8[D]'))

ValueError: cannot include dtype 'M' in a buffer

Ah, this is going to be a tough one. I don't think memoryview supports datetime.

@abalkin

@abalkin: what do you even want to have happen, given that memoryview
itself does not support datetime64? I guess we could export it as an array
of int64s, but that's already possible with an explicit cast
(arr.view(np.int64)), and probably the explicit form is better anyway...
On 22 Aug 2014 22:01, "Charles Harris" [email protected] wrote:

Ah, this is going to be a tough one. I don't think memoryview supports
datetime.

—
Reply to this email directly or view it on GitHub
#4983 (comment).

I would like to be able to access datetime64 data at the C level without introducing dependency on numpy API. AFAIK, memoryview does not check what the PEP 3118 buffer interface puts in format. It only needs this for its tolist() method which is not implemented many types anyways. For example, in Python 3.4.2:

>>> a = numpy.array([None])
>>> m = memoryview(a)
>>> m.tolist()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: format O not supported

On the other hand, format is happily reported:

>>> m.format
'O'

My proposal is that datetime64 arrays should behave as int64, but report their format as 'M[..]'.

Why does memoryview(zeros(10, dtype="m8[s]")) fail as noted in this bug report, but the following not throw any error?

In [14]: memoryview(zeros(10, dtype=[("A", "m8[s]")]))
Out[14]: <memory at 0x7f1d45d6ff20>

It does report an error if I assign it to a variable. See http://stackoverflow.com/q/28708411/974555 .

@gerritholl

@gerritholl - this is something that should be reported upstream.

@abalkin

@abalkin The upstream behaviour has changed with issue 23571 in Python 3.5, and now it triggers a SystemError. See also #7270. Is it still an upstream bug?

With Python 3.5.1, I get

>>> from numpy import *
>>> memoryview(zeros(10, dtype=[("A", "m8[s]")]))
ValueError: cannot include dtype 'm' in a buffer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: <class 'memoryview'> returned a result with an error set

I believe this shows that there is a bug in Python itself. Apparently, in this situation memoryview constructor sets a ValueError, but returns a non-NULL result.

I suspect the problem is at Objects/memoryobject.c:731,

   (void)PyBuffer_FillInfo(&mbuf->master, NULL, mem, size, readonly,
                            PyBUF_FULL_RO);

where the return value of PyBuffer_FillInfo is ignored. I'll submit an upstream issue.

I'm also seeing this error (python 3.5.1 Numpy 1.11.0), but it looks like people might have solved a very similar problem?

joblib/joblib#188

The accepted solution on Stack Overflow for how to take specific columns of a recarray fails because of this. See http://stackoverflow.com/a/21819324/4323 for the full details, or try:

a = np.array([('a', 'b', '2017-01-01'), ('d', 'e', '2017-01-02')], [('x', 'S1'), ('y', 'S1'), ('z', 'M8[D]')])
np.ndarray(a.shape, a.dtype.descr[:2], a, 0, a.strides)

The expected result is:

array([(b'a', b'b'), (b'd', b'e')], 
      dtype=[('x', 'S1'), ('y', 'S1')])

But it fails with ValueError: cannot include dtype 'M' in a buffer.

So it looks like PR ( #11694 ) made it so that one could do this...

In [1]: import numpy as np                                                      

In [2]: d = np.datetime64('2015-07-04 12:59:59.50', 'ns')                       

In [3]: m = memoryview(d)

However running this as well results in an exception...

In [4]: a = np.asarray([d])                                                     

In [5]: m2 = memoryview(a)                                                      
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-b1ff5f2613f7> in <module>
----> 1 m2 = memoryview(a)

ValueError: cannot include dtype 'M' in a buffer

Would it be possible to extend the same scalar fix to datetime64 and timedelta64 arrays generally?

An idea I had about this (which I might have put elsewhere) was to embed numpy type data in the existing buffer spec by exploiting the fact that T{} uses no storage space.

def _as_buffer(arr):
    try:
        return memoryview(arr)
    except ValueError:
        dt = arr.dtype
        new_dt = np.dtype([
            ('__numpy_value', (np.void, dt.itemsize)),
            ('__numpy_dtype', [
                (str(dt), [])
            ])
        ])
        return memoryview(arr.view(new_dt))

def _from_buffer(b):
    arr = np.asarray(b)
    if np.issubdtype(arr.dtype, np.void) and arr.dtype.names == ('__numpy_value', '__numpy_dtype'):
        new_dtype_name, = arr.dtype['__numpy_dtype'].names
        return arr['__numpy_value'].view(new_dtype_name)
    else:
        return arr

Which works as

>>> a = np.array(['now', 'now'], np.datetime64) + [0, 1]
>>> buffer_a = _as_bufferable(a)
>>> buffer_a.format
'T{8x:__numpy_value:T{T{}:datetime64[s]:}:__numpy_dtype:}'

A scarier approach might be to use pickle.dumps(arr.dtype).encode('hex') as the field name, to ensure that absolutely any user dtype can be used.

Navigation Menu

Repository navigation

datetime64 arrays don't support buffer protocol #4983

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

datetime64 arrays don't support buffer protocol #4983

Description

Activity

charris commented on Aug 22, 2014

njsmith commented on Aug 22, 2014

abalkin commented on Oct 20, 2014

gerritholl commented on Feb 24, 2015

abalkin commented on Mar 15, 2015

gerritholl commented on Feb 17, 2016

abalkin commented on Mar 18, 2016

abalkin commented on Mar 18, 2016

TDaltonC commented on Jul 6, 2016

jzwinck commented on Apr 24, 2017

jakirkham commented on Nov 24, 2018

eric-wieser commented on Jan 10, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions