dtype field names cannot be unicode (Trac #1814)

@jonovik

Original ticket http://projects.scipy.org/numpy/ticket/1814 on 2011-04-29 by @jonovik, assigned to unknown.

Is there a reason why Unicode strings are not accepted as field names for record arrays?

>>> np.dtype([("a", int)])
dtype([('a', '<i4')])
>>> np.dtype([(u"a", int)])
TypeError: data type not understood

A workaround is .encode("ascii").

>>> np.dtype([(u"a".encode("ascii"), int)])
dtype([('a', '<i4')])

It is okay for the type specification to be Unicode.

>>> np.dtype([("a", u"S2")])
dtype([('a', '|S2')])}}}

I came across this while building a record array from data in a Microsoft Excel spreadsheet using pythoncom and win32com. Converting to ascii isn't too much of a hassle, but maybe it wouldn't be difficult to allow Unicode strings as field names?

trac user jkmacc wrote on 2011-11-01

Bump?

I run into this issue all the time when I try to be python3 forward compatible and do

from __future__ import unicode_literals

which means that all strings are unicode. I wonder how this is handled in true python3, since numpy now supports it?

No idea if this is being addressed, but I'm still running into it (python 2.7 + numpy 1.7.0 and python 3.3 + numpy 1.8 dev).
One horrendous workaround I've found to have the code compatible with both python 2 and 3, and have from __future__ import unicode_literals, is using a ternary expression:

np.dtype([("a" if isinstance("a", str) else b"a", int)])

I think you can just use b"foo" and skip the ternary ugliness, on both py2
and py3.
On 7 May 2013 08:02, "evertrol" [email protected] wrote:

No idea if this is being addressed, but I'm still running into it (python
2.7 + numpy 1.7.0 and python 3.3 + numpy 1.8 dev).
One horrendous workaround I've found to have the code compatible with both
python 2 and 3, and have from future import unicode_literals, is
using a ternary expression:

np.dtype([("a" if isinstance("a", str) else b"a", int)])

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2407#issuecomment-17537900
.

No, that's where things go wrong in Python 3:

 >>> np.dtype([(b'a', int)])
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: data type not understood

numpy doesn't like bytes either for the field name. So the more correct title of this issue would be "dtype field names cannot be unicode or bytes".
Identical to above remarks, bytes can be used for the field type specification.

Hence I think this is something that needs to be addressed on the numpy side, so that either unicode or bytes are allowed as the field name (next to str).

On Python 2 and Python 3, the field names must be of type str, so b"foo" won't do. In itself, this is perfectly consistent behavior.

What can be done is allowing ascii encoding on entry. Allowing both str and unicode internally probably leads to a mess.

Oh, I see, sorry. Field names have to be bytes on py2 and unicode on py3,
so normally you just write "foo" and it works on both, but unicode_literals
on py2 breaks this.

In that case wouldn't a workaround be to write str("foo")?

Aside from workarounds, what kind of change would you propose? I don't
think we're likely to start storing Unicode field names on py2; the
compatibility issues would be a mess. Should py2 automatically convert
incoming Unicode strings to bytes, using the default encoding? That's also
a mess, but at least it's an instance of a mess that py3 solves, so
eventually it will go away when the py3 migration is complete... Or if the
str() hack works maybe that's good enough?
On 7 May 2013 08:15, "Pauli Virtanen" [email protected] wrote:

On Python 2 and Python 3, the field names must be of type str so b"foo"won't do.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2407#issuecomment-17538453
.

Doh: str("foo") works. For some reason, I probably figured it couldn't be that easy and tried the hard way. Perhaps I thought that str wouldn't be able to convert the Python 2 unicode to string, but for basic ASCII strings, that's of course not a problem. For me, the str() work-around is good enough; hopefully for others as well. Thanks for that.

Maybe the str() workaround could be included somehow in numpy ? Or is it considered a corner case ? from __future__ import unicode_literals is quite common with the transition between python 2 and 3.

Yes, adding something like

#if defined(NPY_PY3)
tmp = PyUnicode_FromEncodedObject(obj, "ascii", "strict");
if (tmp == NULL) { goto fail; }
#else
tmp = PyUnicode_AsASCIIString(obj);
if (tmp == NULL) { goto fail; }
#endif

probably would be OK.

@pv

I strongly agree with @pv! There is one more corner case, where this solution would be helpful: When unpickling data in python 3 that was written in python 2, one might end up having byte strings in the field names. Currently, unpickle defaults to unpickle byte strings as unicode objects, but this breaks all objects that pickle true binary data as byte strings. In this case, one tweaks the unpickler to leave byte strings as is. See also http://bugs.python.org/issue6784 .

Anyone want to step up to this?

In numpy/core/src/multiarray/descriptor.c field names are tested in two ways.

If the dtype specification is of the list of tuples type, it checks whether each name is a string (as defined by py2 or 3)

PyUString_Check(name)

But if the dtype specification is a dictionary {'names':[ alist], 'formats':[alist]...}, the py2 case also allows unicode names:

#if defined(NPY_PY3K)
        if (!PyUString_Check(name)) {
#else
        if (!(PyUString_Check(name) || PyUnicode_Check(name))) {
#endif

Byte strings are disallowed in PY3 in both cases. titles allow strings and unicode.

http://stackoverflow.com/questions/28586238/does-numpy-recfunctions-append-fields-fail-when-when-array-names-are-unicode/

In this Stackoverflow question, the user created an array with

np.core.records.fromarrays([x1,x2,x3],names=[u'a',u'b',u'c'])

but ran into problems when trying to extend it with

recfunctions.append_fields(FileData,'DateTime', data=DT)

fromarrays uses format_parser, which in turn creates the dtype with the dictionary format. append_fields creates an empty array with a list of tuples dtype: base.dtype.descr + data.dtype.descr. The first allows unicode names (in py2), but the second only strings.

A possible fix is to allow unicode names in the list of tuples case as well. That would be in the _convert_from_array_descr() function.

I haven't looked yet, at whether the unittesting addresses this issue or not. I think the pickling code uses unicode names for maximum compatibility between py2 and py3.

Since this has not been addressed, it is not possible to open numpy structured arrays saved in python3 in python2.

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11)     
>>> a = np.zeros(4, dtype=[('x',int)])
>>> np.save('a.npy', {'a': a})
>>> 
$ python2
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
>>> np.load('a.npy')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 393, in load
    return format.read_array(fid)
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/format.py", line 602, in read_array
    array = pickle.load(fp)
ValueError: non-string names in Numpy dtype unpickling

Is such functionality not going to be supported by numpy?

BUG: fixing to allow unpickling of PY3 pickles from PY2

Would it make sense to merge all of the commits for this issue into numpy/numpy master branch?

@rsaim

@rsaim: What do you mean " the commits for this issue"? The commits that are listed above are just any commit message across the whole of github that contains numpy/numpy#2407 or similar.

Many of those commits are from gh-14275, which was merged already.

@seberg

@seberg, can this be closed in your opinion?

Sounds like gh-14275 fixed the last remaining part here, so yeah.

Navigation Menu

Repository navigation

dtype field names cannot be unicode (Trac #1814) #2407

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

dtype field names cannot be unicode (Trac #1814) #2407

Description

Activity

numpy-gitbot commented on Oct 19, 2012

burnpanck commented on Feb 22, 2013

evertrol commented on May 7, 2013

njsmith commented on May 7, 2013

evertrol commented on May 7, 2013

pv commented on May 7, 2013

njsmith commented on May 7, 2013

evertrol commented on May 7, 2013

Nodd commented on Oct 9, 2013

pv commented on Oct 9, 2013

burnpanck commented on Oct 10, 2013

charris commented on Feb 20, 2014

hpaulj commented on Feb 20, 2015

jmlarson1 commented on Apr 17, 2015

21 remaining items

rsaim commented on Nov 25, 2019

eric-wieser commented on Nov 25, 2019

eric-wieser commented on Nov 25, 2019

seberg commented on Nov 25, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions