Unicode scalar problem

Creating an array from a list of unicode scalars doesn't work when the scalars contain non-ascii characters.

I'm using python 2.7 (32-bit) on windows with numpy 1.8.1 .

Example:

In [1]: import numpy as np

In [2]: np.array(u"åäö")
Out[2]:
array(u'\xe5\xe4\xf6', dtype='<U3')

In [3]: [np.array(u"åäö")]
Out[3]:
[array(u'\xe5\xe4\xf6', dtype='<U3')]

In [4]: np.array([np.array([u"åäö"])])
Out[4]:
array([[u'\xe5\xe4\xf6']], dtype='<U3')

In [5]: np.array([np.array(u"åäö")])
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-4-918a52a6946b> in <module>()
----> 1 np.array([np.array(u"åäö")])

c:\python27\lib\site-packages\numpy\core\numeric.pyc in array_str(a, max_line_width, precision, suppress_small)
   1613
   1614     """
-> 1615     return array2string(a, max_line_width, precision, suppress_small, ' ', "", str)
   1616
   1617 def set_string_function(f, repr=True):

c:\python27\lib\site-packages\numpy\core\arrayprint.pyc in array2string(a, max_line_width, precision, suppress_small, separator, prefix, style, formatter)
    446             if isinstance(x, tuple):
    447                 x = _convert_arrays(x)
--> 448             lst = style(x)
    449     elif reduce(product, a.shape) == 0:
    450         # treat as a null array if any of shape elements == 0

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

The issue seems to be only in printing the arrays, creation etc. themselves seem to work fine.

array_print is called with str which won't work in python2
maybe one can add a try except around it and use unicode

For me it is not only on printing but also on creation see below where I try to assign the array to a variable.

>>> a = np.array([np.array(u"åäö")])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\python27\lib\site-packages\numpy\core\numeric.py", line 1615, in array_str
    return array2string(a, max_line_width, precision, suppress_small, ' ', "", str)
  File "C:\python27\lib\site-packages\numpy\core\arrayprint.py", line 448, in array2string
    lst = style(x)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

I do not see this problem on python 3.4 with numpy 1.9.2 but it is still present for python 2.7 with numpy 1.9.2

Seems like something pretty nasty is going on here - array_str should definitely not be on the construction path for np.array.

If you modify array2string to catch this array and emit a new string, then this string ends up in the array instead!

Reproduction on python 3 with this small function

from contextlib import contextmanager

@contextmanager
def alarming_str():
    np.core.numeric.set_string_function(lambda x: 'WAT', repr=False)
    try:
        yield
    finally:
         np.core.numeric.set_string_function(None, repr=False)

giving

>>> with alarming_str():
...     q = np.array([np.array(u"åäö")])
>>> q[0]
'WAT'

And

>>> q = np.array([u'abc'])
>>> q0 = np.array(u'def')
>>> with alarming_str():
...     q[0] = q0
>>> q[0]
'WAT'

Cause: UNICODE_setitem is alarmingly different from STRING_setitem, and each probably have their own set of bugs

@jstenar

Thanks for the report, @jstenar!

Navigation Menu

Repository navigation

Unicode scalar problem #5049

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unicode scalar problem #5049

Description

Activity

pv commented on Sep 7, 2014

juliantaylor commented on Sep 7, 2014

jstenar commented on Sep 7, 2014

jstenar commented on Jun 3, 2015

eric-wieser commented on Apr 6, 2017

eric-wieser commented on Apr 6, 2017

eric-wieser commented on Apr 6, 2017

eric-wieser commented on Apr 11, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions