Repository navigation
BUG: np.array() assumes that bytestrings are ASCII, if mixed with strings #23899
Description
TL;DR: either np.array(["str", b"bytes"]) should be an error for mixing string types, or np.array(["str", b"non-ascii: \xff"]) should not crash. This is probably low priority though.
$ python # version 3.10
>>> import numpy as np
>>> np.__version__
'1.23.5'
# Looks like we coerce input to string types, whether unicode or bytes
>>> np.array([1.337, "str"])
array(['1.337', 'str'], dtype='<U32')
>>> np.array([1.337, b"bytes"])
array([b'1.337', b'bytes'], dtype='|S32')
# and if we _mix_ unicode and bytes (uh-oh!), we silently get unicode
>>> np.array(["str", b"bytes"])
array(['str', 'bytes'], dtype='<U5')
# but this groundlessly assumes that the bytestring is valid ASCII, and boom:
>>> np.array(["str", b"non-ascii: \xff"])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 11: ordinal not in range(128)This is pretty surprising, because in general I expect bytestrings can contain arbitrary bytes, e.g. packed structs, and Numpy is generally pretty good about that. Plausibly related issues:
Activity
seberg commentedon Jun 8, 2023
I think it boils down to NumPy "bytes" still being just a bit like a Python 2 strings :(. Thus, they pretend to be ascii strings (or latin1, but for conversions its ascii).
Not sure how to attack this best though. Probably it needs a way to represent
UCS1(i.e. unicode stored in a single byte) as a decent replacement. There is a need for that anyway for years probably.Once we have that, it would be straight forward to just deprecate that promotion with an error that you should use the UCS1 encoded unicode string.
EDIT: Just for clarity, UCS1 is the same as latin1.
jobh commentedon Jun 8, 2023
ASCII aside, I would expect a string/bytes mixture to end up as a bytes array, since all strings can be promoted to bytes but not vice versa (unicode or not)?
Zac-HD commentedon Jun 8, 2023
All bytestrings can be interpreted as latin-1-encoded text, and all unicode strings can be encoded as utf-32 bytes, but neither of these is compatible with utf-8 and that's probably what users expect.
ngoldbaum commentedon Jun 8, 2023
There’s historical baggage here in that unicode support was added to numpy along with python3 support, and originally the bytes type was the string type. Just pointing that out if what numpy is doing seems weird.
Honestly, mixing strings and bytes like that is unusual enough that if we were going to change numpy’s behavior i’d say we should error in that case and not try to guess about encodings for the user.
ngoldbaum commentedon Oct 18, 2026
Three years later, see #32757.