Repository navigation
Assignment supports only some bytes-like objects #12075
Description
According to Python Glossary, all of bytes, bytearray, array.array and memoryview support bytes-like interface. However, [:] assignment is supported from all but bytes objects:
import array
import numpy as np
b = bytearray(b'\x00'*8)
arr = np.asarray(b)
arr[:] = bytes(b'\x89HDF\r\n\x1a\n') # fail
arr[:] = bytearray(b'\x89HDF\r\n\x1a\n') # success
arr[:] = array.array('B', b'\x89HDF\r\n\x1a\n') # success
arr[:] = memoryview(b'\x89HDF\r\n\x1a\n') # successarr[:] = bytes(b'\x89HDF\r\n\x1a\n')
UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 0: ordinal not in range(128)It is proposed that numpy.array supports assignment from bytes objects.
Update by seberg (~NumPy 1.23) the error is now:
ValueError: invalid literal for int() with base 10: b'\x89HDF\r\n\x1a\n'
Activity
aparamon commentedon Oct 4, 2018
This should make
numpy.arraya reliable target of readinto().eric-wieser commentedon Oct 4, 2018
eric-wieser commentedon Oct 4, 2018
Actually, i have no idea if what I said is true. Can you show what arr.dtype is?
aparamon commentedon Oct 4, 2018
@eric-wieser
arr.dtypeis actuallyuint8, which is in line with my expectations fromarr = np.asarray(b)call.njsmith commentedon Oct 4, 2018
Probably at least partly a holdover from python 2, where bytes was
strand you definitely wouldn't want to silently unpack it into a sequence of individual bytes.In Python 3 this unpacking does feel more natural, but the issue isn't totally gone... Arrays treat
bytesas a scalar in lots of contexts, so treating them as a sequence here could be confusing. I'm not sure.njsmith commentedon Oct 4, 2018
For example, I don't think we can make
np.asarray(b"hello")start returning an array of uint8. That would be really surprising. But I definitely wouldn't expect the code above to work if you wrotearr[:] = np.asarray(b"hello"), since that's an unambiguous dimension mismatch.aparamon commentedon Oct 5, 2018
No disagreement that
np.asarray(b"hello")is expected to fail due to ambiguity. That's why I'm explicitly initializingarr = np.asarray(bytearray(b"hello"))which expectedly and correctly selectsdtype = np.uint8.My point is that assignment
arr[:] = b"hello"should be smart enough to treat the right side asmemoryview(b"hello")in this case, i.e. as bytes-like object not null-terminated byte-string.njsmith commentedon Oct 5, 2018
Generally speaking though,
arr[:] = ...is equivalent toarr[:] = np.asarray(...). For example:So should we break this, or should
...assignment do something different depending on the dimension of the target array, or... what?aparamon commentedon Oct 5, 2018
I see no reason why we can't have both
and
depending on the
dtype.aparamon commentedon Oct 5, 2018
Very basically, I'm proposing
(please do not consider a PR ;-) )
eric-wieser commentedon Oct 5, 2018
At the very least I think we can produce a more useful error message here. Getting a unicode decode error here is super surprising
njsmith commentedon Oct 5, 2018
That's true... why do we get a unicode decode error here?
I'm very unenthused about the idea of adding (more) dtype-based special-cases to
__setattr__though...@aparamon what are you trying to do here by the way? I'm actually somewhat surprised that
np.asarray(bytearray(...))gives an array of uint8 instead of a bytes scalar...aparamon commentedon Oct 5, 2018
np.version.versiongives "1.14.3".I believe
np.asarray()regardingbytearrayas array of bytes (np.uint8s) is perfectly logical :-)But let's make sure we're not discussing pros and contras of this behavior. I should have probably used
for more straightforward example.
Successful assignment into
np.ndarrayofnp.uint8s from different bytes-like objects, followed by failure to assign from actual bytes caught me by large surprise.I'm using
np.ndarrayofnp.uint8s as the universal target of binaryreadinto():https://github.com/h5py/h5py/blob/master/h5py/h5fd.pyx#L159
because
np.asarray()appeared to be the only universally supported (among many Pythons, Cythons) way to wrap C bytes buffer for write without memcpy.njsmith commentedon Oct 5, 2018
I feel like
memoryviewwould be a better tool to use there? It's in Python since 2.7, and if anything modern Cython actually prefers that raw byte interfaces use memoryviews instead of ndarray...aparamon commentedon Oct 5, 2018
@njsmith I think you are mistaken:
PyMemoryView_FromMemoryis only available in Python 3:https://docs.python.org/3/c-api/memoryview.html
np.ndarrayseemed a life-saver initially...njsmith commentedon Oct 5, 2018
I'm pretty sure Cython is actually converting your data into a memoryview, before passing it to
np.asarray...PyMemoryView_FromMemorymay not exist on python 2, but cython rarely lets things like that stop it :-)aparamon commentedon Oct 5, 2018
@njsmith Haha, you were absolutely right! The code works equally well (and even better wrt assignment from bytes) without
np.asarray()calls.But still, I stay confident that the numpy issue reported should be addressed.
aparamon commentedon Oct 5, 2018
My joy was a bit premature: Cython
memoryviewcan't be assigned from bytes too, this time with message:https://stackoverflow.com/a/29067248/751247
I will submit an issue for Cython too.
eric-wieser commentedon Oct 15, 2018
@njsmith
You can thank 737f690, which seems to work around a python bug. For some reason,
float(b'1.0')andint(b'1')are both fine, yetcomplex(b'1')is not. To work around this, we try to decode to unicode first.Note that https://bugs.python.org/issue27572 is trying to deprecate the set of these that do currently work anyway.