Repository navigation
Unexpected dependence on chunks with JSON and MsgPack codecs #258
Description
import numcodecs
import zarr
z = zarr.empty(1, dtype=object, object_codec=numcodecs.JSON(), chunks=(1,))
z[0] = ["11"]
print(z[:])
z[0] = ["11", "1"]
print(z[:]) # Borksgives
['11']
Traceback (most recent call last):
File "dev.py", line 204, in <module>
print(z[:])
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 555, in __getitem__
return self.get_basic_selection(selection, fields=fields)
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 681, in get_basic_selection
fields=fields)
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 723, in _get_basic_selection_nd
return self._get_selection(indexer=indexer, out=out, fields=fields)
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 1011, in _get_selection
drop_axes=indexer.drop_axes, fields=fields)
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 1604, in _chunk_getitem
chunk = self._decode_chunk(cdata)
File "/home/jk/.local/lib/python3.5/site-packages/zarr/core.py", line 1747, in _decode_chunk
chunk = chunk.reshape(self._chunks, order=self._order)
ValueError: cannot reshape array of size 2 into shape (1,)
Problem description
When using the JSON on MsgPack codecs to store an array of lists of strings, I get the above weird behaviour. It's easiest to just play around with the snippet above and try out different things, as it's tricky to explain what's happening. Basically though, it seems that this code path is sensitive to the chunk size in a way that others aren't --- typically, the library is perfectly happy with any chunk size.
Activity
jeromekelleher commentedon Apr 26, 2018
This happens with non-trivial chunk sizes too. Here we've got a chunk size of 1024 and I'm after inserting 1024 copies of ["0", "1"].
alimanfoo commentedon Apr 26, 2018
Thanks, looks like the shape of the array is not getting preserved when encoding then decoding a chunk. Both of these codecs first convert an incoming numpy array to a list, then encode, e.g. this line. In the opposite direction, the JSON is decoded to a list, then the list is passed to np.array(), e.g. this line. Suspect the problem will be there somewhere.
jeromekelleher commentedon Apr 27, 2018
Closed in favour of zarr-developers/numcodecs#76
jeromekelleher commentedon May 3, 2018
Reopening this one, as I think the problem really is in zarr rather than numcodecs. So, I think the solution to the problem is to force object arrays to always be 1D. Somehow, when calling setitem for these arrays we need to do
rather than
I had a quick look around, but I wasn't able to figure out how we could do this specifically for object arrays.
What do you think @alimanfoo?
jeromekelleher commentedon May 3, 2018
Closing again... turns out we do have a problem in numcodecs.