Repository navigation
Supporting structured array records with non-trivial shape #111
Description
With a structured array in NumPy, it is possible to create a record that has a non-trivial shape of its own in addition to the shape of the recorded array. An example use case would be having a record that is a centroid of some object. The centroid will have dimensions of whatever it had come from in addition to there being one centroid per record for how many ever records there are. It appears that Zarr does not support this use case ATM. Not sure if that is a bug/unsupported feature or an intentional restriction.
Example of the error:
Details
In [1]: import numpy
In [2]: import zarr
In [3]: a = numpy.array([(2, [4.0, 6.0]), (3, [5.0, 7.0])], dtype=[("a", numpy.uint64), ("b", numpy.float, (2,))])
In [4]: z = zarr.array(a)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_array_metadata(s)
23 try:
---> 24 dtype = decode_dtype(meta['dtype'])
25 fill_value = decode_fill_value(meta['fill_value'], dtype)
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_dtype(d)
78 def decode_dtype(d):
---> 79 d = _decode_dtype_descr(d)
80 return np.dtype(d)
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in _decode_dtype_descr(d)
73 else:
---> 74 d = [(f, _decode_dtype_descr(v)) for f, v in d]
75 return d
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in <listcomp>(.0)
73 else:
---> 74 d = [(f, _decode_dtype_descr(v)) for f, v in d]
75 return d
ValueError: too many values to unpack (expected 2)
During handling of the above exception, another exception occurred:
MetadataError Traceback (most recent call last)
<ipython-input-4-3d14a571b8d9> in <module>()
----> 1 z = zarr.array(a)
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/creation.py in array(data, **kwargs)
306
307 # instantiate array
--> 308 z = create(**kwargs)
309
310 # fill with data
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/creation.py in create(shape, chunks, dtype, compressor, fill_value, order, store, synchronizer, overwrite, path, chunk_store, filters, cache_metadata, **kwargs)
88 # instantiate array
89 z = Array(store, path=path, chunk_store=chunk_store,
---> 90 synchronizer=synchronizer, cache_metadata=cache_metadata)
91
92 return z
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in __init__(self, store, path, read_only, chunk_store, synchronizer, cache_metadata)
98
99 # initialize metadata
--> 100 self._load_metadata()
101
102 # initialize attributes
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in _load_metadata(self)
108 """(Re)load metadata from store."""
109 if self._synchronizer is None:
--> 110 self._load_metadata_nosync()
111 else:
112 mkey = self._key_prefix + array_meta_key
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in _load_metadata_nosync(self)
123
124 # decode and store metadata
--> 125 meta = decode_array_metadata(meta_bytes)
126 self._meta = meta
127 self._shape = meta['shape']
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_array_metadata(s)
35 )
36 except Exception as e:
---> 37 raise MetadataError('error decoding metadata: %s' % e)
38 else:
39 return meta
MetadataError: error decoding metadata: too many values to unpack (expected 2)Conda environment:
Details
name: test
channels: !!python/tuple
- conda-forge
- defaults
dependencies:
- conda-forge::blas=1.1=openblas
- conda-forge::ca-certificates=2017.1.23=0
- conda-forge::certifi=2017.1.23=py35_0
- conda-forge::decorator=4.0.11=py35_0
- conda-forge::fasteners=0.14.1=py35_2
- conda-forge::ipython=5.2.2=py35_0
- conda-forge::ipython_genutils=0.1.0=py35_0
- conda-forge::monotonic=1.2=py35_1
- conda-forge::ncurses=5.9=10
- conda-forge::numpy=1.12.0=py35_blas_openblas_200
- conda-forge::openblas=0.2.19=0
- conda-forge::openssl=1.0.2h=3
- conda-forge::pexpect=4.2.1=py35_0
- conda-forge::pickleshare=0.7.3=py35_0
- conda-forge::pip=9.0.1=py35_0
- conda-forge::prompt_toolkit=1.0.13=py35_0
- conda-forge::ptyprocess=0.5.1=py35_0
- conda-forge::pygments=2.2.0=py35_0
- conda-forge::python=3.5.3=1
- conda-forge::readline=6.2=0
- conda-forge::setuptools=33.1.0=py35_0
- conda-forge::simplegeneric=0.8.1=py35_0
- conda-forge::six=1.10.0=py35_1
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.0=py35_0
- conda-forge::wcwidth=0.1.7=py35_0
- conda-forge::wheel=0.29.0=py35_0
- conda-forge::xz=5.2.2=0
- conda-forge::zarr=2.1.4=py35_0
- conda-forge::zlib=1.2.11=0
- libgfortran=3.0.0=1
- pip:
- ipython-genutils==0.1.0
- prompt-toolkit==1.0.13
prefix: /opt/conda/envs/test
Activity
alimanfoo commentedon Feb 10, 2017
jakirkham commentedon Feb 10, 2017
Ok, wasn't sure if this was an encoding issue with metadata only or if it would effect storage format as well. IIUC it is a metadata only issue (with some updated docs).
alimanfoo commentedon Feb 10, 2017
vellamike commentedon Jan 11, 2019
What is the ETA for the 2.3 release? I am hitting this issue and see that it has been resolved in 2.3
alimanfoo commentedon Jan 11, 2019
FWIW I'd like to get 2.3 out before the end of January, although I haven't said that out loud yet.
vellamike commentedon Jan 16, 2019
@alimanfoo that would be great.