Repository navigation
zarr read random indices support #77
Description
I wonder if zarr can read a list of random indices:
In [65]: a = np.arange(2000000).reshape(1000000, 2)
In [66]: a[[5,2,3]]
Out[66]:
array([[10, 11],
[ 4, 5],
[ 6, 7]])
In [67]: z = zarr.array(a, chunks=(50000, 2))
In [68]: z[[5,2,3]]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-68-007f030471a9> in <module>()
----> 1 z[[5,2,3]]
/Users/akravchenko/research/superpython/anaconda/lib/python3.5/site-packages/zarr/core.py in __getitem__(self, item)
446
447 # normalize selection
--> 448 selection = normalize_array_selection(item, self._shape)
449
450 # determine output array shape
/Users/akravchenko/research/superpython/anaconda/lib/python3.5/site-packages/zarr/util.py in normalize_array_selection(item, shape)
193
194 else:
--> 195 raise TypeError('expected indices or slice, found: %r' % item)
196
197
TypeError: expected indices or slice, found: [5, 2, 3]
In [69]: zarr.__version__, np.__version__
Out[69]: ('2.1.1', '1.11.1')Thank you,
Anton
Activity
ankravch commentedon Sep 23, 2016
ok, it does
alimanfoo commentedon Sep 23, 2016
Hi @ankravch,
Technically speaking Zarr does not support fancy indexing with either a
list of indices or a boolean array. If your data are not too big then you
can do as you showed, i.e., z[:][indices], however note that z[:] has the
effect of loading the entire array into memory as a numpy array, and so it
is numpy that is providing the fancy indexing.
If you want out-of-core fancy indexing then you can try dask.array. I.e.,
da.from_array(z, chunks=z.chunks)[indices].compute(optimize_graph=False)
should give the same result, but will not load all data into memory at the
same time. Note that the best way to do this via dask may change in a
future release, see also dask/dask#1528.
Hth,
Alistair
On Friday, September 23, 2016, ankravch [email protected] wrote:
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: [email protected]
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721