Structured Arrays #110

Activity

1. alimanfoo commented on Feb 10, 2017

   

   alimanfoo

   on Feb 10, 2017

   Member

   More actions

   Zarr does support storing structured arrays, e.g.:

   In [9]: import numpy as np
   
   In [10]: import zarr
   
   In [11]: a = np.array([(b'a', 1), (b'b', 2)], dtype=[('foo', 'S1'), ('bar', int)])
   
   In [12]: z = zarr.array(a)
   
   In [13]: z
   Out[13]: 
   Array((2,), [('foo', 'S1'), ('bar', '<i8')], chunks=(2,), order=C)
     nbytes: 18; nbytes_stored: 438; ratio: 0.0; initialized: 1/1
     compressor: Blosc(cname='lz4', clevel=5, shuffle=1)
     store: dict
   

   However, one nice feature that h5py has which zarr doesn't currently have is the ability to load a specific field or fields, e.g., z[:, 'foo']. I have actually wanted this recently.

   FWIW regarding the choice of single array with structured dtype versus one array with simple dtype per column, I've generally found the latter (i.e., columnar) storage to be more flexible and more efficient for a variety of uses. However I still do use structured arrays too and so would be interested to support both patterns.
2. jakirkham commented on Feb 10, 2017

   

   jakirkham

   on Feb 10, 2017

   Last edited by jakirkham

   MemberAuthor

   More actions

   Oh, that's good to know. Then maybe I'm running into a bug or an unsupported edge case. ( https://github.com/alimanfoo/zarr/issues/111 ) Will close this out.

   However, one nice feature that h5py has which zarr doesn't currently have is the ability to load a specific field or fields, e.g., z[:, 'foo']. I have actually wanted this recently.

   Indeed, that would be a very nice feature for Zarr. Used this with h5py some myself.

   Edit: Opened issue ( https://github.com/alimanfoo/zarr/issues/112 ) on this this point.

   ...I've generally found the latter (i.e., columnar) storage to be more flexible and more efficient for a variety of uses. However I still do use structured arrays too and so would be interested to support both patterns.

   By columnar, I'm assuming you mean having a single type for the array. In which case, I do agree with you. Still sometimes a structured array is just the right data structure for the problem.
3. jakirkham

   closed this as completedon Feb 10, 2017
4. jhamman

   added a commit that references this issue on Apr 20, 2024

   feature(store): make list_* methods async generators (zarr-developers…

   ...

   49f1505
5. d-v-b

   added a commit that references this issue on May 15, 2024

   groundwork for V3 group tests (#1743)

   ...

   f3305d9