Complex dtype error  #244

Activity

1. alimanfoo

   added

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on Mar 7, 2018
2. chairmank commented on Jun 7, 2018

   

   chairmank

   on Jun 7, 2018

   More actions

   To fix this problem, what do you think about evolving the specification

   https://zarr.readthedocs.io/en/stable/spec/v2.html#fill-value-encoding

   so that the fill value in the metadata JSON is a base64-encoded binary representation of a value with the specified data type? For a given NumPy-style type string (even for structs), we can unambiguously decode the binary representation. To distinguish it from legacy fill_value specifications, this binary representation could be packaged in a special JSON object. Examples are below.

   For backwards compatibility, continue to parse JSON numeric literals:

   {
       "dtype": "<f8",
       "fill_value": 0.9868011474609375,
    }
   

   For backwards compatibility, continue to recognize strings like "NaN" as special values:

   {
       "dtype": "<f8",
       "fill_value": "NaN",
    }
   

   New style binary representation of fill value. "AAAAAAAA+H8=" is the base64 encoding of '\x00\x00\x00\x00\x00\x00\xf8\x7f which is NaN as a little-endian float64.

   {
       "dtype": "<f8",
       "fill_value": {
         "data": "AAAAAAAA+H8="
       }
    }
   
3. chairmank commented on Jun 21, 2018

   

   chairmank

   on Jun 21, 2018

   More actions

   @jakirkham suggested a similar idea on #216
4. alimanfoo

   added this to the v2.3 milestone on Oct 19, 2018
5. jakirkham commented on Dec 10, 2018

   

   jakirkham

   on Dec 10, 2018

   Last edited by jakirkham

   Member

   More actions

   As a potentially quicker fix, we could do something similar to what we do with datetime/timedelta types and cast the complex values to something else (maybe a record with floats).

   zarr.array((1j*np.arange(10)).view(np.dtype([("real", np.float64), ("imag", np.float64)])))

   Edit: Should add this makes the fill value less human readable (it is effectively base64 encoded as that's how structured arrays are handled currently). Also there may need to be some explicit casting of the data afterwards to actually use it as complex. So while this is a viable workaround, it's probably less preferable to directly handling the encoding/decoding problem.
6. jakirkham commented on Dec 15, 2018

   

   jakirkham

   on Dec 15, 2018

   Member

   More actions

   After giving this some thought, figured a reasonable solution would be to just make the fill value for complex numbers a list with two floats (e.g. real and imaginary). This keeps the fill values human readable, is supported by JSON, and is pretty easy for us to handle (as it can leverage how we handle float value encoding/decoding currently).

   Went ahead and added PR ( #363 ) to demonstrate this approach for discussion. Though would be perfectly happy just moving ahead with that approach if it is agreeable.
7. jakirkham

   closed this as completedin #363on Feb 26, 2019