Use JSON instead of pickle for array metadata #6

Activity

1. alimanfoo commented on Dec 22, 2015

   

   alimanfoo

   on Dec 22, 2015

   Member

   More actions

   Good point. Very happy to consider switching to JSON. See also comments in #5.
2. alimanfoo

   changed the title [-]Use JSON instead of pickle for array metadata?[/-] [+]Use JSON instead of pickle for array metadata[/+] on Dec 22, 2015
3. alimanfoo

   modified the milestone: v0.4 on Apr 11, 2016
4. alimanfoo commented on Apr 13, 2016

   

   alimanfoo

   on Apr 13, 2016

   Member

   More actions

   I'm looking into this but hitting an issue regarding serialization of the array's numpy dtype. I need to save the numpy dtype as one of the fields in the array metadata, however I'm struggling to find a way to convert a dtype to/from a string that supports simple and structured dtypes and preserves endianness.

   I looked at bcolz and that uses str(), effectively:

   dt = ...  # some numpy dtype
   s = str(dt)  # convert dtype to string
   dt = np.dtype(s)  # convert string to dtype

   However this fails on a structured dtype. Also it doesn't preserve endianness for simple dtypes in some cases.

   Some other folks use the .str attribute on the dtype, which does preserve endianness but collapses structured dtypes down to '|V...' so losing the internal dtype structure.

   There is also the .descr attribute which has all the dtype information, but then it does slightly weird things with simple dtypes, e.g.::

   In [74]: d = np.dtype('f4')
   
   In [75]: d.descr
   Out[75]: [('', '<f4')]
   
   In [76]: np.dtype(d.descr)
   Out[76]: dtype([('f0', '<f4')])

   Any help appreciated, cc @shoyer @mrocklin @FrancescAlted.
5. FrancescAlted commented on Apr 13, 2016

   

   FrancescAlted

   on Apr 13, 2016

   More actions

   Well, str() still keeps track of the endianness, although in a strange form. Look at this:

   In []: d3 = np.dtype("f4")
   
   In []: str(d3)
   Out[]: 'float32'
   
   In []: d3 = np.dtype(">f4")
   
   In []: str(d3)
   Out[]: '>f4'
   

   so, it only 'looses' the endianness indicator only when it is the same than current platform (yeah, funny). Anyway, here it is a quick portable solution (I think):

   In []: d = np.dtype('f4')
   
   In []: st = np.dtype("f4,i8")
   
   In []: mydtype = lambda t: t.descr if t.descr[0][0] else t.descr[0][1]
   
   In []: mydtype(d)
   Out[]: '<f4'
   
   In []: mydtype(st)
   Out[]: [('f0', '<f4'), ('f1', '<i8')]
   

   Hope this helps.

   👍React with 👍1Reacted by jakirkham
6. alimanfoo commented on Apr 13, 2016

   

   alimanfoo

   on Apr 13, 2016

   Member

   More actions

   Thanks @FrancescAlted, very helpful. I have a working solution in #14 which does something similar but feels hacky. The essence of it is these two functions:

   def encode_dtype(d):
       if d.fields is None:
           return d.str
       else:
           return str(d)
   
   
   def decode_dtype(s):
       try:
           return np.dtype(s)
       except ValueError:
           return np.dtype(ast.literal_eval(s))

   These functions allow a numpy dtype to be encoded as a string for writing as a single string value to the JSON file, then also decodes that string back to a dtype object.

   Any comments very welcome.
7. FrancescAlted commented on Apr 13, 2016

   

   FrancescAlted

   on Apr 13, 2016

   More actions

   Whatever works for you ;) I did not know about ast.literal_eval(). Interesting.
8. shoyer commented on Apr 13, 2016

   

   shoyer

   on Apr 13, 2016

   ContributorAuthor

   More actions

   Indeed, I also did not realize ast.literal_eval() was a thing :). But given that you're using JSON there is something to be said for making the dtype machine readable. I would probably opt for either nested lists [['f0', '<f4'], ['f1', '<i8']] or dict of lists {'names': ['f0', 'f1'], 'formats': ['<f4', '<i8']}. Both are obviously based on NumPy but would make writing an interface in another language more straightforward.

   👍React with 👍1Reacted by jakirkham
9. alimanfoo commented on Apr 13, 2016

   

   alimanfoo

   on Apr 13, 2016

   Member

   More actions

   Thanks @shoyer, I've changed the implementation to use nested lists for structured dtypes. I'll merge tomorrow if no further comments, then work on #5.
10. alimanfoo commented on Apr 14, 2016

    

    alimanfoo

    on Apr 14, 2016

    Member

    More actions

    Closed via #14.
11. alimanfoo

    closed this as completedon Apr 14, 2016
12. maxrjones

    added a commit that references this issue on Feb 5, 2025

    Merge pull request #6 from kylebarron/kyle/fix-transform-list-dir

    ...

    72bf9f4