Support structured quantity, including scalars. - #12509

Can you add a comment about what the deep=True is doing? I see the comment below about deep but it wasn't enough to make me understand.

https://stackoverflow.com/questions/43812020/what-does-deep-true-do-in-pyyaml-loader-construct-mapping

I had the same question.

Without deep=True, the list produced by the structured dtype is not interpreted. It took me a while to figure out why I got dtype=[], so I should indeed have added a comment!

Actually, might as well ask: my first attempt kept just str(dtype) in the dumper, which means a structured dtype looks something like:

import numpy as np
dt = np.dtype('f8,f8')
str(dt)
# "[('f0', '<f8'), ('f1', '<f8')]"

But to interpret this in the loader, one would have to eval the string, which seemed less good than just letting yaml take care of the lists and tuples, even though that is substantially longer,

from astropy.io.misc.yaml import dump

print(dump(dt.descr))
- !!python/tuple [f0, <f8]
- !!python/tuple [f1, <f8]

If you think keeping just the string is better (and ideally have a better suggestion that eval!), I can do that too.

Can ast's safe eval work here?

Even though the ast.literal_eval is totally safe, I still prefer representing the dtype with "native" YAML lists and tuples.

ast.literal_eval works indeed:

np.dtype(ast.literal_eval(str(np.dtype('f8, f8'))))
dtype([('f0', '<f8'), ('f1', '<f8')])

and implementing it would mean adding the following to the constructor:

    if map['dtype'].startswith('['):
        map['dtype'] = ast.literal_eval(map['dtype'])

So, it seems it would be mostly a matter of preference:

print(dump([(1., 2.)] * u.Unit('m, m/s')))
# this PR
!astropy.units.Quantity
unit: !astropy.units.Unit {unit: '(m, m / s)'}
value: !numpy.ndarray
  buffer: !!binary |
    QUFBQUFBQUE4RDhBQUFBQUFBQUFRQT09
  dtype:
  - !!python/tuple [f0, <f8]
  - !!python/tuple [f1, <f8]
  order: C
  shape: !!python/tuple [1]

# with str(dtype)
!astropy.units.Quantity
unit: !astropy.units.Unit {unit: '(m, m / s)'}
value: !numpy.ndarray
  buffer: !!binary |
    QUFBQUFBQUE4RDhBQUFBQUFBQUFRQT09
  dtype: '[(''f0'', ''<f8''), (''f1'', ''<f8'')]'
  order: C
  shape: !!python/tuple [1]

The literal_eval has the advantage of keeping the strings shorter and being closer to what one would see by just looking at the dtype, but what I have here is nicer in that one can just put the contents in np.dtype directly. I end up being torn.

But since you both seem to prefer the yaml list option, I just made some updates to that, so this should then be ready for final review. Obviously, if you change your mind, I'm happy to change.

p.s. Yet another option is to register the dtype with the dumper/loader, but the yaml then becomes even longer... Indeed, just for fun, if one just let's yaml.dump itself do it, one gets the following rather hopeless output:

In [6]: print(yaml.dump(np.dtype('f8,f8')))
!!python/object/apply:numpy.dtype
args:
- V16
- false
- true
state: !!python/tuple
- 3
- '|'
- null
- !!python/tuple
  - f0
  - f1
- f0: !!python/tuple
  - &id001 !!python/object/apply:numpy.dtype
    args:
    - f8
    - false
    - true
    state: !!python/tuple
    - 3
    - <
    - null
    - null
    - null
    - -1
    - -1
    - 0
  - 0
  f1: !!python/tuple
  - *id001
  - 8
- 16
- 1
- 16

Agreed that registering the dtype is a mess!
Right now either the ast.literal_eval or the YAML look fine.
dtype can store metadata in a dictionary, for which the YAML option might be more suited...

@mhvk. Is this comment resolved and/or the PR ready for a review?

@nstarman - in my last comment I just thought I'd give you and @taldcroft a last look at how the two options of serializing the dtype look in practice. I think you both preferred to stick to native yaml as much as possible, so that is the version currently there. If after having a look, that is still the preference, then yes, this should be good to go. If not, and we go with literal_eval after all, then it is a very quick extra commit to change it to that.