Skip to content

YamlParseError when reading mixin columns with Numpy 2.0 #16414

New issue
New issue
Closed
Closed
YamlParseError when reading mixin columns with Numpy 2.0#16414
Labels
🔥CriticalBugDuplicatenumpy-devtable
Milestone
v6.1.1

Description

@astrofrog
astrofrog
opened on May 8, 2024

Description

When reading a table with mixin columns and Numpy 2.0rc1 installed, I get a YamlParseError

Expected behavior

The table should load

How to Reproduce

from astropy.table import Table
from astropy.coordinates import SkyCoord
from astropy import units as u

t1 = Table()
t1['coords'] = SkyCoord([1, 2, 3] * u.deg, [4, 5, 6] * u.deg, frame='galactic')
t1.write('table.fits', format='fits', overwrite=True)
t2 = Table.read('table.fits')

gives:

Traceback (most recent call last):
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/table/meta.py", line 431, in get_header_from_yaml
    header = yaml.load(header_yaml, Loader=TableLoader)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/misc/yaml.py", line 86, in _serialized_column_constructor
    map = loader.construct_mapping(node)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/misc/yaml.py", line 176, in constructor
    map = loader.construct_mapping(node)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/yaml/constructor.py", line 292, in construct_yaml_float
    return sign*float(value)
                ^^^^^^^^^^^^
ValueError: could not convert string to float: 'np.float64(360.0)'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tom/tmp/test_io.py", line 8, in <module>
    t2 = Table.read('table.fits')
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/table/connect.py", line 62, in __call__
    out = self.registry.read(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/registry/core.py", line 221, in read
    data = reader(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/fits/connect.py", line 247, in read_table_fits
    return read_table_fits(
           ^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/fits/connect.py", line 349, in read_table_fits
    t = _decode_mixins(t)
        ^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/io/fits/connect.py", line 103, in _decode_mixins
    info = meta.get_header_from_yaml(lines)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tom/python/dev/lib/python3.11/site-packages/astropy/table/meta.py", line 433, in get_header_from_yaml
    raise YamlParseError() from err
astropy.table.meta.YamlParseError

Versions

macOS-14.1.1-arm64-arm-64bit
Python 3.11.2 (main, Mar 23 2023, 23:28:33) [Clang 14.0.0 (clang-1400.0.29.202)]
astropy 7.0.0.dev14+gaae4425033.d20240411
Numpy 2.0.0rc1
pyerfa 2.0.1.4

Activity

  1. astrofrog
    added
    Bug
    on May 8, 2024

  2. astrofrog commented on May 8, 2024

    @astrofrog
    astrofrog
    on May 8, 2024
    MemberAuthor

    Looks like an issue during serialization probably? We should definitely fix this before Numpy 2.0 final is out as otherwise people will start writing invalid files.

  3. neutrinoceros commented on May 8, 2024

    @neutrinoceros
    neutrinoceros
    on May 8, 2024
    Contributor

    Data must be converted to string via __repr__ at some point during serialization. Off hand, it's not clear wether the bug is in astropy or in pyyaml. I can take a look today.

  4. pllim
    added
    table
    on May 8, 2024

  5. pllim commented on May 8, 2024

    @pllim
    pllim
    on May 8, 2024
    Member

    FWIW same problem in QTable. I think it comes from how numpy 2 is printing out floats now.

    File yaml/constructor.py:292, in SafeConstructor.construct_yaml_float(self, node)
    291 else:
--> 292     return sign*float(value)

ValueError: could not convert string to float: 'np.float64(360.0)'

The above exception was the direct cause of the following exception:

YamlParseError                            Traceback (most recent call last)
  6. pllim
    added
    numpy-dev
    on May 8, 2024
  7. pllim
    added this to the v6.1.1 milestone on May 8, 2024
  8. pllim
    added
    🔥Critical
    on May 8, 2024

  9. pllim commented on May 8, 2024

    @pllim
    pllim
    on May 8, 2024
    Member

    Since we purposely released v6.1 early for numpy 2 compat, I will make this a blocker for v6.1.1 (hence the milestone), and slap a 🔥 label on it. I wonder why our CI didn't catch this, so probably also need a new regression test.

  10. neutrinoceros commented on May 8, 2024

    @neutrinoceros
    neutrinoceros
    on May 8, 2024
    Contributor

    I'm almost sure the problem happens at

    astropy/astropy/table/meta.py

    Lines 374 to 376 in 7a30e35

    lines = yaml.dump(
    header, default_flow_style=None, Dumper=TableDumper, width=130
    ).splitlines()

    where a seemingly innocuous header

    {'datatype': [{'datatype': 'float64', 'name': 'coords.l', 'unit': 'deg'},
              {'datatype': 'float64', 'name': 'coords.b', 'unit': 'deg'}],
 'meta': OrderedDict([('__serialized_columns__',
                       {'coords': {'__class__': 'astropy.coordinates.sky_coordinate.SkyCoord',
                                   'b': {'__class__': 'astropy.coordinates.angles.core.Latitude',
                                         'unit': Unit("deg"),
                                         'value': {'name': 'coords.b'}},
                                   'frame': 'galactic',
                                   'l': {'__class__': 'astropy.coordinates.angles.core.Longitude',
                                         'unit': Unit("deg"),
                                         'value': {'name': 'coords.l'},
                                         'wrap_angle': <Angle 360. deg>},
                                   'representation_type': 'spherical'}})])}

    gets dumped as

    ['datatype:',
 '- {name: coords.l, unit: deg, datatype: float64}',
 '- {name: coords.b, unit: deg, datatype: float64}',
 'meta: !!omap',
 '- __serialized_columns__:',
 '    coords:',
 '      __class__: astropy.coordinates.sky_coordinate.SkyCoord',
 '      b: !astropy.table.SerializedColumn',
 '        __class__: astropy.coordinates.angles.core.Latitude',
 '        unit: &id001 !astropy.units.Unit {unit: deg}',
 '        value: !astropy.table.SerializedColumn {name: coords.b}',
 '      frame: galactic',
 '      l: !astropy.table.SerializedColumn',
 '        __class__: astropy.coordinates.angles.core.Longitude',
 '        unit: *id001',
 '        value: !astropy.table.SerializedColumn {name: coords.l}',
 '        wrap_angle: !astropy.coordinates.Angle',
 '          unit: *id001',
 "          value: !!float 'np.float64(360.0)'",
 '      representation_type: spherical']

    (note the 'np.float64(360.0)' on the second to last line)
    I assume the fix would be to add a custom representer to TableDumper to handle numpy scalars, but I've already spent much longer than I expected chasing this bug, so this might be despair talking.

  11. pllim commented on May 8, 2024

    @pllim
    pllim
    on May 8, 2024
    Member

    cc @taldcroft

  12. taldcroft commented on May 8, 2024

    @taldcroft
    taldcroft
    on May 8, 2024
    Member

    FWIW I think this is a duplicate of #15792, which should be fixed by #15065. That one is marked that it shouldn't be merged prior to numpy 2.0 release but I'm not sure why that was the case. Let's take the discussion over to #15065.

  13. pllim
    added
    Duplicate
    on May 8, 2024
  14. pllim
    closed this as completedon May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    🔥CriticalBugDuplicatenumpy-devtable

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions