Skip to content

Yaml serializer fails with Numpy 2.0 #15792

Closed

Description

@matteobachetti

Description

With the latest numpy dev, yaml often fails to deserialize numpy types.
I think it comes from a change in how Numpy is representing values, a problem which comes out often also in doctests. For example, in Stingray I had a lot of errors in doctests like:

091     >>> get_total_gti_length(gti)
Expected:
    1021
Got:
    np.int64(1021)

And apparently, also yaml serialization suffers from this issue. The MWE below should clarify the kind of errors I'm getting.

Expected behavior

I'd expect something like this to just work. Metadata from numpy types should be serialized and deserialized appropriately.

from astropy.table import Table
import numpy as np
t = Table(dict(b=[0,1]), meta=dict(a=np.float64(1)))
t.write("bu.ecsv")
Table.read("bu.ecsv")

How to Reproduce

MWE:

In [1]: from astropy.table import Table

In [2]: import numpy as np

In [3]: t = Table(dict(b=[0,1]), meta=dict(a=np.float64(1)))

In [4]: t.write("bu.ecsv")

In [5]: Table.read("bu.ecsv")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/astropy/table/meta.py:431, in get_header_from_yaml(lines)
    430 try:
--> 431     header = yaml.load(header_yaml, Loader=TableLoader)
    432 except Exception as err:

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/__init__.py:81, in load(stream, Loader)
     80 try:
---> 81     return loader.get_single_data()
     82 finally:

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:51, in BaseConstructor.get_single_data(self)
     50 if node is not None:
---> 51     return self.construct_document(node)
     52 return None

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:60, in BaseConstructor.construct_document(self, node)
     59 for generator in state_generators:
---> 60     for dummy in generator:
     61         pass

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:413, in SafeConstructor.construct_yaml_map(self, node)
    412 yield data
--> 413 value = self.construct_mapping(node)
    414 data.update(value)

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:218, in SafeConstructor.construct_mapping(self, node, deep)
    217     self.flatten_mapping(node)
--> 218 return super().construct_mapping(node, deep=deep)

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:143, in BaseConstructor.construct_mapping(self, node, deep)
    141     raise ConstructorError("while constructing a mapping", node.start_mark,
    142             "found unhashable key", key_node.start_mark)
--> 143 value = self.construct_object(value_node, deep=deep)
    144 mapping[key] = value

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:100, in BaseConstructor.construct_object(self, node, deep)
     99 if tag_suffix is None:
--> 100     data = constructor(self, node)
    101 else:

File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:292, in SafeConstructor.construct_yaml_float(self, node)
    291 else:
--> 292     return sign*float(value)

ValueError: could not convert string to float: 'np.float64(1.0)'

The above exception was the direct cause of the following exception:

YamlParseError                            Traceback (most recent call last)

(...)

InconsistentTableError: unable to parse yaml in meta header

Versions

macOS-14.2.1-arm64-arm-64bit
Python 3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ]
astropy 6.0.0
Numpy 2.0.0.dev0+git20231111.9340fca
pyerfa 2.0.1.1
Scipy 1.13.0.dev0+0.c4ce0c4
Matplotlib 3.8.2

Activity

  1. matteobachetti commented on Dec 27, 2023

    @matteobachetti
    ContributorAuthor

    @taldcroft I think so. Sorry for not seeing the PR before!

  2. taldcroft commented on Dec 27, 2023

    @taldcroft
    Member

    OK, I just updated #15065 to say it fixes this issue so it should get closed with that one. But it's worth you trying out #15065 to make sure it works for you!

  3. matteobachetti commented on Dec 27, 2023

    @matteobachetti
    ContributorAuthor

    @taldcroft I verified that adding the represent_float method to AstropyDumper, with the repr->str substitution, and using that representer instead of yaml's default, solved the issue, at least with floats.

  4. mhvk commented on Dec 27, 2023

    @mhvk
    Contributor

    @taldcroft, @matteobachetti - would it be worth it to pull this out of #15065? That PR is rather big as is, and perhaps it is best to separate out the corrections for failing doctests to code changes that actually matter - after all, the latter we have no choice in fixing! (For the doctests, there is an option to stick with pre-2.0 behaviour, which is what we are using until 2.0 is actually out).

  5. matteobachetti commented on Dec 28, 2023

    @matteobachetti
    ContributorAuthor

    @mhvk I totally agree this can be a matter of a smaller PR to be merged quickly. I wonder if it might also be worth suggesting the same kind of modification upstream to pyyaml?

  6. added this to the v6.1.1 milestone on May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions