Repository navigation
Closed
Description
Description
With the latest numpy dev, yaml often fails to deserialize numpy types.
I think it comes from a change in how Numpy is representing values, a problem which comes out often also in doctests. For example, in Stingray I had a lot of errors in doctests like:
091 >>> get_total_gti_length(gti)
Expected:
1021
Got:
np.int64(1021)
And apparently, also yaml serialization suffers from this issue. The MWE below should clarify the kind of errors I'm getting.
Expected behavior
I'd expect something like this to just work. Metadata from numpy types should be serialized and deserialized appropriately.
from astropy.table import Table
import numpy as np
t = Table(dict(b=[0,1]), meta=dict(a=np.float64(1)))
t.write("bu.ecsv")
Table.read("bu.ecsv")How to Reproduce
MWE:
In [1]: from astropy.table import Table
In [2]: import numpy as np
In [3]: t = Table(dict(b=[0,1]), meta=dict(a=np.float64(1)))
In [4]: t.write("bu.ecsv")
In [5]: Table.read("bu.ecsv")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/astropy/table/meta.py:431, in get_header_from_yaml(lines)
430 try:
--> 431 header = yaml.load(header_yaml, Loader=TableLoader)
432 except Exception as err:
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/__init__.py:81, in load(stream, Loader)
80 try:
---> 81 return loader.get_single_data()
82 finally:
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:51, in BaseConstructor.get_single_data(self)
50 if node is not None:
---> 51 return self.construct_document(node)
52 return None
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:60, in BaseConstructor.construct_document(self, node)
59 for generator in state_generators:
---> 60 for dummy in generator:
61 pass
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:413, in SafeConstructor.construct_yaml_map(self, node)
412 yield data
--> 413 value = self.construct_mapping(node)
414 data.update(value)
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:218, in SafeConstructor.construct_mapping(self, node, deep)
217 self.flatten_mapping(node)
--> 218 return super().construct_mapping(node, deep=deep)
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:143, in BaseConstructor.construct_mapping(self, node, deep)
141 raise ConstructorError("while constructing a mapping", node.start_mark,
142 "found unhashable key", key_node.start_mark)
--> 143 value = self.construct_object(value_node, deep=deep)
144 mapping[key] = value
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:100, in BaseConstructor.construct_object(self, node, deep)
99 if tag_suffix is None:
--> 100 data = constructor(self, node)
101 else:
File ~/opt/anaconda3/envs/py311dev/lib/python3.11/site-packages/yaml/constructor.py:292, in SafeConstructor.construct_yaml_float(self, node)
291 else:
--> 292 return sign*float(value)
ValueError: could not convert string to float: 'np.float64(1.0)'
The above exception was the direct cause of the following exception:
YamlParseError Traceback (most recent call last)
(...)
InconsistentTableError: unable to parse yaml in meta headerVersions
macOS-14.2.1-arm64-arm-64bit
Python 3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ]
astropy 6.0.0
Numpy 2.0.0.dev0+git20231111.9340fca
pyerfa 2.0.1.1
Scipy 1.13.0.dev0+0.c4ce0c4
Matplotlib 3.8.2
Activity
taldcroft commentedon Dec 27, 2023
Is this fixed by #15065 (https://github.com/astropy/astropy/pull/15065/files#diff-1b1d70dbe87104145fb163be930c74e756358290bbee80127ced65a2b90c0555)?
matteobachetti commentedon Dec 27, 2023
@taldcroft I think so. Sorry for not seeing the PR before!
taldcroft commentedon Dec 27, 2023
OK, I just updated #15065 to say it fixes this issue so it should get closed with that one. But it's worth you trying out #15065 to make sure it works for you!
matteobachetti commentedon Dec 27, 2023
@taldcroft I verified that adding the
represent_floatmethod toAstropyDumper, with therepr->strsubstitution, and using that representer instead ofyaml's default, solved the issue, at least with floats.mhvk commentedon Dec 27, 2023
@taldcroft, @matteobachetti - would it be worth it to pull this out of #15065? That PR is rather big as is, and perhaps it is best to separate out the corrections for failing doctests to code changes that actually matter - after all, the latter we have no choice in fixing! (For the doctests, there is an option to stick with pre-2.0 behaviour, which is what we are using until 2.0 is actually out).
matteobachetti commentedon Dec 28, 2023
@mhvk I totally agree this can be a matter of a smaller PR to be merged quickly. I wonder if it might also be worth suggesting the same kind of modification upstream to
pyyaml?