Repository navigation
Description
Description
Reading the following ECSV table
# %ECSV 0.9
# ---
# datatype:
# - {name: source, datatype: object}
# - {name: obstime, datatype: string}
# - {name: ra, datatype: float64}
# - {name: dec, datatype: float64}
# - {name: pa_v3, datatype: float64}
# schema: astropy-2.0
source obstime ra dec pa_v3
None 2016-01-18T15:25:23.776 133.3599835511599 -14.886980131752205 -161.3183607231063
which was written out by a recent (and current) release version of astropy now fails when read via the current astropy dev:
In [1]: import astropy
In [2]: print(astropy.__version__)
4.3.dev1662+gdf850cb7c
In [4]: from astropy.table import Table
In [5]: table = Table.read("jwst/lib/tests/data/v1_calc_truth.ecsv")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-2e4749b2facf> in <module>
----> 1 table = Table.read("jwst/lib/tests/data/v1_calc_truth.ecsv")
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
59 descriptions = kwargs.pop('descriptions', None)
60
---> 61 out = registry.read(cls, *args, **kwargs)
62
63 # For some readers (e.g., ascii.ecsv), the returned `out` class is not
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/io/registry.py in read(cls, format, cache, *args, **kwargs)
525
526 reader = get_reader(format, cls)
--> 527 data = reader(*args, **kwargs)
528
529 if not isinstance(data, cls):
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/io/ascii/connect.py in io_read(format, filename, **kwargs)
16 format = re.sub(r'^ascii\.', '', format)
17 kwargs['format'] = format
---> 18 return read(filename, **kwargs)
19
20
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
367 else:
368 reader = get_reader(**new_kwargs)
--> 369 dat = reader.read(table)
370 _read_trace.append({'kwargs': copy.deepcopy(new_kwargs),
371 'Reader': reader.__class__,
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/io/ascii/core.py in read(self, table)
1336
1337 # Get the table column definitions
-> 1338 self.header.get_cols(self.lines)
1339
1340 # Make sure columns are valid
~/miniconda3/envs/dev/lib/python3.9/site-packages/astropy/io/ascii/ecsv.py in get_cols(self, lines)
184 col.dtype = header_cols[col.name]['datatype']
185 if col.dtype not in ECSV_DATATYPES:
--> 186 raise ValueError(f'datatype {col.dtype!r} of column {col.name!r} '
187 f'is not in allowed values {ECSV_DATATYPES}')
188
ValueError: datatype 'object' of column 'source' is not in allowed values ('bool', 'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'float16', 'float32', 'float64', 'float128', 'string')
Expected behavior
Under the release version (astropy 4.2.1) it reads in fine without issue.
This looks like an issue where numpy object types can no longer be serialized, and of course None is an object for numpy.
A workaround fix is to modify by hand the ECSV file header, changing the column dtype from object to string, and all works fine. But this is a backwards incompatibility in the ECSV reader, and will mean that many (some? few? 😂 ) tables written out by the current release of astropy will no longer be able to be read in in version 4.3.
Perhaps casting to string and raising a big DeprecationWarning might be better? Or something else? Clearly the general feature of serializing python objects is not possible in ECSV format, but clearly it was allowed before (and currently) in release versions.
The break seems to have happened in e807dbf (EDIT: #11569)
System Details
>>> import platform; print(platform.platform())
macOS-10.15.7-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.9.5 (default, May 18 2021, 12:31:01)
[Clang 10.0.0 ]
>>> import numpy; print("Numpy", numpy.__version__)
Numpy 1.20.3
>>> import astropy; print("astropy", astropy.__version__)
astropy 4.3.dev1662+gdf850cb7c
>>> import scipy; print("Scipy", scipy.__version__)
Scipy 1.6.3
>>> import matplotlib; print("Matplotlib", matplotlib.__version__)
Matplotlib 3.4.2
Activity
pllim commentedon Jun 1, 2021
Academic curiosity: How was this data created? Can you recreate it with astropy dev and see if the output file is actually different now?
taldcroft commentedon Jun 1, 2021
@jdavies-st - Indeed this is an API change of sorts, and it may warrant more explicit mention in the What's New.
The reason I say "of sorts" is that the previous behavior of successfully outputting an ECSV file for object-type columns was undocumented and could be considered a bug. The ECSV 0.9 standard did not include
objectas an allowed type, so astropy <= 4.2 was creating an invalid ECSV file.Another issue is that the previous astropy behavior was also wrong for reading. Note that while the original table had an
objectcolumn withNone, this gets read back as an object column with the raw strings corresponding to thereprof those objects, i.e. the string"None".Given those considerations, my preference at this point is to accept the API change as is. It should be relatively straightforward to modify your file or even make a little function to fix the header. Something like:
github-actions commentedon Jun 1, 2021
This issue was labeled as Close?. Remove Close? label or this will be closed after 7 days. (This is currently a dry-run.)
pllim commentedon Jun 1, 2021
OPs... heh heh... ignore the bot for now.
[-]Table ECSV reader barfs on dtype=object[/-][+]ECSV reader fails on datatype=object column created with astropy < 4.3[/+]jdavies-st commentedon Jun 2, 2021
Thanks @taldcroft. I agree. I just wanted to bring this up in case this was inadvertent or in case there was more documentation or guardrails needed. I actually think the current error message is quite good and allowed me to diagnose the problem immediately. Feel free to update what you think is needed in docs (or not) and close this.
taldcroft commentedon Jun 2, 2021
OK, closing.
privong commentedon Aug 16, 2021
Hi all, I just wanted to chime in on this. I encountered this issue as well, but nothing jumped out at me from checking the astropy release notes. So I'd advocate for making this issue more visible to users.
As a complement to the workaround code provided by @taldcroft, this shell command worked for me to fix incorrectly written ecsv's (tested on MacOS):