Repository navigation
TypeError on write when stacking an empty table with a table that has a Time column #16370
Description
Description
I need to vstack an arbitrary number of tables, that could be 1 or more. I create an empty table and then vstack to it all the tables I have read (astropy tables stored as ecsv) and then write back a merged table to an ecsv file. This was working fine with astropy 5.X, however crashes now with the following error:
TypeError: Iterator operand 1 dtype could not be cast from dtype([('year', '?'), ('month', '?'), ('day', '?'), ('hour', '?'), ('minute', '?'), ('second', '?')]) to dtype('bool') according to the rule 'unsafe'
However, if I vstack two non-empty two non-empty tables, the writing still works.
Expected behavior
The code shall not crash and the Time type shall be converted (decomposed) to ecsv schema properly.
How to Reproduce
Install astropy-6.X, then the following code snippet can be used to reproduce:
>>> from astropy.table import Table, vstack
>>> test_data = Table.read('/tmp/meteo_data/copernicus_0.ecsv')
>>> test_data.info
<Table length=1036>
name dtype unit class
----------------------- ------- ------------- ------
Latitude float64 deg Column
Longitude float64 deg Column
Timestamp object Time
Pressure float64 hPa Column
Geopotential float64 m2 / s2 Column
Ozone mass mixing ratio float64 Column
Potential vorticity float64 m2 K / (kg s) Column
Relative humidity float64 % Column
Temperature float64 K Column
U component of wind float64 m / s Column
V component of wind float64 m / s Column
Vertical velocity float64 Pa / s Column
Divergence float64 1 / s Column
Altitude float64 m Column
Density float64 1 / cm3 Column
Exponential Density float64 Column
Wind Speed float64 m / s Column
Wind Direction float64 rad Column
>>> test_data.write('/tmp/meteo_data/test_data_write.ecsv')
>>> empty_table = Table()
>>> merged_table = vstack([empty_table, test_data])
>>> merged_table.info
<Table length=1036>
name dtype unit class
----------------------- ------- ------------- ------------
Latitude float64 deg MaskedColumn
Longitude float64 deg MaskedColumn
Timestamp object Time
Pressure float64 hPa MaskedColumn
Geopotential float64 m2 / s2 MaskedColumn
Ozone mass mixing ratio float64 MaskedColumn
Potential vorticity float64 m2 K / (kg s) MaskedColumn
Relative humidity float64 % MaskedColumn
Temperature float64 K MaskedColumn
U component of wind float64 m / s MaskedColumn
V component of wind float64 m / s MaskedColumn
Vertical velocity float64 Pa / s MaskedColumn
Divergence float64 1 / s MaskedColumn
Altitude float64 m MaskedColumn
Density float64 1 / cm3 MaskedColumn
Exponential Density float64 MaskedColumn
Wind Speed float64 m / s MaskedColumn
Wind Direction float64 rad MaskedColumn
>>> merged_table.write('/tmp/meteo_data/test_merged_data_write.ecsv')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/table/connect.py", line 130, in __call__
self.registry.write(instance, *args, **kwargs)
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/io/registry/core.py", line 386, in write
return writer(data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/io/ascii/connect.py", line 28, in io_write
return write(table, filename, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/utils/decorators.py", line 604, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/io/ascii/ui.py", line 1036, in write
lines = writer.write(table)
^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/io/ascii/core.py", line 1562, in write
table = self.update_table_data(table)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/io/ascii/ecsv.py", line 518, in update_table_data
out = serialize.represent_mixins_as_columns(table)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/table/serialize.py", line 295, in represent_mixins_as_columns
_represent_mixin_as_column(
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/table/serialize.py", line 209, in _represent_mixin_as_column
_represent_mixin_as_column(data, new_name, new_cols, obj_attrs)
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/astropy/table/serialize.py", line 196, in _represent_mixin_as_column
if (hasattr(data, "mask") and np.any(data.mask))
^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/numpy/core/fromnumeric.py", line 2412, in any
return _wrapreduction(a, np.logical_or, 'any', axis, None, out,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/mambaforge/envs/calibpipe/lib/python3.12/site-packages/numpy/core/fromnumeric.py", line 88, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Iterator operand 1 dtype could not be cast from dtype([('year', '?'), ('month', '?'), ('day', '?'), ('hour', '?'), ('minute', '?'), ('second', '?')]) to dtype('bool') according to the rule 'unsafe'
>>> another_test_data_instance = Table.read('/tmp/meteo_data/copernicus_0.ecsv')
>>> merged_table_with_non_empty = vstack([another_test_data_instance, test_data])
>>> merged_table_with_non_empty.info
<Table length=2072>
name dtype unit class
----------------------- ------- ------------- ------
Latitude float64 deg Column
Longitude float64 deg Column
Timestamp object Time
Pressure float64 hPa Column
Geopotential float64 m2 / s2 Column
Ozone mass mixing ratio float64 Column
Potential vorticity float64 m2 K / (kg s) Column
Relative humidity float64 % Column
Temperature float64 K Column
U component of wind float64 m / s Column
V component of wind float64 m / s Column
Vertical velocity float64 Pa / s Column
Divergence float64 1 / s Column
Altitude float64 m Column
Density float64 1 / cm3 Column
Exponential Density float64 Column
Wind Speed float64 m / s Column
Wind Direction float64 rad Column
>>> merged_table_with_non_empty.write('/tmp/meteo_data/merged_table_with_non_empty_write.ecsv')An example ecsv table is attached (just rename txt to ecsv, GitHub does not allow ecsv attachments).
copernicus_0.txt
Versions
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import astropy; print("astropy", astropy.__version__)
import numpy; print("Numpy", numpy.__version__)
import erfa; print("pyerfa", erfa.__version__)
try:
import scipy
print("Scipy", scipy.__version__)
except ImportError:
print("Scipy not installed")
try:
import matplotlib
print("Matplotlib", matplotlib.__version__)
except ImportError:
print("Matplotlib not installed")macOS-13.5.1-x86_64-i386-64bit
Python 3.12.3 | packaged by conda-forge | (main, Apr 15 2024, 18:50:49) [Clang 16.0.6 ]
astropy 6.0.1
Numpy 1.26.4
pyerfa 2.0.1.4
Scipy 1.13.0
Matplotlib 3.8.4
Also, the same error is reproduced in a number of platforms in CI. See here for the reference:
https://gitlab.cta-observatory.org/cta-computing/dpps/calibrationpipeline/calibpipe/-/jobs/183758
Activity
github-actions commentedon May 2, 2024
Welcome to Astropy 👋 and thank you for your first issue!
A project member will respond to you as soon as possible; in the meantime, please double-check the guidelines for submitting issues and make sure you've provided the requested details.
GitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the Astropy Discourse user forum and close this issue.
If you feel that this issue has not been responded to in a timely manner, please send a message directly to the development mailing list. If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail [email protected].
neutrinoceros commentedon May 2, 2024
Thanks for reporting this @mexanick !
The the regression bisects down to a9b2e42, from #15231
Since that was a fairly heavy PR, let's ping @mhvk first in case the problem is obvious to him.
mhvk commentedon May 3, 2024
@mexanick - thanks for reporting, and @neutrinoceros thanks for the bisection. I guess the good news is that the particular commit did not touch table, so hopefully it will be a relatively simple change. Looking a bit closer, it is the presence of a masked
Time(which happens when merging with an empty table) with format "ydmhms" that is the problem:In the end, the problem is doing
np.any(data.mask)where that mask is structured.I think I can have a fix relatively quickly...
mhvk commentedon May 3, 2024
Took longer than I had hoped, but should be fixed by #16380...