ascii.write fails with nondescript error on structured (multi-D) columns #11148

Activity

1. dhomeier

   added

   table

   io.ascii

   Bug

   Effort-low

   on Dec 8, 2020
2. taldcroft commented on Dec 8, 2020

   

   taldcroft

   on Dec 8, 2020

   Member

   More actions

   I have thought about adding support in ECSV for N-d columns. In theory one can make column names like R.0 or R.1.2. This would be the right format for such support since there can be appropriate metadata in the file to allow round-trip.

   Note that ECSV does already detect N-d columns and give a good message, so this can be a template for other readers:

   ValueError: ECSV format does not support multidimensional column 'd'
   One can filter out such columns using:
   names = [name for name in tbl.colnames if len(tbl[name].shape) <= 1]
   tbl[names].write(...)
   

   👍React with 👍1Reacted by Derek Homeier
3. taldcroft commented on Dec 8, 2020

   

   taldcroft

   on Dec 8, 2020

   Member

   More actions

   The code to actually do this is probably not that difficult using the serialization protocols in place for mixin columns. Instead of a SkyCoord that writes out columns like sc.ra and sc.dec, you have Column know that it needs to break down into component columns for each sub-dimension.
4. dhomeier commented on Dec 8, 2020

   

   dhomeier

   on Dec 8, 2020

   ContributorAuthor

   More actions

   names = [name for name in tbl.colnames if len(tbl[name].shape) <= 1]
   tbl[names].write(...)

   My suggested workaround to the OP could probably be generalised (for now for ndim=2) to

   for n in names:
       ncol = tbl[n].dtype[1])
       newnames = [[f'{n}.{i}' for i in range(ncol)]
       indices = t.index_column(n)+np.zeros(ncol), dtype='i')
       t.add_columns(t[n].T, indexes=indices, names=newnames)
       t.remove_column(n)
   

   – which one could at least try and see if any naming conflicts arise, but having the writer handle this directly at the column level is of course much better.
5. taldcroft commented on Dec 9, 2020

   

   taldcroft

   on Dec 9, 2020

   Member

   More actions

   @dhomeier - good idea to provide a workaround. Here is a generalized version that makes a new table:

   import itertools
   
   def flatten_nd_columns(tbl):
       """Return a new table with any multidimensional columns flattened to a set
       of 1-d columns.
   
       Example
       -------
   
         >>> t = Table()
         >>> t['a'] = np.arange(12).reshape(2, 2, 3)
         >>> t['b'] = ['a', 'b']
         >>> flatten_nd_columns(t)
         <Table length=2>
         a.0_0 a.0_1 a.0_2 a.1_0 a.1_1 a.1_2  b
         int64 int64 int64 int64 int64 int64 str1
         ----- ----- ----- ----- ----- ----- ----
             0     1     2     3     4     5    a
             6     7     8     9    10    11    b
       """
       out = {}
       for col in tbl.itercols():
           if col.ndim == 1:
               out[col.info.name] = col
           else:
               ranges = [range(ii) for ii in col.shape[1:]]
               for dims in itertools.product(*ranges):
                   name = col.info.name + '.' + '_'.join(str(ii) for ii in dims)
                   out[name] = col[(...,) + dims]
       return tbl.__class__(out)
   

   👍React with 👍1Reacted by Prashin Jethwa
6. dhomeier commented on Dec 11, 2020

   

   dhomeier

   on Dec 11, 2020

   ContributorAuthor

   More actions

   Thanks for the worked-out solution @taldcroft! I had been thinking about adding this as a convenience function, but returning a new table is probably a better option than making potentially non-reversible modifications to the existing one.
   Looking for a good place to implement this, I noticed that to_pandas also has a similar check in

   astropy/astropy/table/table.py

   Line 3411 in 2388de9

   badcols = [name for name, col in self.columns.items() if len(col.shape) > 1]

   
   so perhaps this could be added as a Table method?
7. taldcroft commented on Dec 11, 2020

   

   taldcroft

   on Dec 11, 2020

   Member

   More actions

   I was thinking about a new method also. But in fact what we probably want is to also flatten any mixin columns down to plain columns using represent_mixins_as_columns. Or at least have that option.
8. astrofrog

   added

   needs-reconfirmation

   on Oct 21, 2025
9. astrofrog commented on Jun 3, 2026

   

   astrofrog

   on Jun 3, 2026

   Member

   More actions

   Ok so the original 'bug' is fixed here in the sense there is now a better error message:

   In [6]: import numpy as np
   
   In [7]: from astropy.table import Table, Column
   
   In [8]:  t = Table([Column(np.ones(3, dtype=('f', (2,))), name='R')])
   
   In [9]: t
   Out[9]: 
   <Table length=3>
       R     
   float32[2]
   ----------
   1.0 .. 1.0
   1.0 .. 1.0
   1.0 .. 1.0
   
   In [10]: t.write('test.txt', format='ascii.basic')
   ---------------------------------------------------------------------------
   ValueError                                Traceback (most recent call last)
   Cell In[10], line 1
   ----> 1 t.write('test.txt', format='ascii.basic')
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/table/connect.py:146, in TableWrite.__call__(self, serialize_method, write_indices, *args, **kwargs)
       144 if write_indices and tbl.indices:
       145     tbl = represent_indices(tbl)
   --> 146 self.registry.write(tbl, *args, **kwargs)
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/registry/core.py:395, in UnifiedOutputRegistry.write(self, data, format, *args, **kwargs)
       390     format = self._get_valid_format(
       391         "write", data.__class__, path, fileobj, args, kwargs
       392     )
       394 writer = self.get_writer(format, data.__class__)
   --> 395 return writer(data, *args, **kwargs)
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/ascii/connect.py:28, in io_write(format, table, filename, **kwargs)
        26     format = re.sub(r"^ascii\.", "", format)
        27     kwargs["format"] = format
   ---> 28 return write(table, filename, **kwargs)
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/ascii/ui.py:1083, in write(table, output, format, fast_writer, overwrite, **kwargs)
      1081 writer = get_writer(writer_cls=writer_cls, fast_writer=fast_writer, **kwargs)
      1082 if writer._format_name in core.FAST_CLASSES:
   -> 1083     writer.write(table, output)
      1084     return
      1086 lines = writer.write(table)
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/ascii/fastbasic.py:195, in FastBasic.write(self, table, output)
       190 def write(self, table, output):
       191     """
       192     Use a fast Cython method to write table data to output,
       193     where output is a filename or file-like object.
       194     """
   --> 195     self._write(table, output, {})
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/ascii/fastbasic.py:201, in FastBasic._write(self, table, output, default_kwargs, header_output, output_types)
       197 def _write(
       198     self, table, output, default_kwargs, header_output=True, output_types=False
       199 ):
       200     # Fast writer supports only 1-d columns
   --> 201     core._check_multidim_table(table, max_ndim=1)
       203     write_kwargs = {
       204         "delimiter": self.delimiter,
       205         "quotechar": self.quotechar,
       206         "strip_whitespace": self.strip_whitespace_fields,
       207         "comment": self.write_comment,
       208     }
       209     write_kwargs.update(default_kwargs)
   
   File ~/python/dev/lib/python3.13/site-packages/astropy/io/ascii/core.py:57, in _check_multidim_table(table, max_ndim)
        55 nd_names = [col.info.name for col in table.itercols() if len(col.shape) > max_ndim]
        56 if nd_names:
   ---> 57     raise ValueError(
        58         f"column(s) with dimension > {max_ndim} "
        59         "cannot be be written with this format, try using 'ecsv' "
        60         "(Enhanced CSV) format"
        61     )
   
   ValueError: column(s) with dimension > 1 cannot be be written with this format, try using 'ecsv' (Enhanced CSV) format

   but it seems this has now changed to be a feature request for a convenience method to flatten columns?
10. astrofrog

    added

    Feature Request

    and removed

    Bug

    on Jun 3, 2026
11. taldcroft commented on Jun 3, 2026

    

    taldcroft

    on Jun 3, 2026

    Member

    More actions

    Sounds like a good feature which is not too much effort. I think it should flatten structured or multi-dim columns along with represent_mixins_as_columns. The latter is now exposed as a public function. I'll try to have a go at this.

    👍React with 👍1Reacted by Derek Homeier
12. pllim

    removed

    needs-reconfirmation

    on Jun 3, 2026
13. taldcroft commented on Jul 28, 2026

    

    taldcroft

    on Jul 28, 2026

    Member

    More actions

    @astrofrog @dhomeier - please see #20138.

    👍React with 👍1Reacted by Derek Homeier
14. taldcroft

    closed this as completedin #20138on Jul 31, 2026