Add option to break apart structured array columns

What is the problem this feature will solve?

Some table I/O formats, like MRT, do not support structured array columns.
It would be good to have a method to break a structured array into a set of single columns and to encode this information such that the original structured array can be recovered.

Describe the desired outcome

Maybe add a kwarg to astropy.table.serialize.represent_mixins_as_columns to break apart structured arrays.
I would suggest serializing the names as original[i] (number or name) for each field i in the original array.

Additional context

Discovered in #14883

I have an example of the output structure in 2550247

@mhvk

Ping @mhvk @taldcroft

Effectively we have this method already for ECSV, so is your suggestion just to expose that. Of course, by default the table would then not round-trip, unless we also give an option on input.

Effectively we have this method already for ECSV, so is your suggestion just to expose that.

That would be great!

Of course, by default the table would then not round-trip, unless we also give an option on input.

Couldn't it automatically round trip if we used the same SerializedColumn scheme as is used for all the other formats: SkyCoord, etc?

Just an FYI that for ECSV the naming convention is name.subname not name[subname]. That follow the general ECSV convention for specifying object attribute names for columns. Of course that is different from how we represent the table in repr(), so I'm not sure what to recommend.

In [8]: t
Out[8]: 
<Table length=3>
  a      b     c        tn [a, b, c]     
int64 float64 str1 (int64, float64, str1)
----- ------- ---- ----------------------
    1     1.0    c           (1, 1., 'c')
    2     2.0    d           (2, 2., 'd')
    3     3.0    e           (3, 3., 'e')

In [9]: ascii.write(t, "ecsv")

In [10]: 

In [10]: ascii.write(t, format="ecsv")
# %ECSV 1.0
# ---
# datatype:
# - {name: a, datatype: int64}
# - {name: b, datatype: float64}
# - {name: c, datatype: string}
# - {name: tn.a, datatype: int64}
# - {name: tn.b, datatype: float64}
# - {name: tn.c, datatype: string}
# meta: !!omap
# - __serialized_columns__:
#     tn:
#       __class__: astropy.table.column.Column
#       data: !astropy.table.SerializedColumn
#         a: !astropy.table.SerializedColumn {name: tn.a}
#         b: !astropy.table.SerializedColumn {name: tn.b}
#         c: !astropy.table.SerializedColumn {name: tn.c}
# schema: astropy-2.0
a b c tn.a tn.b tn.c
1 1.0 c 1 1.0 c
2 2.0 d 2 2.0 d
3 3.0 e 3 3.0 e

Another consideration with round-tripping is that it can't be done uniquely without additional metadata in the table. ECSV provides this in the __serialized_columns__ which defines exactly how to reconstruct the table. Otherwise the code needs to guess that tn.a, tn.b, tn.c are elements of a single structured array tn.

The top description also mentions the possibility of original[0], original[1] etc, so is this idea meant to apply to N-d Columns (where N > 1)? I don't think of those as "structured arrays".

@nstarman

@nstarman - like @taldcroft, I think name.field would be the best way to name columns. The main issue is that MRT, unlike ECSV, has no well-defined way to indicate that a column belongs to a structure, so one would need an option for interpreting every name with a dot in it is part of a structured column. That would not be hard, but probably should default to False for Table.read -- obviously, the default could differ for Cosmology.read!

p.s. This does make me think of #14780, where you are introducing a rename option. In a way, that is a subset of serialization, and one could imagine having a general option to (de)serialize columns, just following the current practice, where one inputs a table and gets out a new table with different columns (plus meta information on what was done).

It might be worth considering that the CDS / MRT formats do allow for arbitrary unstructured notes to be included, and that could be used to include the metadata needed to reconstruct a table. See e.g. https://journals.aas.org/mrt-standards/. This was undoubtedly not the intent, but it you can imagine that all the name.field columns are supplied with a note (N) which is then like below. I'm not sure ApJ would appreciate this hack. 😄

Note (1):
     tn:
       __class__: astropy.table.column.Column
       data: !astropy.table.SerializedColumn
         a: !astropy.table.SerializedColumn {name: tn.a}
         b: !astropy.table.SerializedColumn {name: tn.b}
         c: !astropy.table.SerializedColumn {name: tn.c}

The top description also mentions the possibility of original[0], original[1] etc, so is this idea meant to apply to N-d Columns (where N > 1)? I don't think of those as "structured arrays".

Being able to break apart N-d arrays would also be nice! And , for users, probably even more commonly encountered than structured arrays .

It might be worth considering that the CDS / MRT formats do allow for arbitrary unstructured notes to be included, and that could be used to include the metadata needed to reconstruct a table.

That might be nice. Perhaps guarded by a flag?

Navigation Menu

Repository navigation

Add option to break apart structured array columns #14913

What is the problem this feature will solve?

Describe the desired outcome

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add option to break apart structured array columns #14913

Description

What is the problem this feature will solve?

Describe the desired outcome

Additional context

Activity

nstarman commented on Jun 4, 2023

mhvk commented on Jun 5, 2023

nstarman commented on Jun 5, 2023

taldcroft commented on Jun 5, 2023

taldcroft commented on Jun 5, 2023

taldcroft commented on Jun 5, 2023

mhvk commented on Jun 5, 2023

mhvk commented on Jun 5, 2023

taldcroft commented on Jun 5, 2023

nstarman commented on Jun 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions