Support reading and writing multidimensional columns in ECSV format by taldcroft · Pull Request #11155 · astropy/astropy

taldcroft · 2020-12-08T22:51:57Z

Description

This pull request is to support reading and writing multidimensional columns in ECSV format.

This also does consistent checking of table column dimensions for all built-in ASCII formats and raises a useful error message for multidimensional columns for formats that cannot support it.

For instance the following will round-trip and give back the same N-d column.

In [6]: t = Table()                                                                                                                                       

In [7]: t['nd'] = np.arange(18).reshape(3, 2, 3)                                                                                                          

In [8]: ascii.write(t, format='ecsv')                                                                                                                     
# %ECSV 0.9
# ---
# datatype:
# - {name: nd.0_0, datatype: int64}
# - {name: nd.0_1, datatype: int64}
# - {name: nd.0_2, datatype: int64}
# - {name: nd.1_0, datatype: int64}
# - {name: nd.1_1, datatype: int64}
# - {name: nd.1_2, datatype: int64}
# meta: !!omap
# - __serialized_columns__:
#     nd:
#       '0_0': !astropy.table.SerializedColumn {name: nd.0_0}
#       '0_1': !astropy.table.SerializedColumn {name: nd.0_1}
#       '0_2': !astropy.table.SerializedColumn {name: nd.0_2}
#       '1_0': !astropy.table.SerializedColumn {name: nd.1_0}
#       '1_1': !astropy.table.SerializedColumn {name: nd.1_1}
#       '1_2': !astropy.table.SerializedColumn {name: nd.1_2}
#       __class__: astropy.table.column.Column
# schema: astropy-2.0
nd.0_0 nd.0_1 nd.0_2 nd.1_0 nd.1_1 nd.1_2
0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17

Fixes #11148

taldcroft · 2020-12-08T23:15:11Z

~~This doesn't work for N-d Quantity right now.~~

mhvk

Just a quick flyby of the draft: I like the approach! So far only a comment about a mistake in the comments...

Also add FLATTEN_MULTIDIM constant. This specifies the context types for which multidimensional data will be flattened to a list of 1-d arrays.

taldcroft · 2021-02-21T14:05:21Z

@mhvk - I rebased this and did a bit of touchup. I think this is ready for review now.

taldcroft

@mhvk - I made some comments that might help you a bit in review.

taldcroft · 2021-02-21T14:06:01Z

 astropy.time
 ^^^^^^^^^^^^

+- Add new time format ``unix_ptp`` (Precision Time Protocol) which provides


Oops, something went wrong in the rebase. Will fix.

taldcroft · 2021-02-21T14:12:39Z

 import pytest
 import numpy as np

-from astropy.table import Table, Column, QTable, NdarrayMixin


In this file I refactored the serialization testing out to a new table module test_serialize. The goal is to unify such testing for FITS, HDF5 and ECSV. However this proved a little tricky on my first attempt so the refactor only currently applies to ECSV. In the context of this PR that should be OK?

taldcroft · 2021-02-21T14:13:28Z

    """
-    t = Table([np.arange(4).reshape(2, 2), [1, 2]], names=['a', 'b'])
+    t = Table()
+    t['a'] = np.arange(24).reshape(2, 3, 4)


Basic testing here, more comprehensive in test_serialize.py.

taldcroft · 2021-02-21T14:18:27Z

+        (for N > 1) by optionally turning it into the required number of 1-d
+        arrays (e.g. for ECSV or None context).
+
+        For instance a Column with shape (2, 2, 20) will turn into four columns


Will fix the (2, 2, 20) here.

taldcroft · 2021-02-21T14:21:38Z

+    ndarrays within a Table.  This is a subclass of numpy.ndarray
+    and has the same initialization options as ndarray().
+    """
+    __module__ = 'astropy.table.table'


This class definition is the same as the original that was taken out of table.py, with the exception of this line and the one below. I don't remember precisely what hacking the __module__ here was doing, but it might have been for the docs.

taldcroft · 2021-02-21T14:27:56Z


+# Context types for which multidimensional data will be flattened to
+# a list of 1-d arrays.
+FLATTEN_MULTIDIM = ['ecsv']


Not clear if there will ever be another context that supports N-d serialization by flattening, but this seemed better than hardwiring 'ecsv' in the code.

taldcroft · 2021-02-21T14:28:21Z


-Creating a table with a `~astropy.coordinates.SkyCoord` column can be accomplished with a mixin
-column, which is supported by `ECSV <https://docs.astropy.org/en/stable/api/astropy.io.ascii.Ecsv.html>`_. To store a mixin column::
+Creating a table with a `~astropy.coordinates.SkyCoord` column can be


Just a reformat for column width.

taldcroft · 2021-02-21T14:29:26Z

        if not has_info_class(data, MixinInfo):
            col_cls = MaskedColumn if (hasattr(data, 'mask')
                                       and np.any(data.mask)) else Column
+            # TODO: I think the **info here is not really needed and tests pass


Nominally planning to leave this TODO for another time since the impact is trickier to understand.

taldcroft · 2021-02-21T14:30:24Z

        else:
            new_name = name + '.' + data_attr

+        # If the data attribute is not a Mixin (aka a Column or ndarray) and


This is the core inner change that makes it all work.

taldcroft · 2021-02-21T14:30:49Z

+class NdarrayMixinInfo(ParentDtypeInfo):
+    _represent_as_dict_primary_data = 'data'
+
+    def _represent_as_dict(self):


This is new.

taldcroft · 2021-03-06T23:20:07Z

Parenthetical to astropy/astropy-APEs#64, I see the section below this change which discusses multidimensional columns. With this PR it might be possible to make that scheme part of the standard, though it does rely on a special column naming convention. This is unique if the serialized metadata are parsed, but that is outside of the standard currently.

taldcroft · 2021-04-04T10:46:20Z

Closing in favor of ideas mentioned in #11368

taldcroft added table io.ascii Enhancement labels Dec 8, 2020

taldcroft added this to the v4.3 milestone Dec 8, 2020

taldcroft added the whatsnew-needed label Dec 8, 2020

embray reviewed Dec 9, 2020

View reviewed changes

Comment thread astropy/io/ascii/ecsv.py Resolved

taldcroft marked this pull request as draft December 10, 2020 15:22

mhvk reviewed Dec 10, 2020

View reviewed changes

Comment thread astropy/table/column.py Resolved

taldcroft added 8 commits February 21, 2021 05:49

Support reading and writing multidimensional columns in ECSV format

e23b978

Support NdarrayMixin and better implementation for Column

bf6070d

Consistent checking of col ndim and useful error msg [skip ci]

36f0b58

Convert multi-dim data attributes to NdarrayMixin for serialization

3652793

Also add FLATTEN_MULTIDIM constant. This specifies the context types for which multidimensional data will be flattened to a list of 1-d arrays.

Update ECSV test to include 2- and 3-d columns

40979e4

Move some ECSV tests to test_serialize

7b98d6e

Fix rebase conflicts

ea3c47f

Improved comments, minor cleanup

2710bd3

taldcroft force-pushed the ascii-serialize-nd-cols branch from 4b8c811 to 2710bd3 Compare February 21, 2021 12:00

Add documentation on ECSV multidimensional columns

02dfc72

taldcroft requested a review from mhvk February 21, 2021 14:04

taldcroft commented Feb 21, 2021

View reviewed changes

taldcroft mentioned this pull request Feb 23, 2021

Fail to index table with iloc[] after dump/load table into pickle file #11332

Open

Skip doctest that requires YAML

11ab09f

mbtaylor mentioned this pull request Mar 4, 2021

ECSV array/object-valued column support #11368

Closed

taldcroft closed this Apr 4, 2021

taldcroft deleted the ascii-serialize-nd-cols branch April 4, 2021 10:46

taldcroft mentioned this pull request Apr 14, 2021

Support reading and writing multidimensional and object columns in ECSV #11569

Merged

1 task done

taldcroft mentioned this pull request Dec 13, 2021

Allow structured dtype for column serialization. #12589

Closed

taldcroft mentioned this pull request Dec 27, 2021

Table as serializable mixin column #12610

Closed

Conversation

taldcroft commented Dec 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

taldcroft commented Dec 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taldcroft commented Feb 21, 2021

Uh oh!

taldcroft left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taldcroft commented Mar 6, 2021

Uh oh!

taldcroft commented Apr 4, 2021

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taldcroft commented Dec 8, 2020 •

edited

Loading

taldcroft commented Dec 8, 2020 •

edited

Loading