@taldcroft - as you'll see, @wkerzendorf and I had been working on the same problem, but from quite a different angle (#1835). I'll look at your implementation in more detail tomorrow.

@taldcroft so basically your Column is a time object. I'm was wondering - what do we need Columns for then? We have Quantity to store numbers and units, NDData to store numbers, units and metadata, time, coordinates ... . The table could just deal with the formatting. in #1835 there's still a small shim for Columns, but I'm wondering if we couldn't drop columns completely (table would know the column name).

@wkerzendorf - currently Quantity only handles numbers, not strings, or objects. Both of these need representation in a table. Since NDData is a container class (not inherited from ndarray) it is unsuitable for use as a column class (see below).

Basically my view on any fundamental changes to Table is that they satisfy:

1. All current tests pass
2. The behavior of every allowed numpy operation on table columns is unchanged from current. Having deep numpy compatibility is crucial. We decided early on to inherit from ndarray and I think this has proved a good choice.
3. Full support for all numpy array data types.
4. Full support for masking (again with no API changes). Even though numpy.ma has its problems, it basically works quite well and there are no competing alternatives. We do not want to roll our own.

Even though Quantity is really great in a lot of contexts, I don't think it is ideal for a base column class. As we learned from Coordinates, there can be substantial overheads that aren't apparent in simple testing. But more importantly it is sufficiently different from ndarray that it would almost certainly create a lot of API breakage.

One thing I have taken away from #1835 is the question of whether we might do away with the underlying structured array that contains the table data in favor of a list of Column or MaskedColumn objects (where I mean the current versions). That might actually simplify things a fair bit, at the expense of making the row access code more complex and slower. A lot of the subtlety in the current code is maintaining the columns as views into the structured array. That might not be needed and would make support for generalized columns much easier.

@taldcroft - While I agree generally with your conditions, I think it is important to note that some current behaviour of Table is wrong. In particular, I would argue that if a Column has a unit, this should be taken into account, and all numpy routines should behave as if this Column held a Quantity. (This is not to say that Column should be a Quantity; string-valued columns are an obvious countercase.)

I think that treating a column with units as Quantity should be an opt-in choice, not the default. There is a very long legacy of people writing computational code where the units are handled manually and we should not presume to change that tradition over night. I for one do not want columns automatically turning into Quantity. So I could imagine a table method like enable_quantity() that converts Column to Quantity where possible.

@taldcroft Maybe, I should clarify - I didn't mean that a Column would be Quantity. I think a column can be Quantity. A column can also be an ndarray (of the right dimension) that can hold for example strings (objects) and no unit in that case. I wanna do completely away with columns (wondering if that works) and not bring about a new column construct that just inherits from Quantity. Similar where you say now t['b'] == TimeObject. I want that actually to be a time object stored in the table. I envision that for most columns a Quantity would be used (just a few numbers and a unit).

@taldcroft - I'm not sure I agree on the ability to have columns with a unit that are not Quantity, I don't think one should be able to add wrong units in astropy altogether; e.g., do you really want to allow one to join two Column's that have different units?
That said, as far as I currently implemented it, as long as a Column holds an ndarray, its unit attribute is essentially treated as part of meta-data that carries no meaning in doing any calculation.

Anyway, I'm not sure we need to decide this now;; I really need to look at what you do in some detail, and think it would be more productive to focus first on, e.g., the Time-side of things, or, more generally, what does a class need to provide to enable it to become a column in a Table. As you'll see from the TimeArrayRepr class I wrote, our idea was that the only requirement would be for the class to present itself as a single, though possibly structured array, so that it can be view'd as different classes (i.e., column/table data, or back to Time), modulo some trivial resetting of properties. The nice part about this is that Time does not have to know anything about how tables work.

@mhvk - on the second point of what to decide now, I think we are actually not so far apart in the general view. The idea is that a class like Time needs to be able to participate in all the supported operations of a Table. You did this with a wrapper class TimeArrayRepr while I did it with a protocol that says "to participate in a Table a class must provide certain behaviors (methods, attributes)". The key points in my version are the two methods that basically answer two questions:

• Give me the columns corresponding to the "internal" data to put in the table ndarray.
• Create on object of this class from specified "internal" data from the table ndarray.

Then there is the data property that gives access to the representation of the object as a single column of values. Note that both of our implementations give you the actual class object when accessing a column. I.e. type(t['time_col']) is Time.

Having done all this, I'm not convinced my approach of putting the internal data into the table _data ndarray is the best path. It works pretty nicely for Time, but I ran into problems for Coordinates where the internal data were segmented in memory (for reasons I don't understand). The added complexity might not be worth it, though I haven't entirely given up.

For these mixin columns there is perhaps no reason to store the internal data in the table _data at all. That simplifies the protocol and implementation considerably and starts to look like your version. The important part of the protocol idea is that when writing table code you need to make certain assumptions about how a column behaves and what attributes are available.

As mentioned earlier one can go further and not have a _data container at all! In this case all the Column objects would hold their own data independently. This is more extreme, and even though _data is nominally private I bet anything there is plenty of code out there that uses it...

@taldcroft - indeed, the background ideas are definitely similar, with the class simply needing to provide the ability to be turned into correctly sequentiable ndarray and the reverse. The one (small) difference is that in my implementation type(t['time_col'] is still a Column, though every attribute including all operators that are not defined in the Column class get interpreted as attributes/methods of the underlying Time object, i.e., the column behaves as if it is a virtual TimeColumn(Column, Time) (reminds me -- I should ensure that isinstance(t['time_col'], Time)==True). The possible advantage is that one still has access to self.parent_table, BaseColumn.__repr__, etc.

The part that I perhaps like most about what I did, is that any ndarray subclass can trivially be made into a Column, with no changes whatsoever needed (at least for the ones we have so far, Quantity, Angle, etc.); I maybe should try to split this out.

But better back to some astronomy now...

Passing tests now (at least locally).

This will need rebasing now that #1842 has been merged (it makes Travis work again)

I need to take a closer look at this, but 👍 in principle. This might end up being very, very useful for FITS tables.

Closing because this PR is incompatible with #2790 and is superceded by #3011.