Add depth-wise table stacking (dstack?)

@eblur

Depth-wise stacking, where columns are stacked into a new dimension. This assumes t1, t2, t3 all have the same columns and length, though with masking one doesn't necessarily have that requirement. There is probably a cleaner numpy way to do this without transposing (using axis) but I forget at the moment.

In [18]: tbls = [t1, t2, t3]

In [19]: out = Table()

In [20]: for name in tbls[0].colnames:
    ...:     out[name] = np.vstack([tbl[name] for tbl in tbls]).transpose()
    ...:     
    ...:     

In [21]: out
Out[21]: 
<Table length=3>
a [3]    b [3]    c [3] 
int64   float64    str1 
------ ---------- ------
1 .. 1 1.0 .. 1.0 c .. c
2 .. 2 2.0 .. 2.0 d .. d
3 .. 3 3.0 .. 3.0 e .. e

This is inspired by a tutorial that was written and comment from @eblur.

I just came on here looking to see if this was in development! I heartily support having dstack for tables.

Also, numpy has gotten rid of hstack/vstack/dstack in favor of concatenate and specifying the axis (though h/v/dstack is still back-compatible at this moment). What are your thoughts on following suit vs keeping as-is?

@abigailStev

Like the idea. As mentioned by @abigailStev, numpy is trying to remove hstack and vstack, in favour of a general stack with an axis. For tables, I think it is particularly intuitive to just call this stack. I think it would still have an axis argument (where axis=0 is the same as vstack and there is no numerical axis for which one recovers hstack [it could perhaps be None as in, this is not an axis along the actual arrays in a column]).

It needs some thought on what to do for multi-dimensional columns; would one by default add an axis at the end? Or at axis 1 (cannot be 0, since that is along the column).

I'm open for discussion, but I don't quickly see how the axis argument can map intuitively to a table since there is no unambiguous relation between axis and columns. I definitely can't quite see how a number could intuitively distinguish between stacking the entire table list horizontally vs stacking the tables depth-wise. We could have a stack(direction='horizontal|vertical|depth')?

In other news I really want to make all these Table methods instead of module functions.

@taldcroft

I support making it a method instead of module functions. I primarily use 2D and 3D tables that have dimensions [a,b] or [a,b,c], so in my mind it's easy to relate to stacking along axis 0=a or 1=b or 2=c (for example, I want to concatenate along b, but I often forget whether 'vertical' is along a or b). This being said, I don't know enough about the inner workings of table columns, so if a stack method with specified direction like @taldcroft's example is much better, I'm ok with that.

I think I managed to confuse myself... in numpy speak, stacking always creates a new axis, and concatenate is used to join arrays along an existing axis. So table's vstack is really concatenate(..., axis=0), and hstack does not really have a direct analogue (since we just create more columns).

But this means that for a new stack, one can follow numpy quite directly, and have any axis argument simply apply to the columns, treating those as input arrays for np.stack, i.e., axis specifies the axis in the output result that will be the "stack". axis=0 would then be somewhat strange (but perhaps useful) in creating a new table with length , axis=1 would lead to columns whose first dimension remains the normal table length, and second dimension the input table, and axis=-1 would make the last dimension of the result the index to the original table. Of those, really axis=1 seems the only one that has a clear use, but since this really becomes a method that can act on columns and uses the fact that they are ndarray (or ShapedLikeNDArray), one might as well support all axes.

p.s. In a possible concatenate, axis=0 would equal the current vstack, but I cannot see that much use for any other axis. It might make sense as a general method for columns, though.

@mhvk

@mhvk - leaning on the heritage of numpy is a good thing, but honestly your proposal seems really confusing and I still don't understand it entirely. Having gone over to the pandas page on concat, and looking at all the pictures of table-level concat, the paradigm that @abigailStev proposed seems simple and makes sense to me (and probably pandas users).

To reiterate we think of a table as 2D array and then stack accordingly. So the "shape" of a table would be (n_rows, n_cols). The main problem is that stacking on axis=0 or axis=1 is concatenation (does NOT create a new axis, in line with pandas or numpy concatenation) while stacking on axis=2 is really making a new axis, in line with numpy stack. So there is no single word for all these that is consistent with usage in other packages.

We could just punt on perfection and call the new unified method concat in agreement with pandas/numpy for 2 of 3 cases. So concat(.., axis=2) would be depth-wise stacking and would effectively do np.stack([t1[col], t2[col], .., tn[col]], axis=-1) on the individual columns. In other words the last dimension is always the index to the original table. Since tables are heterogenous and you don't normally expect all columns to be multi-d or have the same shape, it seems reasonable to have a simple rule that axis=2 concatenation stacks the individual column on the last axis. In this version of concat, axis could only take the values 0, 1, or 2.

@taldcroft

@taldcroft - my logic is actually relatively simple: that for everything except hstack, the operations we are talking about are really operations on the individual columns, and they might as well be treated as such. Indeed, I would suggest to implement them at the column level and then use it for the table. I do agree that stacking on the last axis is a logical choice, but I think stacking on the first axis (the one just after the table length) is just as logical. (I guess my axis=0 case is more like the row-equivalent of a table of tables that you once mentioned).

But I am not against a new method name, though in that case I'd suggest to not use axis at all, which is clearly rather confusing in the table context, but rather a string keyword, which takes the values "rows", `"columns", "depth" or so.

Separately, I'd guess that for some "depth stacking", one would want to exclude some columns (which perhaps hold identical information anyway).

for everything except hstack, the operations we are talking about are really operations on the individual columns, and they might as well be treated as such.

Yes, but hstack is 1/3 of the stacking operations so this not a good rule.. basically there is no consistent pattern:

hstack is not at all like numpy stack or concatenate
vstack is like numpy concatentate, except that axis must be 0.
dstack (depthwise) is like numpy stack, except that axis cannot be 0 (in the plan to this point).

But now I'm finally understanding your proposal (had to come to it myself by a different path)... and I think it would make sense if it were labeled cstack (or ??) which means "column stack" to emphasize that it is not a table-level stack operation like hstack and vstack, but applies to columns individually. I think a default of axis=-1 would make sense, but maybe there is an argument for axis=1? In any case as you say axis=0 would be funky but formally allowed.

The main point is to not subsume hstack and vstack into the new method because they really don't all fit into one pattern. In particular I can't see how a single arg like axis can handle all the cases in hstack, vstack and cstack. So that would mean having two args, where like direction and axis, where axis would be ignored for direction='horizontal'.

Just adding a cstack is nicer because it maintains the "one and only one way" to do things.

And in cstack one could also include an option to stack only a subset of columns, which also would not be meaningful for vstack or hstack.

👍 to something like cstack. I agree that we should not try to mash hstack and vstack in there, they are really concatenation operations.

OK. I think the implementation might be close to vstack but with a different output shape. Need to think about the right way to fix up Info.new_like() to handle this case:

    def new_like(self, cols, length, metadata_conflicts='warn', name=None):
        """
        Return a new Quantity instance which is consistent with the
        input ``cols`` and has ``length`` rows.

Clearly length is no longer enough, we need shape. But the idea of new_like is that it is supposed to do all the work instead of the calling method knowing anything about the input cols.

Yes, we should go through new_like somehow. Maybe if it is a tuple, it should just be the shape, in which case the input column shapes are ignored altogether? Have to try to do something that doesn't change the call for the current join operations.

#8939 has addressed this.

Navigation Menu

Repository navigation

Add depth-wise table stacking (dstack?) #6660

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add depth-wise table stacking (dstack?) #6660

Description

Activity

abigailStev commented on Oct 5, 2017

abigailStev commented on Oct 5, 2017

mhvk commented on Oct 5, 2017

taldcroft commented on Oct 5, 2017

abigailStev commented on Oct 5, 2017

mhvk commented on Oct 5, 2017

taldcroft commented on Oct 7, 2017

mhvk commented on Oct 7, 2017

taldcroft commented on Oct 7, 2017

taldcroft commented on Oct 7, 2017

mhvk commented on Oct 7, 2017

taldcroft commented on Oct 7, 2017

mhvk commented on Oct 7, 2017

bsipocz commented on Aug 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions