Skip to content

Provide table join method - #903

Merged
taldcroft merged 27 commits into
astropy:masterfrom
taldcroft:table/join
Mar 29, 2013
Merged

taldcroft merged 27 commits into
astropy:masterfrom
taldcroft:table/join

Conversation

@taldcroft

Copy link
Copy Markdown
Member

This PR provides a full-featured join method for the Table class. It includes support for inner, left, right, and outer joins. Metadata are merged as well (see below).

This PR also includes a new module astropy/utils/meta.py which provides functions for merging metadata. This includes a simple recursive merge algorithm which is tested. Some hooks are available for adjusting the merge rules, but this is preliminary and not well tested.

Comment thread astropy/table/_np_utils.pyx Outdated

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I ended up rewriting all this from scratch. There are just a few lines of remnant code from recfunctions.join_by().

The row order in the join output is driven by details in the
numpy sort routine.  This ends up being different between Mac
and Linux.  To handle this the table outputs are sorted as a
list before being compared.  Since the join row order is not
guaranteed to be consistent across platforms this is OK.
Previously get_merge_descrs() took an explicit left and right array.
It was updated, along with common_dtype(), to accept a list of arrays.
This is in preparation for other table operations like horizontal
and vertical stacking of lists of tables.
@hamogu

hamogu commented Mar 27, 2013

Copy link
Copy Markdown
Member

I looked at the documentation and I like the interface. It's clear, concise and well documented. I am not sure if I would have chosen join_type = 'inner' as the default, but I don't see any strong reason for any other type as default either.

Comment thread astropy/table/table.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we now have the astropy.units framework, should there be at least a warning if columns with unequal units are merged ?
In simple operations add, multiply etc. the units checking is very strict, so as a user I could naively expect that values are transformed automatically if I join to tables that have fluxes as e.g. ergs/s and W/s.
I don't think that needs to be implemented right now, but I would like some warning output that is it not (as that's easy to overlook in the documentation).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting warnings here for inconsistent values is a good idea. Will do.

@astrofrog

Copy link
Copy Markdown
Member

I'll echo @hamogu's comment about the default operation - I would personally have gone with an outer join as the default as it's more inclusive, but this is just a personal preference.

@astrofrog

Copy link
Copy Markdown
Member

This is great stuff by the way! I'll give it a proper review next week (unless it gets merged before) as I'm going to be away for the next few days.

@taldcroft

Copy link
Copy Markdown
Member Author

@hamogu and @astrofrog - about the default join_type, I was following the lead of pandas.merge and recfunctions.join_by, both of which default to inner. I guess for me inner is the more intuitive choice, in part because it never results in missing values so that you always get an unmasked table out if both inputs are not masked. I'm not adamant about the choice though.

@taldcroft

Copy link
Copy Markdown
Member Author

This will close half of #262.

taldcroft added a commit that referenced this pull request Mar 29, 2013
@taldcroft
taldcroft merged commit a0870ee into astropy:master Mar 29, 2013
@embray

embray commented Mar 29, 2013

Copy link
Copy Markdown
Member

Oops. I wanted a chance to look at this before merging but I dragged my feet on it for too long. Oh well.
In any case, this is awesome. Could we just rename astropy.utils.meta to astropy.utils.metadata? The former I think is too vague.

Comment thread astropy/table/np_utils.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor point, but could we instead backport collections.Counter in the compat module and just use that instead? I realize this is a trivial thing but if/when we ever drop 2.6 support we can immediately drop that too--this is preemptive code rot protection :) I'll do that if you think it makes sense to.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually looked at collections.Counter. It does a whole lot of stuff I never realized beyond just counting, and the class is actually pretty involved at first glance. In the end I decided porting wasn't worth the time to save 4 lines of trivial code. Even if it just works out of the box, you have to port the tests, and so on. If you want to do the port, go for it, but my feeling is that we (you) have higher priorities.

Otherwise it might be worth putting this little counter function in utils as a super-lightweight counter.

@embray

embray commented Mar 29, 2013

Copy link
Copy Markdown
Member

"INNER JOIN" is the default join operation on most SQL databases too. I don't think there's any right answer as to which is "more intuitive" but I think there's an ingrained cultural expectation in software dealing with tables that inner joins are the default.

@taldcroft

Copy link
Copy Markdown
Member Author

Sorry @iguananaut, I got a little impatient with the merge. I've been itching to start work on the other table operations like hstack, vstack, aggregate which depend on this branch. (I guess I could have branched off from table/join, but somehow I hate doing that).

I'll change utils/meta.py to utils/metadata.py per your suggestion.

Comment thread astropy/utils/meta.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not be the best way to go about creating the "out" dict. Some dict subclasses don't have non-empty constructors--the most common that comes to mind would be collections.defaultdict. If left is a defaultdict, if the expectation of this function is that a new defaultdict is returned there would probably also be an expectation that it have the same default_factory.

Instead why not just out = deepcopy(left)? This should preserve any invariants of the original object but still deep-copy all the values (and the keys for that matter, I think).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I don't remember if there was a reason I didn't use deepcopy but this seems like a cleaner approach now.

@taldcroft

Copy link
Copy Markdown
Member Author

@iguananaut - and of course it's not too late for comments / suggestions even though the initial version has been merged.

Comment thread astropy/utils/meta.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of just want to write this as

try:
    return left.__class__(itertools.chain(left, right))
except TypeError:
    raise MergeConflictError()

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I played with this a little bit. I like the idea, but I think it also opens up opportunity for unexpected results. For instance if left='abc' and right='cde' then the above succeeds and returns '<itertools.chain object at 0x5f5ed10>'.

We'll need to come back to merging meta for more realistic use cases, but for now I'll leave the strict but predictable version in place.

@embray

embray commented Mar 29, 2013

Copy link
Copy Markdown
Member

@taldcroft Yeah, no problem. This is awesome stuff so I can't complain :)

@eteq

eteq commented Apr 1, 2013

Copy link
Copy Markdown
Member

@taldcroft - I agree with the meta -> metadata change. I didn't realize that was in here until I saw this in the merged changelog, but I'm happy about it - we'll definitely want to move nddata to that, as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants