Repository navigation
Provide table join method - #903
Conversation
There was a problem hiding this comment.
Oops, I ended up rewriting all this from scratch. There are just a few lines of remnant code from recfunctions.join_by().
The row order in the join output is driven by details in the numpy sort routine. This ends up being different between Mac and Linux. To handle this the table outputs are sorted as a list before being compared. Since the join row order is not guaranteed to be consistent across platforms this is OK.
Previously get_merge_descrs() took an explicit left and right array. It was updated, along with common_dtype(), to accept a list of arrays. This is in preparation for other table operations like horizontal and vertical stacking of lists of tables.
|
I looked at the documentation and I like the interface. It's clear, concise and well documented. I am not sure if I would have chosen |
There was a problem hiding this comment.
Given that we now have the astropy.units framework, should there be at least a warning if columns with unequal units are merged ?
In simple operations add, multiply etc. the units checking is very strict, so as a user I could naively expect that values are transformed automatically if I join to tables that have fluxes as e.g. ergs/s and W/s.
I don't think that needs to be implemented right now, but I would like some warning output that is it not (as that's easy to overlook in the documentation).
There was a problem hiding this comment.
Putting warnings here for inconsistent values is a good idea. Will do.
|
I'll echo @hamogu's comment about the default operation - I would personally have gone with an outer join as the default as it's more inclusive, but this is just a personal preference. |
|
This is great stuff by the way! I'll give it a proper review next week (unless it gets merged before) as I'm going to be away for the next few days. |
|
@hamogu and @astrofrog - about the default |
|
This will close half of #262. |
|
Oops. I wanted a chance to look at this before merging but I dragged my feet on it for too long. Oh well. |
There was a problem hiding this comment.
Minor point, but could we instead backport collections.Counter in the compat module and just use that instead? I realize this is a trivial thing but if/when we ever drop 2.6 support we can immediately drop that too--this is preemptive code rot protection :) I'll do that if you think it makes sense to.
There was a problem hiding this comment.
I actually looked at collections.Counter. It does a whole lot of stuff I never realized beyond just counting, and the class is actually pretty involved at first glance. In the end I decided porting wasn't worth the time to save 4 lines of trivial code. Even if it just works out of the box, you have to port the tests, and so on. If you want to do the port, go for it, but my feeling is that we (you) have higher priorities.
Otherwise it might be worth putting this little counter function in utils as a super-lightweight counter.
|
"INNER JOIN" is the default join operation on most SQL databases too. I don't think there's any right answer as to which is "more intuitive" but I think there's an ingrained cultural expectation in software dealing with tables that inner joins are the default. |
|
Sorry @iguananaut, I got a little impatient with the merge. I've been itching to start work on the other table operations like hstack, vstack, aggregate which depend on this branch. (I guess I could have branched off from table/join, but somehow I hate doing that). I'll change |
There was a problem hiding this comment.
This might not be the best way to go about creating the "out" dict. Some dict subclasses don't have non-empty constructors--the most common that comes to mind would be collections.defaultdict. If left is a defaultdict, if the expectation of this function is that a new defaultdict is returned there would probably also be an expectation that it have the same default_factory.
Instead why not just out = deepcopy(left)? This should preserve any invariants of the original object but still deep-copy all the values (and the keys for that matter, I think).
There was a problem hiding this comment.
Good point. I don't remember if there was a reason I didn't use deepcopy but this seems like a cleaner approach now.
|
@iguananaut - and of course it's not too late for comments / suggestions even though the initial version has been merged. |
There was a problem hiding this comment.
I kind of just want to write this as
try:
return left.__class__(itertools.chain(left, right))
except TypeError:
raise MergeConflictError()
There was a problem hiding this comment.
I played with this a little bit. I like the idea, but I think it also opens up opportunity for unexpected results. For instance if left='abc' and right='cde' then the above succeeds and returns '<itertools.chain object at 0x5f5ed10>'.
We'll need to come back to merging meta for more realistic use cases, but for now I'll leave the strict but predictable version in place.
|
@taldcroft Yeah, no problem. This is awesome stuff so I can't complain :) |
|
@taldcroft - I agree with the |
This PR provides a full-featured join method for the Table class. It includes support for inner, left, right, and outer joins. Metadata are merged as well (see below).
This PR also includes a new module astropy/utils/meta.py which provides functions for merging metadata. This includes a simple recursive merge algorithm which is tested. Some hooks are available for adjusting the merge rules, but this is preliminary and not well tested.