Repository navigation
Support for masked tables with missing/invalid data - #451
Conversation
|
@eteq, @mdboom, @iguananaut - Tom R and I are hoping for some review of this PR when you get a chance. I've fully integrated this into |
There was a problem hiding this comment.
Does this make Column never equal to a MaskedColumn -- and is that what we want? Maybe it should be isinstance(col, BaseColumn)? I could conceive of a case where we would want a Column to be equal to a MaskedColumn that happens to not have any elements masked out.
|
Looks good. We'll probably want to rebase and test on top of master before merging, of course. |
|
@taldcroft - I tried to address some of @mdboom's issues - but I left the remaining ones for you to check. |
|
I've pushed a new branch table-masked-rebase in taldcroft/astropy that is rebased on the upstream master from this morning. This also includes a new commit to use parse_strict='silent' when initializing units in a Column. I haven't had a chance to address the remaining comments here, and I'm a little worried about losing them if I force-push the rebased version (maybe I'm being paranoid?). In any case I may not have much time to work on this over the next few days due to travel. |
|
BTW, thanks @jiffyclub for the pytest 2.3.3 work. The table tests with fixtures are working great now. |
MaskedArray __getitem__ has a strange behavior where if a row mask is all False then it returns a np.void which has no mask attribute. This makes it impossible to then set the mask. Here we recast back to mvoid.
|
I'd be fine with making |
|
@astrofrog @eteq - I hope I have fixed the remaining numpy 1.4 errors (fingers crossed), they should be running now on jenkins. I think of The only other outstanding issue that I see (correct me if I'm wrong) is the documentation example of changing masking. I'm out of time this morning and will be busy during the day. |
|
@taldcroft - I'm pushing to my staging branch too, for the Windows tests (I think I'm the only user with a functional Windows staging branch) |
|
@astrofrog - any news on the Windows tests? Debian and MacOS look good now. |
|
@taldcroft - there were some issues with the running of the tests (it looks like shining panda initiated a build with the windows config, which causes all kinds of problems) but the builds that did run had no failures: I've forced a new build by adding an empty commit, so if we wait a couple of hours, we should know if it's all clear. |
|
I just noticed that |
|
@astrofrog @eteq - Are you OK with making t_masked = Table(t_unmasked, masked=True)This simplifies code, and perhaps is more explicit. |
|
@taldcroft - what about adding a masked column to an unmasked table, which currently makes the table become masked - would you keep that? If so, then I'm fine with your suggestion of removing the attribute/method way of changing the masking. |
|
@astrofrog - yes, the current functionality for add_column and add_row would remain in place (since they work and are tested!). Both of those are more obviously changing the table data. |
|
Ok, then I agree, let's go with your suggestion! |
|
@taldcroft - I pushed to staging too for the Windows tests - results will appear here: in about an ~hour |
|
@astrofrog - good, thanks. Do you see anything else that needs to be done? |
|
@taldcroft - the only thing is adding an entry to the changelog, but other than that, it looks good to me! |
|
@taldcroft - Yep, that solution seems fine to me - as you say, it's probably clearer that way. As far as I'm concerned this is good to go, then - thanks! |
|
Looks like all the WIndows tests passed! This is good to go (apart from the changelog). |
Support for masked tables with missing/invalid data
|
Changelog updated and #451 merged. |
|
@taldcroft - FYI, numpy/numpy#2733 has been merged. |
|
For info, numpy/numpy#2703 has been merged, which should fix the issue with |
Support for masked tables with missing/invalid data
Write error messages on stderr
Accepting as affiliated package: BayesicFitting
This PR adds support in the table package for masked tables that contain missing or invalid data. This provides a wrapper around the numpy.ma masked array package and provides workarounds for a few features of that package.
Notes: