Repository navigation
Table MaskedColumn updating issues #2997
Description
@taldcroft Can you please look at example 1 and 3 in this notebook?
http://nbviewer.ipython.org/gist/cdeil/69b79c04b2477dbf611a
Is this a bug in the masked table implementation?
If not, can you point me to or add documentation how one can update missing data in Astropy tables?
I did look through the docs a bit, but couldn't find it.
Naively this is what I tried ... it results in a table where the mask is in a weird (corrupt?) state:
# Try to update missing information in column `b` from column `c`
mask = t['b'].mask
t['b'][mask] = t['c'][mask]While looking through the docs I saw this section which explained that rows are views for non-masked tables, but copies for masked tables. That's another bug I had in my script. Aargh ... I'm thinking of re-writing my script completely using pandas.DataFrame instead, in the hope that it will let me process my data with missing values more easily.
Activity
astrofrog commentedon Sep 30, 2014
@cdeil - that seems like a bug, so labeling it as such
mhvk commentedon Sep 30, 2014
@cdeil - masked tables are still a source of problems -- some resulting from papering over underlying issues in
MaskedArray-- but to at least have something while this is investigated: inastropy.utils.iers.IERS_A, the following does work:(the need for the
.datais due to a numpy 1.9 bug; numpy/numpy#5095)taldcroft commentedon Sep 30, 2014
Yes, it's a bug. It may not be consolation to you at the moment, but this is apparently fixed by the Table refactor that I'm currently working on (#2790, which I just converted to a PR). This refactor removes the underlying MaskedArray, which I suspect is part of the problem here. Here is your notebook using the #2790 branch:
http://nbviewer.ipython.org/gist/taldcroft/04842c847193579a5bab
cdeil commentedon Sep 30, 2014
OK, thanks for confirming that it's a bug ... it was driving me mad this morning.
I'm surprised no-one has has run into this behaviour before and that it's not covered by a unit test.
@taldcroft Thanks for fixing this in #2790.
cdeil commentedon Sep 30, 2014
I've filed two feature requests that I think would have helped me debug my script much faster (but are really generally useful, I think): #2998 and #2999.
taldcroft commentedon Oct 1, 2014
I marked this as fixed when #2790 gets merged, but we'll leave it open until then. I'll be sure to add tests for this case.
Add test of setting from a masked column astropy#2997
Add test of setting from a masked column astropy#2997
Add test of setting from a masked column astropy#2997
cdeil commentedon Dec 1, 2016
I re-ran the notebook that shows the bug again with Astropy master:
http://nbviewer.jupyter.org/gist/cdeil/a33b8925fe6b0f9fba410175a3ccbf18
As far as I can see, the behaviour and results are OK not, and this issue can be closed.
@taldcroft or @mhvk - Can you please double-check?
The one thing is that there's this warning:
Is it possible to change something in
astropy/table/column.py:1096so that the code works with old / new Numpy and doesn't emit this warning?Is a test case in place?
saimn commentedon Dec 1, 2016
For
MaskedArrayFutureWarningsee #5060: the behavior has not changed yet. I think it will be changed in Numpy 1.13 but I cannot find the related issue.mhvk commentedon Dec 1, 2016
The relevant numpy issues is numpy/numpy#5580
Bottom line: in the future, both data and mask get shared for views. I made a new issue to keep track of that (#5530), but @saimn correctly points out that #5060 already exists. We should move discussion of the warning there.
Anyway, for this issue, we can ignore the warnings, which, I think, means it can be closed (I confirmed that the original example now works). So, I'll do that!