Repository navigation
Fix copy of mask when copying a Table - #7486
Conversation
Fix astropy#7362. Since Numpy 1.14 the mask is no more copied systematically, so it needs to be copied when a copy of a Table is done.
|
Hi there @saimn 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃. Everything looks good from my point of view! 👍 If there are any issues with this message, please report them here. |
| if mask is None: | ||
| if hasattr(data, 'mask'): | ||
| mask = data.mask | ||
| mask = np.array(data.mask, copy=copy) |
There was a problem hiding this comment.
This doesn't quite go right if data.mask is np.ma.nomask - it makes an unnecessary mask array. Perhaps easiest to solve by rewriting the whole stanza a bit:
if mask is None:
mask = getattr(data, 'mask', np.ma.nomask)
if mask is not np.ma.nomask and copy:
mask = np.array(mask) # copy is default; adding.
(Note: np.ma.nomask is better than False - it is what MaskedArray uses when no mask is passed in.)
There was a problem hiding this comment.
So, just to sure, this would also replace the deepcopy case, and the mask = False added by @taldcroft recently (#7422) ? Your version makes sense but this is not the same behavior, as it results in using nomask where currently a full mask is created.
Maybe there is a good reason why a full mask is created ? nomask is often problematic as you need to mask values with the "right way" to be sure that the mask is created when needed.
There was a problem hiding this comment.
82 failed tests (for astropy.table) with nomask, because something like column.mask[column.data == col.null] = True no more works.
There was a problem hiding this comment.
Ah, well, that proofs that nomask is indeed a bad idea (though it must be possible to trigger this in current master by passing in a masked array with nomask...). So, then,
if mask is None:
mask = getattr(data, 'mask', False)
if mask is np.ma.nomask:
mask = False
elif mask is not False and copy:
mask = np.array(mask)
There was a problem hiding this comment.
I was trying to find a similar solution but when mask is not None, it is always copied (the deepcopy line), and there is a test for this. So I end up with something like this:
if mask is None:
# Issue #7399 with fix #7422. Passing mask=None to ma.MaskedArray
# is extremely slow (~3 seconds for 1e7 elements), while mask=False
# gets quickly broadcast to the expected bool array of False.
mask = getattr(data, 'mask', False)
if mask is not False:
mask = np.array(data.mask, copy=copy)
elif mask is np.ma.nomask:
# Force the creation of a full mask array as nomask is tricky to
# use and will fail in an unexpected manner when setting a value
# to the mask.
mask = False
else:
mask = deepcopy(mask)
There was a problem hiding this comment.
Looks OK to me - that even gets rid of data.mask = nomask. (But it should be np.array(mask), not np.array(data.mask).
| def test_mask_copy(): | ||
| """Test that the mask is copied when copying a table (issue #7362).""" | ||
|
|
||
| t = Table([[2, 4, 1, 3], [3., 2., np.nan, 4]], |
There was a problem hiding this comment.
This test is correct (and reflects the original issue) but somehow is more obfuscated than need be.
c = MaskedColumn([1, 2], mask=[False, True])
c2 = MaskedColumn(c, copy=True)
c2.mask[0] = True
assert np.all(c.mask == [False, True])
assert np.all(c2.mask == [True, True])
There was a problem hiding this comment.
Indeed, I simplified with your suggestions, thanks.
| - Fix printing table row indexed with unsigned integer. [#7469] | ||
|
|
||
| - Fix copy of mask when copying a Table, as this is no more done systematically | ||
| by Numpy since version 1.14. [#7486] |
There was a problem hiding this comment.
Add something like
Also fixed a problem when MaskedColumn was initialized with ``mask=np.ma.nomask``.
|
Apart from minor comments looks good to me. |
|
Oops, the 32 bit test didn't pass. If anyone else has time to check... |
|
@saimn @mhvk - Having a quick look those failures may be unrelated and due to pytest 3.6 being used and cc @drdavella |
|
As a test for the above hypothesis I've restarted the latest master merge build for 32-bit that was passing previously. |
|
@bsipocz - thanks for investigating! |
|
The suspicion proved to be right, so I've added a temporary version limitation to the 32bit build, and suggest to merge this and deal with pytest in a separate PR. |
Fix copy of mask when copying a Table
Fix copy of mask when copying a Table
Fix #7362.
Since Numpy 1.14 the mask is no more copied systematically (see #5060), so it needs to be copied when a copy of a Table is done.
Milestone set to 2.0.7 as I think it needs to be backported.