Skip to content

Fix copy of mask when copying a Table - #7486

Merged
taldcroft merged 5 commits into
astropy:masterfrom
saimn:fix-table-mask-copy
May 23, 2018
Merged

taldcroft merged 5 commits into
astropy:masterfrom
saimn:fix-table-mask-copy

Conversation

@saimn

@saimn saimn commented May 22, 2018

Copy link
Copy Markdown
Contributor

Fix #7362.

Since Numpy 1.14 the mask is no more copied systematically (see #5060), so it needs to be copied when a copy of a Table is done.

Milestone set to 2.0.7 as I think it needs to be backported.

saimn added 2 commits May 22, 2018 10:21
Fix astropy#7362.

Since Numpy 1.14 the mask is no more copied systematically, so it needs
to be copied when a copy of a Table is done.
@saimn saimn added the table label May 22, 2018
@saimn saimn added this to the v2.0.7 milestone May 22, 2018
@astropy-bot

astropy-bot Bot commented May 22, 2018 •

Copy link
Copy Markdown

Hi there @saimn 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

@saimn
saimn requested review from mhvk and taldcroft May 22, 2018 08:48
Comment thread astropy/table/column.py Outdated
if mask is None:
if hasattr(data, 'mask'):
mask = data.mask
mask = np.array(data.mask, copy=copy)

@mhvk mhvk May 22, 2018 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't quite go right if data.mask is np.ma.nomask - it makes an unnecessary mask array. Perhaps easiest to solve by rewriting the whole stanza a bit:

if mask is None:
    mask = getattr(data, 'mask', np.ma.nomask)

if mask is not np.ma.nomask and copy:
    mask = np.array(mask)  # copy is default; adding.

(Note: np.ma.nomask is better than False - it is what MaskedArray uses when no mask is passed in.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, just to sure, this would also replace the deepcopy case, and the mask = False added by @taldcroft recently (#7422) ? Your version makes sense but this is not the same behavior, as it results in using nomask where currently a full mask is created.
Maybe there is a good reason why a full mask is created ? nomask is often problematic as you need to mask values with the "right way" to be sure that the mask is created when needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

82 failed tests (for astropy.table) with nomask, because something like column.mask[column.data == col.null] = True no more works.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, well, that proofs that nomask is indeed a bad idea (though it must be possible to trigger this in current master by passing in a masked array with nomask...). So, then,

if mask is None:
    mask = getattr(data, 'mask', False)

if mask is np.ma.nomask:
    mask = False
elif mask is not False and copy:
    mask = np.array(mask)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to find a similar solution but when mask is not None, it is always copied (the deepcopy line), and there is a test for this. So I end up with something like this:

        if mask is None:
            # Issue #7399 with fix #7422.  Passing mask=None to ma.MaskedArray
            # is extremely slow (~3 seconds for 1e7 elements), while mask=False
            # gets quickly broadcast to the expected bool array of False.
            mask = getattr(data, 'mask', False)

            if mask is not False:
                mask = np.array(data.mask, copy=copy)
        elif mask is np.ma.nomask:
            # Force the creation of a full mask array as nomask is tricky to
            # use and will fail in an unexpected manner when setting a value
            # to the mask.
            mask = False
        else:
            mask = deepcopy(mask)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK to me - that even gets rid of data.mask = nomask. (But it should be np.array(mask), not np.array(data.mask).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks.

Comment thread astropy/table/tests/test_masked.py Outdated
def test_mask_copy():
"""Test that the mask is copied when copying a table (issue #7362)."""

t = Table([[2, 4, 1, 3], [3., 2., np.nan, 4]],

@taldcroft taldcroft May 22, 2018 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is correct (and reflects the original issue) but somehow is more obfuscated than need be.

c = MaskedColumn([1, 2], mask=[False, True])
c2 = MaskedColumn(c, copy=True)
c2.mask[0] = True
assert np.all(c.mask == [False, True])
assert np.all(c2.mask == [True, True])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, I simplified with your suggestions, thanks.

Comment thread CHANGES.rst Outdated
- Fix printing table row indexed with unsigned integer. [#7469]

- Fix copy of mask when copying a Table, as this is no more done systematically
by Numpy since version 1.14. [#7486]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add something like

Also fixed a problem when MaskedColumn was initialized with ``mask=np.ma.nomask``.

@taldcroft

Copy link
Copy Markdown
Member

Apart from minor comments looks good to me.

@mhvk mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all OK to me, so I'll merge. Thanks, @saimn!

@mhvk

mhvk commented May 23, 2018

Copy link
Copy Markdown
Contributor

Oops, the 32 bit test didn't pass. If anyone else has time to check...

@bsipocz

bsipocz commented May 23, 2018

Copy link
Copy Markdown
Member

@saimn @mhvk - Having a quick look those failures may be unrelated and due to pytest 3.6 being used and RemovedInPytest4Warnings are being issued.
(I've randomly checked a few other passing builds and they still use pytest <3.5).

cc @drdavella

_________________ ERROR at setup of test_highlevel_api_remote __________________
[gw3] linux -- Python 3.6.5 /opt/python/cp36-cp36m/bin/python

item = <Function 'test_highlevel_api_remote'>

    def pytest_runtest_setup(item):
    
        remote_data = item.get_marker('remote_data')
        internet_off = item.get_marker('internet_off')
    
        remote_data_config = item.config.getvalue("remote_data")
    
        if remote_data is not None and internet_off is not None:
            raise ValueError("remote_data and internet_off are not compatible")
    
        if remote_data is not None:
>           source = remote_data.kwargs.get('source', 'any')
E           _pytest.deprecated.RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
E           Please use node.get_closest_marker(name) or node.iter_markers(name).
E           Docs: https://docs.pytest.org/en/latest/mark.html#updating-code

/opt/python/cp36-cp36m/lib/python3.6/site-packages/pytest_remotedata/plugin.py:68: RemovedInPytest4Warning
______________________ ER

@bsipocz

bsipocz commented May 23, 2018

Copy link
Copy Markdown
Member

As a test for the above hypothesis I've restarted the latest master merge build for 32-bit that was passing previously.

@mhvk

mhvk commented May 23, 2018

Copy link
Copy Markdown
Contributor

@bsipocz - thanks for investigating!

@bsipocz

bsipocz commented May 23, 2018

Copy link
Copy Markdown
Member

The suspicion proved to be right, so I've added a temporary version limitation to the 32bit build, and suggest to merge this and deal with pytest in a separate PR.

@taldcroft taldcroft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @saimn !

@taldcroft
taldcroft merged commit a51d710 into astropy:master May 23, 2018
@saimn
saimn deleted the fix-table-mask-copy branch May 23, 2018 19:27
bsipocz pushed a commit that referenced this pull request May 28, 2018
Fix copy of mask when copying a Table
bsipocz pushed a commit that referenced this pull request May 28, 2018
Fix copy of mask when copying a Table
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants