Skip to content

Do not copy info when getting scalar Time item - #10889

Merged
taldcroft merged 5 commits into
astropy:masterfrom
taldcroft:time-info-scalar-no-copy
Oct 22, 2020
Merged

taldcroft merged 5 commits into
astropy:masterfrom
taldcroft:time-info-scalar-no-copy

Conversation

@taldcroft

Copy link
Copy Markdown
Member

Description

This pull request is to address a memory and speed performance problem when iterating over a Time column that is indexed (e.g. in a TimeSeries object). See #10688 for details.

The simple fix is to not copy the info attribute when getting a scalar Time from an array-valued Time object. This is consistent with the behavior of Column.

Being very conservative I have milestoned this for 4.2 instead of as a pure performance fix. It seems quite unlikely, but maybe possible that some code is using the info data that were previously being copied for a scalar getitem.

Fixes #10688

@mhvk

mhvk commented Oct 17, 2020

Copy link
Copy Markdown
Contributor

The PR looks OK, but I'm still wondering whether it is the right approach: the following behaviour seems weird to me:

import numpy as np
from astropy.time import Time
from astropy.table import Table
t = Table([Time(np.arange(50000, 50010), format='mjd')], names=['t'])
t.add_index('t')
cx = t['t'][3:4]
cx.info.indices
[   t    rows
 ------- ----
 50000.0    0
 50001.0    1
 50002.0    2
 50003.0    3
 50004.0    4
 50005.0    5
 50006.0    6
 50007.0    7
 50008.0    8
 50009.0    9]
# Yet
cx.info.parent_table
# gives None

Why would a sliced time still have the full index associated with it?

In contrast, I can see the logic of

tx = t[3:4]
tx['t'].info.indices
[Index slice (3, 4, 1) of
    t    rows
 ------- ----
 50000.0    0
 50001.0    1
 50002.0    2
 50003.0    3
 50004.0    4
 50005.0    5
 50006.0    6
 50007.0    7
 50008.0    8
 50009.0    9]
tx['t'].info.parent_table
<Table length=1>
   t   
 object
-------
50003.0

I guess I wonder if indices should be omitted if the column is sliced; it looks like this would work fine, since the table already takes care of updating it if it is the table rather than the column that is sliced.

The advantage of solving it at the table level would be that the same solution would work for all other possible mixin columns too.

@mhvk

mhvk commented Oct 17, 2020

Copy link
Copy Markdown
Contributor

I think I've got an alternative, I think - see #10890

@mhvk

mhvk commented Oct 18, 2020

Copy link
Copy Markdown
Contributor

Moving discussion here about whether this is still needed given #10890. I can see the argument in #10890 (comment) that

for a scalar getitem I really think we don't want to copy info at all. For instance, the info attribute can have meta, and that might be a 5-megabyte structure. I just confirmed that a scalar getitem makes a deepcopy of that meta (in master and in this PR), so doing np.array(tm) (where tm is a large array-valued Time object with a big info.meta) will explode memory again.

Since this would hold also for SkyCoord, Quantity, etc., I think we'd need to do the same things for those, which suggests involving info itself in the decision about what would need to be copied. E.g., in the mechanism for DataInfo.__set__ one could add a check on its parents shape and do nothing if that shape was ():

if isinstance(value, DataInfo):
info = instance.__dict__['info'] = self.__class__(bound=True)
for attr in info.attr_names - info.attrs_from_parent - info._attrs_no_copy:
info._attrs[attr] = deepcopy(getattr(value, attr))

Note that we struggled a bit with this before - see #5047. I think the conclusion there was a good one: we should have clear, predictable behaviour...

p.s. Another possibility for implementation is to use that for most classes any subsetting, etc., goes through _apply() - we could change that to call a new, to-be-defined DataInfo._apply - that would allow including information about what is being done. But probably overkill.

@taldcroft

Copy link
Copy Markdown
Member Author

Since this would hold also for SkyCoord, Quantity, etc., I think we'd need to do the same things for those, which suggests involving info itself in the decision

Good plan. The question is whether this should only apply to the BaseColumnInfo class or DataInfo itself. I'm hesitant about extending this behavior change outside of tables. For BaseColumnInfo I just tried this out and it passes the new test in this PR (after reverting the change in time/core.py):

    def __set__(self, instance, value):
        # For Table columns do not set `info` when the instance is a scalar.
        try:
            if not instance.shape:
                return
        except AttributeError:
            pass

        super().__set__(instance, value)

@mhvk

mhvk commented Oct 19, 2020

Copy link
Copy Markdown
Contributor

Nice! I think I agree that it makes most sense to do it to BaseColumnInfo, since then a class that uses it has more or less explicitly signed up for making itself work with Table - and for Table's Column, scalars also have no connection to the table any more.

@taldcroft
taldcroft force-pushed the time-info-scalar-no-copy branch from f21286d to 2dfca24 Compare October 21, 2020 10:22
@taldcroft
taldcroft requested a review from mhvk October 21, 2020 10:30
@taldcroft

Copy link
Copy Markdown
Member Author

@mhvk - this should be ready for review.

@mhvk mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good modulo the left-over comment in Time._apply, which I think can either be removed or should be clarified. Approving since it is so minor.

Comment thread astropy/time/core.py
setattr(tm, attr, val)

# Copy other 'info' attr only if it has actually been defined.
# Copy other 'info' attr only if it has actually been defined and the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can now be removed/reverted? Or link to this PR perhaps instead of the issue.

@mhvk

mhvk commented Oct 21, 2020

Copy link
Copy Markdown
Contributor

p.s. Really like the detailed and thorough tests!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Excessive delay and memory usage when calling unsupported Numpy functions on TimeSeries.time

2 participants