Skip to content

Fix more reference leaks involving FITS_rec - #4539

Merged
embray merged 5 commits into
astropy:masterfrom
embray:fits/fix-fitsrec-reference-leak
Jan 28, 2016
Merged

embray merged 5 commits into
astropy:masterfrom
embray:fits/fix-fitsrec-reference-leak

Conversation

@embray

@embray embray commented Jan 27, 2016

Copy link
Copy Markdown
Member

This fixes additional reference leaks involving FITS_rec and Column that I mentioned in #520. Originally I wasn't going to spend any more time on this for now, but I had an "Aha!" moment for how to accomplish this.

Before this is merged I want to update this with more in-line comments, since otherwise this looks pretty mystifying. I also want to backport this fix the release branches, which means I need to use something other than WeakSet (which is not available on Python 2.6....though maybe there's a backport?)

Update: Yes, there is a backport of WeakSet for Python 2.6: https://pypi.python.org/pypi/weakrefset I'll see if I can get that going, and the make a separate PR for backporting this fix.

as explained here:
https://github.com/astropy/astropy/pull/520/files#diff-8ba48829876ba755e67054a1a90ec124R2367

I will add more in-line documentation of this in a subsequent commit.
But basically what's going on is this:  Whenever a Column object's
.array object is assigned to, it actually walks that array's bases to
check for the reference cycle, by seeing if one of those bases is a
FITS_rec instance with the present Column in its ColDefs.  If so, rather
than creating a hard reference it stores a weak reference to the
original FITS_rec, and the .array property returns the array via that
weak reference.

That way the reference cycle never gets created in the first place.

On the FITS_rec end, it actually also keeps a set of weak references to
all Column objects associated with that FITS_rec through its ._coldefs.
When FITS_rec.__del__ is called (because its reference count has gone to
zero) it deletes self._coldefs.  If that ColDefs object has no
references outside the FITS_rec (i.e. a user has not pointed a variable
to it), it and all Columns it contains are properly deleted.  This also
removes the column from the FITS_rec's _col_weakrefs set.

However, if there are still references to the Column (again, by a user
for example) then the data for that array is copied (maybe instead we
can just restore the reference to the original FITS_rec?), so the user
can continue to access that data.  This is allowed because by the time
FITS_rec.__del__ is called its underlying memory has not been freed yet.
The one exception to this was where data.deleter closed any mmap
underlying the data.  But I fixed the order of things so that the mmap
is still available to perform that copy.

This is all probably still fairly fragile, but in the vast majority of
cases it fixes the chance for any reference leaks.
@embray embray changed the title Fix more reference leaks involving FITS_rec Fix more reference leaks involving FITS_rec Jan 27, 2016
@embray embray added this to the v1.0.9 milestone Jan 27, 2016
…ibiting the reference leak. Remove the previous note in the test for astropy#520 about the existing reference leak, since it *shouldn't* apply any longer.
@embray embray modified the milestones: v1.2.0, v1.0.9 Jan 28, 2016
@embray

embray commented Jan 28, 2016

Copy link
Copy Markdown
Member Author

Okay, this PR will just go into master so I set the milestone to v1.2.0. But I'm going to make a separate PR of this for v1.0.x and v1.1.x

… make sure that when a weakref is assigned for Column.array, that the original .array contents, if any, are dropped.
embray added a commit to embray/astropy that referenced this pull request Jan 28, 2016
…ibiting the reference leak. Remove the previous note in the test for astropy#520 about the existing reference leak, since it *shouldn't* apply any longer.
embray added a commit to embray/astropy that referenced this pull request Jan 28, 2016
embray added a commit that referenced this pull request Jan 28, 2016
PR #4539 backport to v1.0.x with Python 2.6 support
embray added a commit that referenced this pull request Jan 28, 2016
PR #4539 backport to v1.0.x with Python 2.6 support
embray added a commit that referenced this pull request Jan 28, 2016
Fix more reference leaks involving FITS_rec
@embray
embray merged commit 650b733 into astropy:master Jan 28, 2016
@embray
embray deleted the fits/fix-fitsrec-reference-leak branch January 28, 2016 21:37
mvnnn pushed a commit to mvnnn/astropy that referenced this pull request Jan 30, 2016
…ibiting the reference leak. Remove the previous note in the test for astropy#520 about the existing reference leak, since it *shouldn't* apply any longer.
@astrofrog astrofrog modified the milestones: v1.2.0, v1.0.9 May 12, 2016
dhomeier pushed a commit to dhomeier/astropy that referenced this pull request Jun 12, 2016
…ibiting the reference leak. Remove the previous note in the test for astropy#520 about the existing reference leak, since it *shouldn't* apply any longer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants