Repository navigation
Memory leak in astropy.table due to circular reference #7447
Description
The following code illustrates a memory leak:
import numpy as np
from astropy.table import Column, Table
for iter in range(1000):
c = Column(data=np.array([1, 2, 3]))
t = Table([c])
c = t = None
import objgraph
objgraph.show_backrefs(objgraph.by_type('Column'))The resulting graph shows 48 Column objects that all look like:
The issue is that Column.info._parent points to the Column object itself. We should use a weak reference to avoid this. I believe this is the root cause of #7440 so this shows that the bug has real-life impact.
Activity
astrofrog commentedon May 10, 2018
This was originally fixed in #6277 then reverted in #6342 and is technically a duplicate of #6276. I'd rather leave this open for now for more visibility.
astrofrog commentedon May 10, 2018
@taldcroft - I wonder if we should try and open a PR for #6277 again to see if we still get failures. It looks like after we had reverted it I then commented that the failures had started appearing before #6277 was merged: #6291 (comment)
So maybe it wasn't due to that. Could you open a PR with #6277 and we can try and restart the CI a large number of times to see if we ever get a failure?
pllim commentedon May 10, 2018
@astrofrog @taldcroft -- I revived it in #7448 . FYI. Feel free to push to that branch as needed using your maintainer superpower.
bsipocz commentedon May 10, 2018
What about merging it in if it passes, but then keep an eye on possible travis failures? I don't think we can restart it enough times to trigger it with high confidence (at least previously similar attempts of mine were not very successful).
saimn commentedon Jun 5, 2018
So, I guess this is fixed by #7448 ? (At least with @astrofrog 's example above I get no leaked references ! :))
bsipocz commentedon Jun 5, 2018
We're still keeping an eye on that, but since it's merged I think we should close this, too. (And revisit if it comes back again).