Memory leak in Table related to ref cycle in info attribute

The following demonstrates a memory leak via a reference cycle in the info attribute:

c = Column(np.arange(1e8), name='a')
for i in range(5):
    t = Table([c])

Well, this is not good. I have not dug into the implications, but here is where it showed up. I had stated (incorrectly) that introducing a reference cycle was not a problem.

neptune$ git bisect good
f112b3501c05c058ac8e83fdf6d4f0f646b56169 is the first bad commit
commit f112b3501c05c058ac8e83fdf6d4f0f646b56169
Author: Tom Aldcroft <[email protected]>
Date:   Sun Jun 28 19:28:41 2015 -0400

    Change weakref _parent_ref to direct ref _parent.
    
    This introduces a reference cycle, but should be OK.  Unlike
    e.g. Table and Column, there is very little risk of wanting
    an info object without the parent data object.

:040000 040000 32f9dd1f5886ec37701ace3c3dc49696743804a2 4a7f1cd9fde8e43c741e9c3d6823440117650821 M

Should be able to turn the Info._parent ref back into a weakref. This is working now except for some strange failure here:

astropy/io/ascii/tests/test_html.py ........s....F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    def test_multicolumn_write():

That's not a memory-leak, that's normal behavior for circular references. They just need an explicit or implicit call to gc.collect().

If I use:

from astropy.table import Table, Column
import psutil
import gc
import numpy as np

c = Column(np.arange(1e7), name='a')
for i in range(20):
    print(psutil.virtual_memory())
    t = Table([c])
    gc.collect()   # <-- explicit call
    print(psutil.virtual_memory())

it works fine. Nevertheless reference cycles are annoying because the memory-free "time" isn't predictable anymore (according to the Python data model no object deletion is predictable but in practice it's just reference cycles that aren't) so I definitely support using a weakref instead. :)

But it isn't "critical" or a "bug" in my view. The current behavior is just "really annoying".

@taldcroft

I re-opened this because of #6342 ... But @taldcroft, feel free to close this and open a separate issue if that's your preference

Can be closed since #6277 was merged again (in #7448).

Navigation Menu

Repository navigation

Memory leak in Table related to ref cycle in info attribute #6276

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Memory leak in Table related to ref cycle in info attribute #6276

Description

Activity

taldcroft commented on Jun 25, 2017

MSeifert04 commented on Jun 25, 2017

eteq commented on Jul 6, 2017

saimn commented on Jun 22, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions