Break reference cycles when deleting TableColumn by taldcroft · Pull Request #6346 · astropy/astropy

taldcroft · 2017-07-07T00:11:00Z

@saimn @eteq @bsipocz @mhvk @astrofrog - this may be the ticket on the Table reference cycle problem. Basically just breaking the reference cycles when the internal TableColumn object is being deleted. In my testing this did the trick and I could do the following with no increment of memory:

In [2]: c = Column(np.arange(1e8), name='a')
In [3]: for ii in range(20): t = Table([c])
In [9]: for ii in range(20): t['a'] = c

Fixes #6276

astropy-bot · 2017-07-07T00:11:01Z

Hi there @taldcroft 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

I noticed the following issue with this pull request:

Changelog entry section (v2.0) inconsistent with milestone (2.0.4)

Would it be possible to fix this? Thanks!

If there are any issues with this message, please report them here.

taldcroft · 2017-07-07T00:16:35Z

Related: #6277 #6279.

taldcroft · 2017-07-07T00:31:23Z

Ugh, nothing is ever easy...

taldcroft · 2017-07-07T01:51:24Z

I don't know what is the right shouting and swearing emoji, but I give up. Now there is some weird (probably intermittent) recursive problem from the same place!

>           assert out.pformat() == ['   a       b   ',
                                     '   km          ',
                                     '-------- ------',
                                     '0.000000    foo',
                                     '0.000010    bar',
                                     '0.002000    pez',
                                     '0.003000    sez',
                                     '0.000000    foo',
                                     '1.000000    bar']
astropy/table/tests/test_operations.py:685: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
astropy/table/table.py:1181: in pformat
    tableid=tableid, tableclass=tableclass, align=align)
astropy/table/pprint.py:541: in _pformat_table
    align=align_)
astropy/table/pprint.py:254: in _pformat_col
    col_strs = list(col_strs_iter)
astropy/table/pprint.py:455: in _pformat_col_iter
    col_str = format_func(col_format, col[i])
astropy/table/pprint.py:41: in <lambda>
    else format_func(format_, val))
astropy/table/pprint.py:52: in <lambda>
    yield lambda format_, val: format_ % val
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Quantity 0.0 km>
    def __float__(self):
        try:
            return float(self.to_value(dimensionless_unscaled))
        except (UnitsError, TypeError):
>           raise TypeError('only dimensionless scalar quantities can be '
                            'converted to Python scalars')
E           TypeError: only dimensionless scalar quantities can be converted to Python scalars
astropy/units/quantity.py:1203: TypeError
============================ pytest-warning summary ============================
WC1 None [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
 1 failed, 9245 passed, 629 skipped, 95 xfailed, 3 xpassed, 1 pytest-warnings in 275.96 seconds 
Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: <bound method TableColumns.__del__ of <TableColumns names=('bool','int8','int16','int32','int64','uint8','uint16','uint32','uint64','float16','float32','float64','float128','str')>>
Traceback (most recent call last):
  File "/tmp/astropy-test-5eafmfiq/lib.linux-x86_64-3.4/astropy/table/table.py", line 134, in __del__
  File "/tmp/astropy-test-5eafmfiq/lib.linux-x86_64-3.4/astropy/utils/data_info.py", line 232, in __get__
  File "/tmp/astropy-test-5eafmfiq/lib.linux-x86_64-3.4/astropy/utils/data_info.py", line 295, in __setattr__
TypeError: must be type, not None
Exception ignored in: Exception ignored in:

astrofrog · 2017-07-07T10:15:09Z

See #6291 (comment) before proceeding - the weakref was definitely not the issue.

taldcroft · 2017-07-07T11:25:24Z

FWIW I restarted the one failing job and it passed.

bsipocz · 2017-07-07T11:27:31Z

@taldcroft - And would you like to get this in to 2.0.0, not that @astrofrog found out this wasn't the cause for the issues we saw? (I'm waiting for one more appveyor at the moment, so could merge this in, too, and then start the release procedure again).

bsipocz · 2017-12-05T15:37:41Z

@taldcroft - Could you rebase? I've remilestoned this to the next bugfix, but if it gets an approval very soon, I'm happy to include it in 2.0.3

astropy-bot · 2017-12-14T12:39:17Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 5 months. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock. You may also consider sending a reminder e-mail about it to the astropy-dev mailing list.

If you believe I commented on this pull request incorrectly, please report this here.

mhvk · 2017-12-14T13:32:44Z

@taldcroft - I actually do not remember what the status of this problem is...

taldcroft · 2017-12-14T14:09:45Z

I don't recall precisely either, but I think the comment above is where things left off: "I don't know what is the right shouting and swearing emoji, but I give up. Now there is some weird (probably intermittent) recursive problem from the same place!"

astropy-bot · 2018-01-11T12:41:18Z

⏰ Time's up! ⏰

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

Break reference cycles when deleting TableColumn

6f45bf5

taldcroft force-pushed the table-del branch from a622ad0 to 6f45bf5 Compare July 7, 2017 00:14

taldcroft added Affects-release Bug labels Jul 7, 2017

taldcroft added this to the v2.0.0 milestone Jul 7, 2017

taldcroft added the table label Jul 7, 2017

taldcroft requested a review from astrofrog July 7, 2017 00:24

Make table tests pass

a5fb4f5

taldcroft removed this from the v2.0.0 milestone Jul 7, 2017

bsipocz modified the milestones: v2.0.0, v2.0.1 Jul 7, 2017

bsipocz modified the milestones: v2.0.1, v2.0.2 Jul 28, 2017

bsipocz modified the milestones: v2.0.2, v2.0.3 Sep 7, 2017

saimn mentioned this pull request Oct 10, 2017

Table column print format cache bug #6703

Closed

bsipocz modified the milestones: v2.0.3, 2.0.4 Dec 5, 2017

astropy-bot Bot closed this Jan 11, 2018

pllim added the closed-by-bot Closed by stale bot label May 25, 2018

yymao mentioned this pull request Sep 19, 2018

Memory leak (or circular reference) when slicing Column #7831

Closed

taldcroft deleted the table-del branch October 1, 2019 20:44

Navigation Menu

Repository navigation

Break reference cycles when deleting TableColumn - #6346

Break reference cycles when deleting TableColumn#6346
taldcroft wants to merge 2 commits into
astropy:masterfrom
taldcroft:table-del

taldcroft commented Jul 7, 2017 •

edited

Loading

Uh oh!

astropy-bot Bot commented Jul 7, 2017 •

edited

Loading

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

astrofrog commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

bsipocz commented Jul 7, 2017 •

edited

Loading

Uh oh!

bsipocz commented Dec 5, 2017

Uh oh!

astropy-bot Bot commented Dec 14, 2017

Uh oh!

mhvk commented Dec 14, 2017

Uh oh!

taldcroft commented Dec 14, 2017

Uh oh!

astropy-bot Bot commented Jan 11, 2018

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

taldcroft commented Jul 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astropy-bot Bot commented Jul 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

astrofrog commented Jul 7, 2017

Uh oh!

taldcroft commented Jul 7, 2017

Uh oh!

bsipocz commented Jul 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bsipocz commented Dec 5, 2017

Uh oh!

astropy-bot Bot commented Dec 14, 2017

Uh oh!

mhvk commented Dec 14, 2017

Uh oh!

taldcroft commented Dec 14, 2017

Uh oh!

astropy-bot Bot commented Jan 11, 2018

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

taldcroft commented Jul 7, 2017 •

edited

Loading

astropy-bot Bot commented Jul 7, 2017 •

edited

Loading

bsipocz commented Jul 7, 2017 •

edited

Loading