bpo-31558: Add gc.freeze() #3705

brainfvck · 2017-09-23T01:43:10Z

Introduces a new API that allows for moving all objects currently tracked by the garbage collector to a permanent generation, effectively removing them from future collection events. This can be used to protect those objects from having their PyGC_Head mutated. In effect, this enables great copy-on-write stability at fork(). More details on the issue.

https://bugs.python.org/issue31558

the-knights-who-say-ni · 2017-09-23T01:43:13Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

ambv · 2017-09-23T02:01:02Z

I will add documentation, NEWS.d, and update Misc/ACKS in a separate commit.

ambv · 2017-09-23T02:06:04Z

Include/internal/mem.h

@@ -185,6 +185,7 @@ struct _gc_runtime_state {
       collections, and are awaiting to undergo a full collection for
       the first time. */
    Py_ssize_t long_lived_pending;
+    struct gc_generation permanent_generation;


I would put this field after generation0.

ambv · 2017-09-23T02:10:46Z

Modules/gcmodule.c

+}
+
+PyDoc_STRVAR(gc_get_freeze_stats__doc__,
+"get_freeze_stats() -> n\n"


Since this is not equivalent to what get_stats() is returning, can you rename to get_frozen_count()?

ambv · 2017-09-23T02:14:33Z

Modules/gcmodule.c

+);
+
+static PyObject *
+gc_freeze(PyObject *module)


We'd prefer if you used ArgumentClinic for those functions. It's very easy to use: https://docs.python.org/3/howto/clinic.html

elprans · 2017-09-26T22:56:24Z

Modules/gcmodule.c

@@ -813,6 +817,8 @@ collect(int generation, Py_ssize_t *n_collected, Py_ssize_t *n_uncollectable,
        for (i = 0; i < NUM_GENERATIONS; i++)
            PySys_FormatStderr(" %zd",
                              gc_list_size(GEN_HEAD(i)));
+        PySys_WriteStderr("\ngc: objects in permanent generation: %d",


pitrou · 2017-10-06T16:48:41Z

Lib/test/test_gc.py

@@ -734,6 +734,10 @@ def test_get_stats(self):
        self.assertEqual(new[1]["collections"], old[1]["collections"])
        self.assertEqual(new[2]["collections"], old[2]["collections"] + 1)

+    def test_freeze(self):
+        gc.freeze()


You should really undo the freezing after this test. We don't want tests to have so large side effects.

Good point! This will require adding the "thaw" or "unfreeze" function that will move objects back from the permanent generation to a properly collected one. Do you think it would be enough for that call to move all of them to gen0? This won't be exactly "undoing the freeze" but doesn't require keeping track of which object belonged to which generation before.

Anything that empties the permanent generation is good enough IMHO. People can call gc.collect explicitly afterwards if they want to collect those objects.

PS: as a non-native English locutor, I much prefer "unfreeze" to "thaw" :-)

Oh, didn't notice your "general comment" until now. So it seems we agree that moving everything to a single generation makes sense. But you're suggesting the oldest generation (gen2) instead. Makes sense.

ok, I was just too lazy to add it but fine will update..

pitrou · 2017-10-06T16:51:00Z

Modules/gcmodule.c

+
+Freeze all current tracked objects and ignore them for future collections.
+
+This can be used before a fork to make the gc copy-on-write friendly.


Rather than "a fork", perhaps use the wording "a POSIX fork() call", as that may not be obvious to all readers.

pitrou · 2017-10-06T16:53:17Z

Two general comments:

this needs to be documented in Doc/library/gc.rst
we really need a way to unfreeze the current permanent collection; in which generation it gets unfrozen doesn't really matter overall, though it may sound logical to unfreeze it into the oldest generation

pitrou · 2017-10-07T15:43:29Z

Modules/gcmodule.c

+gc_unfreeze_impl(PyObject *module)
+/*[clinic end generated code: output=1c15f2043b25e169 input=2dd52b170f4cef6c]*/
+{
+    gc_list_merge(&_PyRuntime.gc.permanent_generation.head, GEN_HEAD(NUM_GENERATIONS-1));


Do we need to update _PyRuntime.gc.permanent_generation.count after this?

No, it's always 0 since we don't use it, instead gc_list_size() is used for count.
count is not always the number of objects in generations(it is for 0 but not 1 and 2).

pitrou

The current version of the PR looks basically good to me. Thank you!

Somewater · 2018-09-11T07:59:54Z

Does fix already in 3.7.0 release?
I did some test script:
https://gist.github.com/Somewater/40d7a808d1efd7b2c77f22f8bcb73553
I expected that the script allocates 1-2 GB of memory at the beginning and does not require more memory in the process of work. But it does - just after start of iterations in workers:
http://pix.toile-libre.org/?img=1536607288.png
What I did wrong?

pitrou · 2018-09-11T08:28:18Z

@Somewater I think you misunderstood what gc.freeze allows you to do. It prevents the garbage collector from touching existing objects, but reference counting still exists, so if you iterate over those objects, you will still write in their corresponding memory areas (because you're incref'ing then decref'ing them). In short, what gc.freeze allows to share are objects that are never touched after the workers have been started (such as docstrings in modules, etc.). Not "live" objects that are touched by the application.

Somewater · 2018-09-11T08:50:55Z

Thanks for answer. But can I manage with it without full disable of GC? I have big structure in parent process memory (python's objects) and have to read it from forked childs.

pitrou · 2018-09-11T09:01:49Z

If you're actually using those objects in the children, they will be duplicated because of reference counting.

bpo-31558: Add gc.freeze()

6d17860

the-knights-who-say-ni added the CLA not signed label Sep 23, 2017

bedevere-bot added the awaiting review label Sep 23, 2017

ambv added the skip news label Sep 23, 2017

ambv requested review from vstinner, 1st1 and nascheme September 23, 2017 02:01

ambv reviewed Sep 23, 2017

View reviewed changes

elprans reviewed Sep 26, 2017

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Sep 26, 2017

address comments

fdd12ff

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Oct 1, 2017

pitrou reviewed Oct 6, 2017

View reviewed changes

fix a segfault and address comments

03f4915

pitrou reviewed Oct 7, 2017

View reviewed changes

pitrou approved these changes Oct 8, 2017

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 8, 2017

ambv merged commit c75edab into python:master Oct 16, 2017

bedevere-bot removed the awaiting merge label Oct 16, 2017

komuw mentioned this pull request Jan 2, 2018

Evaluate support for preload and monkey-patching benoitc/gunicorn#1566

Closed

z2kviswanathan mentioned this pull request Feb 6, 2018

Zenefits 2.7.6 - Backport gc.freeze(), gc.get_freeze_count() and gc.unfreeze() to Python 2.7 z2kviswanathan/cpython#1

Merged

eduardo-elizondo mentioned this pull request Apr 11, 2020

gh-84436: Implement Immortal Objects #19474

Merged

guevara mentioned this pull request Oct 14, 2024

Why not multiprocessing guevara/read-it-later#11828

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-31558: Add gc.freeze() #3705

bpo-31558: Add gc.freeze() #3705

brainfvck commented Sep 23, 2017 •

edited by ambv

Loading

the-knights-who-say-ni commented Sep 23, 2017

ambv commented Sep 23, 2017

ambv Sep 23, 2017

ambv Sep 23, 2017

ambv Sep 23, 2017

elprans Sep 26, 2017

pitrou Oct 6, 2017

ambv Oct 6, 2017

pitrou Oct 6, 2017

ambv Oct 6, 2017

brainfvck Oct 7, 2017

pitrou Oct 6, 2017

pitrou commented Oct 6, 2017

pitrou Oct 7, 2017

brainfvck Oct 7, 2017

pitrou left a comment

Somewater commented Sep 11, 2018

pitrou commented Sep 11, 2018

Somewater commented Sep 11, 2018

pitrou commented Sep 11, 2018


		Freeze all current tracked objects and ignore them for future collections.

		This can be used before a fork to make the gc copy-on-write friendly.

bpo-31558: Add gc.freeze() #3705

bpo-31558: Add gc.freeze() #3705

Conversation

brainfvck commented Sep 23, 2017 • edited by ambv Loading

the-knights-who-say-ni commented Sep 23, 2017

ambv commented Sep 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou commented Oct 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou left a comment

Choose a reason for hiding this comment

Somewater commented Sep 11, 2018

pitrou commented Sep 11, 2018

Somewater commented Sep 11, 2018

pitrou commented Sep 11, 2018

brainfvck commented Sep 23, 2017 •

edited by ambv

Loading