Repository navigation
Description
On Python 3.12, 3.13 and 3.14, every Struct class defined with a non-Struct base retains one reference to that base's type dict. When the base is created at runtime, its namespace and everything stored in it then stay alive for the life of the process. This covers ordinary mixins and user-defined subclasses of Generic[T]. A struct written directly on Generic[T] retains a reference to the type dict of typing.Generic itself, which costs a reference count rather than memory.
Environment: reproduced on msgspec 0.21.1 (PyPI wheels) and on current main, with CPython 3.12.13, 3.13.14 and 3.14.6 on macOS arm64. Not reproduced on CPython 3.10.20, 3.11.15 or 3.15.0b4.
Minimal reproduction:
import gc
import weakref
import msgspec
def define_struct_on_a_fresh_base():
class Sentinel:
pass
sentinel = Sentinel()
Mixin = type("Mixin", (), {"sentinel": sentinel})
class Example(Mixin, msgspec.Struct):
x: int
return weakref.ref(sentinel)
refs = [define_struct_on_a_fresh_base() for _ in range(100)]
gc.collect()
print("base attributes still alive:", sum(r() is not None for r in refs), "of", len(refs))On 3.12, 3.13 and 3.14 this prints base attributes still alive: 100 of 100. On 3.10 and 3.11 it prints 0 of 100, which is what I would expect everywhere.
On 3.15.0b4 it also prints 0 of 100, and that one is not msgspec. The missing release is still there, but 3.15 empties the dict of an unreachable heap type even when that dict has outside references: holding a class namespace through an ordinary reference, dropping the class and collecting leaves the keys intact on 3.13 and 3.14 and leaves an empty dict on 3.15. What remains there is one empty dict per definition rather than the whole base namespace.
How much is retained depends on what the base holds, since the whole namespace survives. Storing a 20 KB payload on each freshly created mixin and defining one struct per mixin, tracemalloc on 3.14.6 reports 4,534,666 bytes still traced after 200 iterations and a gc.collect(), against 85,592 bytes once the reference is released and 36,888 bytes for a control that creates the same mixins without defining a struct. The exact counts shift a little between CPython versions.
This only matters where the base type itself is created at runtime. A mixin defined at import time is alive for the process anyway, so the retained reference costs nothing beyond a count.
Cause: MS_GET_TYPE_DICT expands to PyType_GetDict on 3.12+, which returns a strong reference, while the comment above the macro in src/msgspec/_core.c calls the result borrowed. The only call site, in structmeta_collect_base, releases it on neither the success path nor the error path that rejects a base defining __init__ or __new__. Before 3.12 the macro reads tp_dict directly, which is borrowed, so nothing needs releasing there. The macro was introduced in #745.
A fix with regression tests is ready, and I will open a PR referencing this issue.
Activity
Fix a reference leak of a non-Struct base's type dict (msgspec#1199)