Repository navigation
Description
Every successful Struct class definition that carries annotations retains one reference to the annotations dict built for that class. The dict and everything it references stay alive for the life of the process, even after the struct type itself has been collected. A class that carries a string annotation beginning ClassVar or typing.ClassVar, with or without a subscript, additionally retains one reference to the __dict__ of the module it was defined in.
This matters only where struct types are created repeatedly, for example one type per request or per schema, or in a long-running process that builds types in a loop. A fixed set of types defined at import time retains a bounded amount of memory.
Environment: reproduced on msgspec 0.21.1 (PyPI wheels) with CPython 3.11.15 and 3.14.6 on macOS arm64, and on current main.
Minimal reproduction:
import gc
import tracemalloc
import weakref
import msgspec
def define_struct_type():
class Annotation:
pass
class Example(msgspec.Struct):
x: Annotation
return weakref.ref(Annotation)
refs = [define_struct_type() for _ in range(100)]
gc.collect()
print("annotation types still alive:", sum(r() is not None for r in refs), "of", len(refs))
def grow(n):
gc.collect()
tracemalloc.start()
for i in range(n):
msgspec.defstruct(f"Temp{i}", [("a", int), ("b", str), ("c", float)])
gc.collect()
current, _ = tracemalloc.get_traced_memory()
tracemalloc.stop()
return current
print("traced memory retained after 10000 struct definitions:", grow(10_000), "bytes")Output on 0.21.1 and on current main, CPython 3.11.15 (the exact byte count varies slightly between Python versions):
annotation types still alive: 100 of 100
traced memory retained after 10000 struct definitions: 1844656 bytes
The same script with a plain class, a dataclass, a NamedTuple or a TypedDict in place of the struct prints 0 of 100, and that is what I would expect here. A struct with no annotations at all does not retain anything.
Cause: in structmeta_collect_fields (src/msgspec/_core.c) both annotations and module_ns are owned references that are released only on the failure paths. The success path returns without releasing either. The annotations half dates from #852, which made the reference owned but added the matching Py_DECREF(annotations) only on the failure paths; before that the reference was borrowed. The module_ns half has been there since #281 added ClassVar support.
A fix with regression tests is ready, and I will open a PR referencing this issue.
Activity
Fix reference leak when defining a Struct type (msgspec#1194)