Skip to content

Commit b2d1b7a

Browse files
Siyetsobolevn
andauthored
Reject gc=False on struct types with a weakref slot (#1207)
On Python 3.12 and later, releasing an instance of a struct type declared with both `gc=False` and `weakref=True` goes wrong. On default builds it reads memory outside the instance's allocation, and creating one instance is enough, since the read happens even when no weak reference was ever taken: ```python import msgspec class Ex(msgspec.Struct, gc=False, weakref=True): x: int Ex(1) ``` The process usually crashes. On free-threaded builds there is no immediate crash: the instance is released from the wrong address, the allocator's own state is corrupted, and the failure arrives later and somewhere else. Since 3.12 the `__weakref__` slot is not inside the object. It sits at a fixed negative offset, in the pre-header that is allocated in front of the instance. A `gc=False` type clears `Py_TPFLAGS_HAVE_GC` and releases the instance with `PyObject_Free` on the object itself, so the block is released from the wrong address. Outside free-threaded builds that pre-header is also smaller, which leaves the slot before the start of the allocation. This rejects the combination at class creation, the way `gc=False` with `dict=True` is already rejected. The check also covers the other routes to it: `defstruct`, inheriting either option from a base class, and a `gc=False` struct whose base is a plain mixin with `__slots__ = ("__weakref__",)`. The structs guide documents the combination as unsupported, and why. The restriction is unconditional, so it also applies to 3.10 and 3.11, where the slot is inside the object and the combination works. That is deliberate: the same class definition then behaves the same way on every supported version, and an upgrade of Python cannot turn working code into an error. An earlier revision of this branch kept such a type GC enabled instead, which also fixed the crash. The discussion below records why the restriction was chosen over it. --------- Co-authored-by: Tseluiko Aleksandr <[email protected]> Co-authored-by: sobolevn <[email protected]>
1 parent f2bbec5 commit b2d1b7a

4 files changed

Lines changed: 83 additions & 0 deletions

File tree

‎docs/changelog.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,14 @@
22

33
## Version 0.22.0 (2026-09-29)
44

5+
- **BREAKING**: Setting `gc=False` on a struct type that has a weakref slot,
6+
whether from `weakref=True` or from a base class, now raises `ValueError`.
7+
On CPython 3.12 and later the weakref slot is stored in a pre-header in
8+
front of the instance, so releasing an instance of a `gc=False` type with
9+
that slot corrupted memory, in most builds crashing the interpreter
10+
outright. The combination worked on 3.10 and 3.11, and the restriction
11+
applies there as well, so that the same class definition behaves the same
12+
way on every supported version ({pr}`1207`).
513
- Add `frozendict` support on Python 3.15+ ({pr}`1052`, {pr}`1105`).
614
- Support passing a callable as `decimal_format` to `msgspec.json.Encoder` and
715
`msgspec.msgpack.Encoder` for custom `Decimal` encoding ({pr}`978`).

‎docs/structs.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1104,6 +1104,12 @@ container types. It is your responsibility to ensure cycles with these objects
11041104
don't occur, as a cycle containing only ``gc=False`` structs will *never* be
11051105
collected (leading to a memory leak).
11061106

1107+
``gc=False`` cannot be combined with a weakref slot, whether that slot comes
1108+
from ``weakref=True`` or from a base class. Setting both raises ``ValueError``
1109+
at class creation. On Python 3.12 and later such a slot is stored in a
1110+
pre-header in front of the instance, and disabling garbage collection for the
1111+
type makes releasing an instance unsafe.
1112+
11071113
.. _type annotations: https://docs.python.org/3/library/typing.html
11081114
.. _pattern matching: https://docs.python.org/3/reference/compound_stmts.html#the-match-statement
11091115
.. _PEP 636: https://peps.python.org/pep-0636/

‎src/msgspec/_core.c‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6646,6 +6646,14 @@ StructMeta_new_inner(
66466646
PyErr_SetString(PyExc_ValueError, "Cannot set gc=False and dict=True");
66476647
goto cleanup;
66486648
}
6649+
else if (info.weakref == OPT_TRUE || info.already_has_weakref) {
6650+
PyErr_SetString(
6651+
PyExc_ValueError,
6652+
"Cannot set gc=False and weakref=True at the same time, due to "
6653+
"possible memory corruption"
6654+
);
6655+
goto cleanup;
6656+
}
66496657
}
66506658

66516659
/* Collect new fields and defaults */

‎tests/unit/test_struct.py‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1115,6 +1115,67 @@ class Base(Struct, gc=False):
11151115
class Test(Base, **opts):
11161116
pass
11171117

1118+
@staticmethod
1119+
def assert_gc_false_with_weakref_rejected(base, **opts):
1120+
with pytest.raises(ValueError, match="Cannot set gc=False and weakref=True"):
1121+
1122+
class Test(base, **opts):
1123+
pass
1124+
1125+
def test_struct_gc_false_forbids_weakref_true(self):
1126+
self.assert_gc_false_with_weakref_rejected(Struct, gc=False, weakref=True)
1127+
1128+
def test_struct_gc_false_forbids_weakref_base(self):
1129+
class Base(Struct, weakref=True):
1130+
pass
1131+
1132+
self.assert_gc_false_with_weakref_rejected(Base, gc=False)
1133+
1134+
def test_struct_gc_false_base_forbids_weakref_true(self):
1135+
class Base(Struct, gc=False):
1136+
pass
1137+
1138+
self.assert_gc_false_with_weakref_rejected(Base, weakref=True)
1139+
1140+
def test_struct_gc_false_forbids_weakref_grandparent(self):
1141+
class Grandparent(Struct, weakref=True):
1142+
pass
1143+
1144+
class Base(Grandparent):
1145+
pass
1146+
1147+
self.assert_gc_false_with_weakref_rejected(Base, gc=False)
1148+
1149+
def test_struct_gc_false_forbids_weakref_slot_from_mixin(self):
1150+
class Mixin:
1151+
__slots__ = ("__weakref__",)
1152+
1153+
class Base(Mixin, Struct):
1154+
pass
1155+
1156+
self.assert_gc_false_with_weakref_rejected(Base, gc=False)
1157+
1158+
def test_struct_gc_false_forbids_weakref_true_defstruct(self):
1159+
with pytest.raises(ValueError, match="Cannot set gc=False and weakref=True"):
1160+
defstruct("Test", ["x"], gc=False, weakref=True)
1161+
1162+
def test_struct_gc_false_and_weakref_still_allowed_apart(self):
1163+
class NoGC(Struct, gc=False):
1164+
x: list
1165+
1166+
class SubNoGC(NoGC, gc=False):
1167+
y: list
1168+
1169+
class WithWeakRef(Struct, weakref=True):
1170+
x: list
1171+
1172+
assert not gc.is_tracked(NoGC([1]))
1173+
assert not gc.is_tracked(SubNoGC([1], [2]))
1174+
1175+
obj = WithWeakRef([1])
1176+
assert gc.is_tracked(obj)
1177+
assert weakref.ref(obj)() is obj
1178+
11181179

11191180
class TestStructDealloc:
11201181
@pytest.mark.parametrize("has_gc", [False, True])

0 commit comments

Comments
 (0)