Skip to content

Commit 6d3443f

Browse files
Siyetsobolevn
andauthored
Fix a reference leak of a non-Struct base's type dict (#1199)
On Python 3.12+ `MS_GET_TYPE_DICT` expands to `PyType_GetDict`, which returns a new reference; in the comment above the macro the result is described as borrowed. At its only call site, in `structmeta_collect_base`, there is no matching release on either exit, so defining a `Struct` with a non-`Struct` base leaks one reference to that base's type dict. Nothing holds that reference afterwards, so the dict is never freed, and on 3.12 through 3.14 that keeps the base class itself alive along with everything in its namespace, for the rest of the process. `typing.Generic` counts as such a base, so generic `Struct` definitions leak one too, in both the explicit and the PEP 695 spelling; there the dict stays reachable regardless, so only the count grows. Adds a release macro and applies it on both exits. It expands to `Py_XDECREF` on 3.12+ and to nothing before that, where the `tp_dict` slot is read directly and the reference is borrowed. Corrects the comment to match. The remaining `tp_dict` uses in the file are direct slot reads guarded against `NULL` and need no release. A base that has not been readied yet, which a C extension can expose, has no type dict, and on main the `__init__` and `__new__` check crashes on such a base on every supported version. Such a base is now rejected with `TypeError` before its type dict or inherited slots are read, and `TypeError` is also raised if a readied base still has no type dict. The check that the base is a type now comes first. Adds a regression test that the base's contents are released when the base passes the `__init__` and `__new__` check. On current main it fails on 3.12, 3.13 and 3.14, and passes on 3.10 and 3.11, where the leak does not exist. It also passes on main on 3.15, where the leaked dict no longer keeps the base class reachable, so the class is collected and its contents are released even though the dict itself is still retained. Fixes #1198. --------- Co-authored-by: Tseluiko Aleksandr <[email protected]> Co-authored-by: sobolevn <[email protected]>
1 parent 2f0b50d commit 6d3443f

2 files changed

Lines changed: 54 additions & 10 deletions

File tree

‎src/msgspec/_core.c‎

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -54,12 +54,16 @@ ms_popcount(uint64_t i) { \
5454
}
5555
#endif
5656

57-
/* In Python 3.12+, tp_dict is NULL for some core types, PyType_GetDict returns
58-
* a borrowed reference to the interpreter or cls mapping */
57+
/* In Python 3.12+, tp_dict is NULL for some core types, so PyType_GetDict is
58+
* used instead. It returns a new reference, which must be released; before
59+
* 3.12 the tp_dict slot is read directly and is borrowed, so there is nothing
60+
* to release. */
5961
#if PY312_PLUS
6062
#define MS_GET_TYPE_DICT(a) PyType_GetDict(a)
63+
#define MS_RELEASE_TYPE_DICT(d) Py_XDECREF(d)
6164
#else
6265
#define MS_GET_TYPE_DICT(a) ((a)->tp_dict)
66+
#define MS_RELEASE_TYPE_DICT(d) ((void)(d))
6367
#endif
6468

6569
#if PY313_PLUS
@@ -5809,8 +5813,25 @@ structmeta_get_module_ns(MsgspecState *mod, StructMetaInfo *info) {
58095813

58105814
static int
58115815
structmeta_collect_base(StructMetaInfo *info, MsgspecState *mod, PyObject *base) {
5816+
if (!PyType_Check(base)) {
5817+
/* CPython's metaclass conflict check will catch this issue earlier on,
5818+
* but it's still good to have this check in place in case that's ever
5819+
* removed */
5820+
PyErr_SetString(PyExc_TypeError, "All base classes must be types");
5821+
return -1;
5822+
}
5823+
58125824
if ((PyTypeObject *)base == &StructMixinType) return 0;
58135825

5826+
/* A base that has not been readied yet, which a C extension can expose,
5827+
* has neither its type dict nor its inherited slots filled in. Readying
5828+
* a type owned by another extension can have side effects, so such a
5829+
* base is rejected instead. */
5830+
if (!PyType_HasFeature((PyTypeObject *)base, Py_TPFLAGS_READY)) {
5831+
PyErr_Format(PyExc_TypeError, "Base class %R is not ready", base);
5832+
return -1;
5833+
}
5834+
58145835
if (((PyTypeObject *)base)->tp_weaklistoffset) {
58155836
info->already_has_weakref = true;
58165837
}
@@ -5819,14 +5840,6 @@ structmeta_collect_base(StructMetaInfo *info, MsgspecState *mod, PyObject *base)
58195840
info->already_has_dict = true;
58205841
}
58215842

5822-
if (!PyType_Check(base)) {
5823-
/* CPython's metaclass conflict check will catch this issue earlier on,
5824-
* but it's still good to have this check in place in case that's ever
5825-
* removed */
5826-
PyErr_SetString(PyExc_TypeError, "All base classes must be types");
5827-
return -1;
5828-
}
5829-
58305843
if (!ms_is_struct_cls(base)) {
58315844
if (((PyTypeObject *)base)->tp_dictoffset) {
58325845
info->has_non_slots_bases = true;
@@ -5835,12 +5848,22 @@ structmeta_collect_base(StructMetaInfo *info, MsgspecState *mod, PyObject *base)
58355848
static const char *attrs[] = {"__init__", "__new__"};
58365849
Py_ssize_t nattrs = 2;
58375850
PyObject *tp_dict = MS_GET_TYPE_DICT((PyTypeObject *)base);
5851+
if (tp_dict == NULL) {
5852+
PyErr_Format(
5853+
PyExc_TypeError,
5854+
"Cannot read the attributes of base class %R",
5855+
base
5856+
);
5857+
return -1;
5858+
}
58385859
for (Py_ssize_t i = 0; i < nattrs; i++) {
58395860
if (PyDict_GetItemString(tp_dict, attrs[i]) != NULL) {
58405861
PyErr_Format(PyExc_TypeError, "Struct base classes cannot define %s", attrs[i]);
5862+
MS_RELEASE_TYPE_DICT(tp_dict);
58415863
return -1;
58425864
}
58435865
}
5866+
MS_RELEASE_TYPE_DICT(tp_dict);
58445867
return 0;
58455868
}
58465869

‎tests/unit/test_struct.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1009,6 +1009,27 @@ def define_in_temp_module():
10091009
assert [ref() for ref in refs] == [None, None, None]
10101010

10111011

1012+
def test_struct_definition_does_not_leak_non_struct_base_dict():
1013+
"""Defining a struct must not retain the type dict of a non-struct base"""
1014+
1015+
class Sentinel:
1016+
pass
1017+
1018+
def define_struct_type():
1019+
sentinel = Sentinel()
1020+
Mixin = type("Mixin", (), {"sentinel": sentinel})
1021+
1022+
class Example(Mixin, Struct):
1023+
x: int
1024+
1025+
return weakref.ref(sentinel)
1026+
1027+
refs = [define_struct_type() for _ in range(3)]
1028+
gc.collect()
1029+
1030+
assert [ref() for ref in refs] == [None, None, None]
1031+
1032+
10121033
def test_struct_gc_not_added_if_not_needed():
10131034
"""Structs aren't tracked by GC until/unless they reference a container type"""
10141035

0 commit comments

Comments
 (0)