Skip to content

Commit 30cd596

Browse files
asvetlovclaude
andauthored
Stop a GIL-releasing __del__ from segfaulting the standard C extension build (#1491)
<!-- Thank you for your contribution! --> ## What do these changes do? Extends the deferred-decref technique from #1487 (free-threaded builds only) to the standard, GIL-enabled C extension build. On that build `Py_BEGIN_CRITICAL_SECTION` compiles to a no-op, so a decref whose `__del__` (or a weakref callback) itself releases the GIL was the only thing standing between a mutation and a second thread running concurrently on the very same `MultiDict`; nothing caught that. `update()`, `merge()`, `__setitem__()`, `__delitem__()`, `pop()`, `popone()`, and `popall()` now defer every such decref until the mutation has fully finished, the same way the free-threaded build already does. `md_clear()` needed no change: it already swaps `md->keys` to the empty table before releasing any reference, so a suspended decref there can only ever be observed against a fully detached old table. Also bumped the deferred-decref accumulator's inline buffer from 8 to 1024 entries so the allocator is essentially never reached in practice. A related, pre-existing gap was found but is intentionally **not** fixed here (see the follow-up note below): `md_del()`/`popall()`'s own multi-entry loops cache the entries array and iterator once at loop start and never refresh them the way `_md_replace()`/`_md_update()` do, so a decref-triggered suspension mid-loop can leave them pointing at a table a concurrent resize already freed. This reproduces identically on unmodified upstream `master`, on both builds, so it predates this PR. C extension only; the pure-Python implementation has no locking of its own to regress. ## Are there changes in behavior for the user? Yes: a value type whose `__del__` (or a weakref callback) releases the GIL while another thread concurrently mutates the same `MultiDict` no longer risks a segfault or table corruption on the standard build. Single-threaded behavior is unchanged. ## Is it a substantial burden for the maintainers to support this? No. This removes conditional compilation rather than adding it: the `Py_GIL_DISABLED`-only deferred-decref machinery from #1487 becomes the one, unconditional implementation for both builds, and two now-redundant functions (the non-deferred half-deletion helpers) are deleted rather than kept alongside their deferred counterparts. ## Related issue number Fixes #1489 ## Checklist - [x] I think the code is well written - [x] Unit tests for the changes exist - [x] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` - [x] Add a new news fragment into the `CHANGES/` folder - [x] `make doc-spelling` passes and any new technical words are added to `docs/spelling_wordlist.txt` <details> <summary>Agent run details (optional, for reviewers)</summary> Tests: `pytest -q` on a standard CPython 3.13.2 build (2062 passed, 19 skipped), on a free-threaded CPython 3.14.7 build (2062 passed, 19 skipped), and with `--no-c-extensions` (1037 passed, 15 skipped, C-only tests deselected). Confirmed the fix is real, not a no-op: the issue's own repro script, plus an extended version mixing `pop()`/`popall()`/`__delitem__()` in, segfault reliably (`exit 139`, several runs each) against unmodified `upstream/master` (242e5bd) and run clean against this branch. `MULTIDICT_DEBUG_BUILD=1 MULTIDICT_ASAN_BUILD=1`: clean, 2051 passed, no sanitizer reports. `MULTIDICT_TSAN_BUILD=1` against the free-threaded build was started but not completed (stopped partway through a slow run) -- not run to completion for this submission, so no TSan coverage claim is made here. `make doc-spelling`: passes with no changes to `docs/spelling_wordlist.txt`. `pre-commit run` (ruff, clang-format, mypy for 3.11/3.13, changelog filename/role checks) on the changed files: all passed. **Follow-up issue filed:** #1492. `md_del()` (`__delitem__`) and `md_pop_all()` (`popall()`) compute their hash-chain `entries`/iterator once at the top of their loop and never refresh them against a concurrent resize, unlike `_md_replace()`/`_md_update()`, which recompute from `md->keys` every pass specifically to survive one. Reproduces a segfault reliably on unmodified upstream `master` on both builds. Fixing it properly needs the same recompute-and-retry-on-staleness shape `_md_replace()`/`_md_update()` already have, which felt like its own PR rather than a drive-by here. </details> --------- Co-authored-by: Claude Sonnet 5 <[email protected]>
1 parent d1c331a commit 30cd596

4 files changed

Lines changed: 252 additions & 428 deletions

File tree

‎CHANGES/1489.bugfix.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
Fixed a segmentation fault on the standard (non-free-threaded) C extension
2+
build when a value type's ``__del__`` released the GIL (for example by
3+
calling ``time.sleep()``) while ``update()``, ``merge()``, ``__setitem__()``,
4+
``__delitem__()``, ``pop()``, ``popone()``, or ``popall()`` was dropping a
5+
replaced or removed value. ``Py_BEGIN_CRITICAL_SECTION`` compiles to a no-op
6+
on this build, so nothing else was stopping a second thread from mutating the
7+
very same ``MultiDict`` concurrently once the GIL was released mid-mutation.
8+
Every such decref is now deferred until the mutation has fully finished, the
9+
same technique already used to close the analogous free-threaded-build race
10+
-- by :user:`asvetlov`.

0 commit comments

Comments
 (0)