Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save clin1234/2469758160721f4f991110167017e4d3 to your computer and use it in GitHub Desktop.

Select an option

Save clin1234/2469758160721f4f991110167017e4d3 to your computer and use it in GitHub Desktop.
Generated with https://github.com/devdanzin/cext-review-toolkit

C Extension Analysis Report

Extension: aiohttp (4 Cython extensions)

Scope: aiohttp/_http_parser.pyx, aiohttp/_http_writer.pyx, aiohttp/_websocket/mask.pyx, aiohttp/_websocket/reader_py.py (compiled as reader_c), supporting headers and build files.


Agents Run

Agent Group Mode
gil-discipline-checker A run
resource-lifecycle-checker A run
type-slot-checker B run
module-state-checker B run
version-compat-scanner C run
parity-checker D run
stable-abi-checker D run
c-complexity-analyzer E run
git-history-analyzer F run (last)
refcount-auditor — skipped (Cython mode: 95–100% FP from generator patterns)
error-path-analyzer — skipped (Cython mode)
null-safety-scanner — skipped (Cython mode)
pyerr-clear-auditor — skipped (Cython mode)

Executive Summary

aiohttp's Cython layer has correctly declared freethreading_compatible=True and removed the only previously identified global C buffer race (_http_writer.pyx, fixed 2025-09-04). Three new correctness gaps remain: the @cython.freelist allocator on both message types is non-atomic and causes use-after-free under concurrent parsers; two parity gaps in _http_parser.pyx are confirmed-incomplete fixes from recent commits (a TCP upgrade bypass and a missing pipeline-queue bound on the response parser). The complexity hotspots in the WebSocket reader and HTTP parser carry the highest probability of harboring the next latent bug, consistent with a fix-to-feature ratio of 27% observed in git history. No _PyEval_StopTheWorld usage exists anywhere in the tree; all fixes are straightforward substitutions or one-line additions.


Extension Profile

Field Value
Modules _http_parser, _http_writer, _websocket.mask, _websocket.reader_c
Code generation Cython 3.2.5
Python targets ≥ 3.10
Free-threading declared Yes — freethreading_compatible=True in Makefile (all 3 Cython rules)
Limited API No — py_limited_api absent from setup.py; PyByteArray_Resize is a blocker
Init style Single-phase (m_size=-1 in generated C)
C library dependency llhttp (vendored) via _cparser.pxd

Key Metrics

Dimension Status FIX CONSIDER Top Finding
GIL Discipline Y 1 1 @cython.freelist(250) non-atomic race on both message types
Resource Lifecycle G 0 1 Py_buffer OOM tail-slice path (HPE_PAUSED)
Type Slots G 0 1 HttpParser instantiable via __new__ without _init
Module State G 0 2 13× cdef object module globals break subinterpreter isolation
ABI Compliance G 0 1 PyByteArray_Resize blocks stable ABI; abi3 infeasible
Version Compat G 0 3 3× DEF deprecated in Cython 4
C/Python Parity Y 2 1 "tcp" absent from ALLOWED_UPGRADES; max_msg_queue_size missing from HttpResponseParser
Complexity G 0 5 _feed_data 183 LOC / CC 27 (WebSocket reader)
Refcount Safety — — — Skipped (Cython mode)
Error Handling — — — Skipped (Cython mode)
NULL Safety — — — Skipped (Cython mode)
PyErr_Clear Safety — — — Skipped (Cython mode)

G = No FIX findings · Y = 1–3 FIX findings · R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 3

# Finding File:Line Severity Agents
1 @cython.freelist(250) on RawRequestMessage and RawResponseMessage — module-global C counter and object-pointer array written non-atomically on every alloc/dealloc; concurrent parsers (e.g. two feed_data calls on the same connection from different threads) cause use-after-free _http_parser.pyx:129,229 CRITICAL gil-discipline-checker, git-history-analyzer
2 ALLOWED_UPGRADES = frozenset({"websocket"}) — "tcp" missing; Python parser's _is_supported_upgrade() accepts {"tcp", "websocket"} (fixed there in commit 8f23712616, March 2024); Cython parser silently rejects TCP upgrades 27 months later _http_parser.pyx:51,484 HIGH parity-checker, git-history-analyzer
3 HttpResponseParser.__init__ does not accept max_msg_queue_size and does not forward it to self._init() — pipelined response queue is unbounded; HttpRequestParser received the parameter in commit 93a2b1c36 but HttpResponseParser was not updated (confirmed twin-class omission) _http_parser.pyx:752–762 HIGH parity-checker, git-history-analyzer

Should Consider (CONSIDER) — 15

# Finding File:Line Severity
4 data[pos:] slice at line 650 can raise MemoryError after llhttp_execute returns HPE_PAUSED; if it does, control jumps out before PyBuffer_Release at line 652 — buffer stays exported and the backing object cannot be resized or freed _http_parser.pyx:648–652 MEDIUM
5 HttpParser and its Cython subclasses have __cinit__ (allocates C structs) but _init is a separate Python-level call; constructing via HttpParser.__new__(HttpResponseParser) without calling __init__ leaves _cparser allocated but unconfigured; _init is never a slot, so tp_init is a no-op on HttpParser itself _http_parser.pyx:343–418 MEDIUM
6 12× cdef object module-level globals in _http_parser.pyx (lines 62–74: EMPTY_PAYLOAD, NO_EXTENSIONS, etc.) and _istr in _http_writer.pyx (line 12) are file-scope C statics (static PyObject*); the module was compiled with m_size=-1, so each subinterpreter shares the same static cells — subinterpreter isolation is broken _http_parser.pyx:62–74, _http_writer.pyx:12 MEDIUM
7 cdef list _http_method populated at module init by a for loop — a mutable list as a module-scope static; mutating it post-init would silently corrupt method-name lookup for all parsers _http_parser.pyx:105–109 LOW
8 DEF BUF_SIZE = 16 * 1024 in _http_writer.pyx — Cython DEF compile-time constants are deprecated since Cython 3.0 and removed in Cython 4; this DEF also feeds a C array dimension, which will need a const or enum replacement before migration _http_writer.pyx:10 LOW
9 DEF DEFAULT_FREELIST_SIZE = 250 — same deprecation class as Finding 8; feeds the @cython.freelist argument that is also the target of FIX Finding 1 _http_parser.pyx:52 LOW
10 DEF METHODS_COUNT = 46 — same deprecation class; feeds the range in _http_method construction and must stay in sync with the llhttp method enum _http_parser.pyx:103 LOW
11 _write_utf8 silently drops lone surrogates (return 0 at line 72 on \ud800–\udfff) while the pure-Python serializer raises UnicodeEncodeError; clients using the C writer get silent data truncation instead of an error _http_writer.pyx:70–72 MEDIUM
12 PyByteArray_Resize and PyByteArray_GET_SIZE (non-limited API) are called directly from the generated Cython C; py_limited_api is absent from setup.py, so no abi3 claim is made — but if adoption is ever attempted, these two calls are the primary blocker alongside a buffer-protocol Py_buffer floor conflict with Python 3.10 setup.py, _http_parser.pyx LOW
13 No cpXXXt-tagged (free-threaded) wheels are published to PyPI — cibuildwheel has no [tool.cibuildwheel.freethreading] section in pyproject.toml; users on CPython 3.13t/3.14t receive a GIL-compatible wheel that silently re-enables the GIL pyproject.toml (absent) LOW
14 WebSocketReader._feed_data: 183 LOC, cyclomatic complexity 27 — highest CC in the extension layer; recent commit 69344c6ef reorganized the max_msg_size check location, adding a new branch cluster that raised CC _websocket/reader_py.py:332–515 —
15 WebSocketReader._handle_frame: 136 LOC, CC 21 — interleaved opcode dispatch, masking, and continuation-frame logic; fragmented-mask bug was fixed here (commit e0cc020d5, April 2025) at the cost of additional branch depth _websocket/reader_py.py:194–330 —
16 HttpParser.feed_data: 78 LOC, CC 16 — llhttp dispatch + OOM guard + tail logic; commit b5a517076 added buffer-pointer caching here; the HPE_PAUSED slice-raise gap (Finding 4) is in this function _http_parser.pyx:602–680 —
17 HttpParser._on_headers_complete: 68 LOC, CC 14 — highest-churn function in git history alongside feed_data (18 substantive changes in 18 months); upgrade handling (FIX Finding 2) lives here _http_parser.pyx:466–534 —
18 HttpParser._on_status_complete: 47 LOC, nesting depth 5 — response-status dispatch with inlined payload-size accounting _http_parser.pyx:700–747 —

Tensions

  • Module state vs. migration cost (_http_parser.pyx:62–74): module-state-checker flags the 12 cdef object globals as subinterpreter-isolation failures; c-complexity-analyzer notes that migrating them to a PyModule_GetState struct requires threading the module pointer through every callback that currently reads them as file-statics. This is a multi-week refactor; the subinterpreter use case is still rare for aiohttp. Deferring is rational — but the Py_mod_multiple_interpreters slot should be set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED so CPython rejects the mismatch explicitly.

  • Stable ABI infeasibility (stable-abi-checker): PyByteArray_Resize has no limited-API equivalent; adopting abi3 would require a runtime-check workaround or replacement with bytes accumulation. The complexity cost is not justified given aiohttp's standard wheel publishing workflow. Recommendation is to explicitly reject abi3 rather than leave the door open.


Policy Decisions (POLICY) — 7

# Finding
19 Change cdef list _http_method to cdef tuple — matches immutability intent; http_method_str() only reads it; no post-init mutation needed
20 Change ALLOWED_CLOSE_CODES: Final[set[int]] to frozenset — Final does not prevent runtime mutation; frozenset is lock-free under free-threading and matches ALLOWED_UPGRADES style
21 Latent nogil trap: llhttp_execute C callbacks (cb_on_url, cb_on_headers_complete, etc.) call back into Python from C — if a nogil block were ever introduced around llhttp_execute, these callbacks would execute without the GIL. Document this constraint and add a # NOGIL-UNSAFE comment at the llhttp_execute call site
22 Add Py_mod_multiple_interpreters slot to module init (set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED) — signals the incompatibility explicitly rather than silently misbehaving in subinterpreter contexts
23 requirements/cython.in has no upper bound (Cython with no version pin) — Cython 4 will remove DEF (Findings 8–10) and may change freelist behaviour; add Cython>=3.2,<4 until the migration is done
24 abi3 and free-threading are mutually exclusive in the current CPython ABI: do not add py_limited_api=True to setup.py — it would conflict with freethreading_compatible=True and produce incorrect wheel tags
25 cython_limited_api=True is absent from the Makefile; if abi3 is ever revisited, this flag and py_limited_api in setup.py must be set simultaneously; a half-applied change would produce a wheel that lies about its ABI compliance

Dismissed / Superseded Findings

ft-analysis.md Finding 5 — PyBuffer_Release not called on exception paths in feed_data — DISMISSED by resource-lifecycle-checker during this analysis.

The llhttp callbacks (cb_on_url, cb_on_headers_complete, etc.) catch all Python exceptions via try: ... except BaseException as ex: pyparser._last_error = ex; return -1. Because llhttp_execute has no Cython except annotation, control unconditionally falls through to PyBuffer_Release at line 652 regardless of the callback's return value. The stored exception is re-raised at line 668, which is after the release. The ft-analysis was incorrect in classifying this as UNSAFE/HIGH.

Note: Finding 4 in this report identifies a residual, narrower gap on the HPE_PAUSED tail-slice path — a distinct code location that the original ft finding did not specify.


Strengths

  • ALLOWED_UPGRADES and SINGLETON_HEADERS — cdef frozenset; immutable after module init, lock-free under free-threading; frozenset.__contains__ is safe for concurrent reads.
  • _serialize_headers buffer — switched from global cdef char BUFFER[BUF_SIZE] to stack-local allocation with a heap_allocated fallback flag (commit 945837cc2, 2025-09-04); the headline Cython FT race is gone.
  • feed_data buffer-pointer caching — base = <char*>self.py_buf.buf is captured before PyBuffer_Release; commit b5a517076 correctly fixed the use-after-release on the normal path.
  • max_line_size cumulative enforcement — both cb_on_url and cb_on_status use accumulated-length checks; commit 36df6c138 was applied completely to both callbacks in one change; no twin-class gap exists here.
  • mask.pyx — 48 lines, only 2 commits in its entire history; the XOR kernel is minimal and carries no thread-safety concerns.
  • No nogil / Py_BEGIN_ALLOW_THREADS blocks — zero instances of GIL-released regions in any .pyx file; the extension never invokes CPython APIs without the GIL.
  • No _PyEval_StopTheWorld — confirmed by full-repo grep; none of the races require global quiescence to fix.

Git History Insights

File Total commits Recent (2025–2026) Fix-to-feature ratio Risk level
_http_parser.pyx 67 18 ~27% HIGH
_websocket/reader_py.py 18 12 ~33% HIGH
_http_writer.pyx 12 5 ~17% MEDIUM
_websocket/mask.pyx 2 0 0% LOW

Notable commit history:

  • 2018, commit 28f15195c — Introduced @cython.freelist(DEFAULT_FREELIST_SIZE) on both RawRequestMessage and RawResponseMessage simultaneously; they have never been touched since; neither was audited when freethreading_compatible=True was added in 2025.
  • 2024-03, commit 8f23712616 — Added "tcp" to _is_supported_upgrade() in the Python parser, scoped explicitly to "the pure python http parser"; the Cython ALLOWED_UPGRADES frozenset was not updated and still lacks "tcp" 27 months later.
  • 2025-09-04, commit 945837cc2 — Fixed global C buffer in _http_writer.pyx and added freethreading_compatible=True to all Cython invocations; _http_parser.pyx was only touched to remove a comment line — the freelist race was not audited.
  • 2026-06 (recent), commit 93a2b1c36 — Added max_msg_queue_size to HttpRequestParser.__init__ and HttpParser._init; HttpResponseParser.__init__ was not updated (confirmed twin-class omission, trivially reproducible).
  • 2026-06, commit 36df6c138 — Enforced cumulative max_line_size on both cb_on_url and cb_on_status; fix is complete — no gap found.

The fix-to-feature ratio of ~27% across all parser files is elevated relative to a healthy extension (typically ~10–15%). It reflects active correctness pressure on the HTTP parser and suggests a systematic audit sweep is more effective than continued point fixes.


Recommended Action Plan

Immediate (FIX items)

  1. Fix Finding 1 — disable @cython.freelist for free-threading builds

    Both RawRequestMessage (line 129) and RawResponseMessage (line 229) must be addressed together — they were introduced in the same commit and carry identical risk.

    Option A — guard with a runtime check:

    import sys as _sys
    _FREELIST_SIZE = 0 if not getattr(_sys, "_is_gil_enabled", lambda: True)() else DEFAULT_FREELIST_SIZE
    
    @cython.freelist(_FREELIST_SIZE)
    cdef class RawRequestMessage:
        ...

    Option B — remove both decorators entirely and file a tracking issue to restore them only for GIL-enabled builds once Cython's freelist-disable-on-FT path is confirmed in the CI build environment.

  2. Fix Finding 2 — add "tcp" to ALLOWED_UPGRADES

    # _http_parser.pyx line 51:
    cdef frozenset ALLOWED_UPGRADES = frozenset({"websocket", "tcp"})

    One-character change; mirrors the Python parser fix from commit 8f23712616.

  3. Fix Finding 3 — add max_msg_queue_size to HttpResponseParser.__init__

    Mirror HttpRequestParser.__init__ exactly:

    # _http_parser.pyx HttpResponseParser.__init__ (line 752):
    def __init__(
        self, protocol, loop, int limit, timer=None,
            size_t max_line_size=8190, size_t max_headers=128,
            size_t max_field_size=8190, payload_exception=None,
            bint response_with_body=True, bint read_until_eof=False,
            bint auto_decompress=True,
            Py_ssize_t max_msg_queue_size=0    # ADD THIS
    ):
        self._init(cparser.HTTP_RESPONSE, protocol, loop, limit, timer,
                   max_line_size, max_headers, max_field_size,
                   payload_exception, response_with_body, read_until_eof,
                   auto_decompress,
                   max_msg_queue_size)          # FORWARD IT

Short-term (CONSIDER items)

  1. Fix Finding 4 — wrap HPE_PAUSED tail-slice in try/finally

    # _http_parser.pyx feed_data (around line 634):
    PyObject_GetBuffer(data, &self.py_buf, PyBUF_SIMPLE)
    try:
        base = <char*>self.py_buf.buf
        data_len = <size_t>self.py_buf.len
        errno = cparser.llhttp_execute(self._cparser, base, data_len)
        # ... nb / pos / tail logic including data[pos:] slice ...
    finally:
        PyBuffer_Release(&self.py_buf)
    # raise ex follows the finally block unchanged
  2. Fix Finding 5 — document or guard HttpParser.__new__-without-__init__

    The simplest guard is to check self._cparser is not NULL at the top of every public method that calls into llhttp, raising RuntimeError("parser not initialized") if the check fails. Alternatively, initialize _cparser to NULL in __cinit__ and call a safe-default _init from __cinit__ with sentinel values that block feed_data from doing anything.

  3. Fix Finding 11 — align surrogate behaviour with the Python writer

    In _write_utf8, replace the silent return 0 on surrogates with a Cython raise:

    # _http_writer.pyx line 72:
    raise UnicodeEncodeError("utf-8", ...)

    Or pass errors="surrogatepass" in the serialiser and document the intentional deviation. Currently the silence is undocumented and breaks parity.

  4. Fix Findings 8–10 — migrate DEF constants before Cython 4 upgrade

    # Replace:
    DEF BUF_SIZE = 16 * 1024
    DEF DEFAULT_FREELIST_SIZE = 250
    DEF METHODS_COUNT = 46
    
    # With:
    BUF_SIZE = 16 * 1024                    # module-level Python int
    DEFAULT_FREELIST_SIZE = 250
    METHODS_COUNT = 46

    For BUF_SIZE feeding a C array dimension, use cdef enum or a cdef int constant depending on how the C array is declared in the generated code.

  5. Fix Finding 13 — publish free-threaded wheels to PyPI

    Add to pyproject.toml:

    [tool.cibuildwheel.freethreading]
    enable = true

    Also add the Programming Language :: Python :: Free Threading :: 3 - Alpha PyPI classifier once FIX findings 1–3 above are resolved.

Longer-term (POLICY / structural)

  1. Policy 19 — harden _http_method to tuple

    cdef tuple _http_method = tuple(
        cparser.llhttp_method_name(<cparser.llhttp_method_t>i).decode("ascii")
        for i in range(METHODS_COUNT)
    )
  2. Policy 20 — convert ALLOWED_CLOSE_CODES to frozenset

    ALLOWED_CLOSE_CODES: Final[frozenset[int]] = frozenset(int(i) for i in WSCloseCode)
  3. Policy 22 — add Py_mod_multiple_interpreters slot

    In the generated module init (or via a Cython cython: ... module_state directive when available), set m_slots to include:

    {Py_mod_multiple_interpreters, Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED}
  4. Policy 23 — cap Cython version in requirements

    # requirements/cython.in:
    Cython>=3.2,<4
    
  5. Finding 6 — migrate module-level globals to per-module state (long-term)

    All 12 cdef object globals in _http_parser.pyx (lines 62–74) and _istr in _http_writer.pyx (line 12) should migrate to a PyModule_GetState-backed struct. This requires m_size > 0 and threading the module pointer through every init callback. Scope this as a standalone PR; it does not block the FIX items above.


Notes

  • Free-threaded interpreter on this system: python3t.exe (python3t -m pip for package management)
  • The corresponding free-threading analysis (asyncio layer, cookie jar, connector pool, WebSocket writer lock gap) is in ft-analysis.md
  • For a phased migration plan: /ft-review-toolkit:plan or /cext-review-toolkit:plan
  • The ft-analysis.md Finding 5 (PyBuffer_Release on exception) is DISMISSED per the resource-lifecycle-checker findings in this report — do not file it as an issue

C Extension Analysis Report

Extension: _cffi_backend

Scope: src/c/ (+ src/cffi/ for parity check)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker,

resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor,

stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer,

git-history-analyzer

Analysis Date: 2026-06-14


Executive Summary

_cffi_backend is a 12,000-line hand-written C extension that forms the core of the cffi library. The extension is mature and well-structured: GIL discipline is correct, closure lifecycle is clean, and the builder_c_t ownership pattern handles FFI type allocations consistently. The free-threading path was audited separately (see FREE_THREADING_ANALYSIS.md).

Five version-compatibility issues require immediate attention: the PyUnicode_KIND/DATA/READ family is scheduled for removal in CPython 3.14 and will break compilation if cffi supports 3.14; four private struct fields (_PyLong_Sign, gilstate_counter, _status.bound_gilstate, _PyErr_WriteUnraisableMsg) carry no stability guarantee.

One type-slot bug (ffi_traverse omitting init_once_cache) creates uncollectable reference cycles. Two confirmed per-call reference leaks in lib_obj.c (realized C type descriptors leaked on lib attribute access failures) are the most impactful correctness bugs.

The ctypes pure-Python fallback (backend_ctypes.py) is explicitly unsupported but has documented behavioral gaps; integer_const is the one gap reachable through the public API.


Extension Profile

  • Module: _cffi_backend (~25 C files, 8,028 lines main / ~12,000 lines total)
  • Init style: single-phase (PyInit__cffi_backend)
  • Python targets: ≥ 3.9
  • Limited API: no (backend); yes for generated user modules via _cffi_include.h (correct split)
  • Types defined: 13 static PyTypeObject instances
  • Code generation: hand-written

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety 🟡 2 3 ct leaked on 3 error paths in lib_build_and_cache_attr
Error Handling 🟢 0 2 Module-init m/ffi leaked on error paths
NULL Safety 🟡 1 2 mb_ass_item missing NULL guard — crash on del buf[n]
GIL Discipline 🟢 0 2 dlopen/dlsym/dlclose called with GIL held
Resource Lifecycle 🟡 0 2 PyMem_Malloc data buffer never freed in constant case
Module State 🟡 0 5 13 static types + m_size=-1 block sub-interpreter support
Type Slots 🟡 1 3 ffi_traverse missing init_once_cache
PyErr_Clear Safety 🟢 0 3 lib_getattr unguarded PyErr_Clear
ABI Compliance 🟢 0 5 Limited API not feasible for _cffi_backend; correct for generated modules
Version Compat 🔴 5 9 PyUnicode_KIND/DATA/READ removed in 3.14 (12+ sites)
C/Python Parity 🟢 0 5 integer_const missing from api.py FFI class
Complexity 🟢 — 4 b_complete_struct_or_union_lock_held CC≈45, 371 lines

🟢 No FIX findings | 🟡 1–3 FIX findings | 🔴 4+ FIX findings


Findings by Priority

Global non-restarting numbering: FIX (1–9), CONSIDER (10–33), POLICY (34–39).


Must Fix (FIX) — 8

# Finding File:Line Agents
1 ct (CTypeDescrObject) leaked on 3 error paths in lib_build_and_cache_attr _CFFI_OP_CONSTANT branch lib_obj.c:306–342 refcount-auditor, error-path-analyzer
2 ct leaked when cdlopen_fetch fails in _CFFI_OP_GLOBAL_VAR branch of lib_build_and_cache_attr lib_obj.c:358–375 refcount-auditor, error-path-analyzer
3 PyUnicode_KIND, PyUnicode_DATA, PyUnicode_READ, PyUnicode_4BYTE_DATA, PyUnicode_FromKindAndData deprecated in 3.12, removed in 3.14 — 12+ sites wchar_helper_3.h:17,32,38,39,67,87,95,97,99,119,120,124 version-compat-scanner
4 _PyLong_Sign private API used unguarded — public PyLong_GetSign available in 3.13+ _cffi_backend.c:878, 3964 version-compat-scanner
5 _PyErr_WriteUnraisableMsg private API in #else branch (covers 3.9–3.12) _cffi_backend.c:6167–6171 version-compat-scanner
6 tstate->gilstate_counter direct private struct access — no public API equivalent misc_thread_common.h:227, 350, 372 version-compat-scanner
7 tstate->_status.bound_gilstate private struct field, active under >= 3.12 guard misc_thread_common.h:179 version-compat-scanner
8 ffi_traverse omits init_once_cache → uncollectable reference cycles when ffi.init_once() holds back-reference to FFI object ffi_obj.c:78–85 type-slot-checker
9 mb_ass_item (sequence slot) missing other == NULL guard — commit c5212d08 fixed mb_ass_subscript (mapping slot) but left mb_ass_item unpatched; del buf[n] crashes via PyBytes_Check(NULL) minibuffer.h:40 git-history-analyzer

Finding 1+2 Detail

realize_c_type() returns a new reference stored in ct. The success path correctly calls Py_DECREF(ct) before returning, but three early-return error paths in the _CFFI_OP_CONSTANT branch exit without decrementing (size check failure, cdlopen_fetch failure, PyMem_Malloc failure). Similarly in _CFFI_OP_GLOBAL_VAR, the cdlopen_fetch failure returns without releasing ct. These are hit on every failed lib.symbol attribute access — not just module init.

// lib_obj.c CONSTANT branch — fix:
if (ct->ct_size <= 0) {
    PyErr_Format(FFIError, ...);
    Py_DECREF(ct);    /* ADD */
    return NULL;
}
if (g->address == NULL) {
    data = cdlopen_fetch(...);
    if (data == NULL) {
        Py_DECREF(ct);    /* ADD */
        return NULL;
    }
}
data = PyMem_Malloc(ct->ct_size);
if (data == NULL) {
    PyErr_NoMemory();
    Py_DECREF(ct);    /* ADD */
    return NULL;
}

Finding 3 Detail

cffi's wchar_helper_3.h uses the entire deprecated PyUnicode_KIND/DATA family for char16/char32 ↔ Python unicode conversion. These macros were deprecated in 3.12 and are scheduled for removal in 3.14. Since cffi targets 3.14 (per classifiers), this will break compilation.

Replacement strategy:

  • Read path: Use PyUnicode_AsUCS4 / PyUnicode_AsUCS4Copy (stable)
  • Write path: Use PyUnicode_New + PyUnicode_WriteChar (stable)
  • PyUnicode_FromKindAndData: Replace with PyUnicode_New + per-character PyUnicode_WriteChar

The guard at _cffi_backend.c:382–386 (#ifdef PyUnicode_KIND) should also be removed — it is always true on Python ≥ 3.9, so wchar_helper.h (the pre-3.3 path) is dead code.

Finding 9 Detail

Commit c5212d08 ("do not crash on __delitem__") added value == NULL guards to the mapping slot (mp_ass_subscript/mb_ass_subscript) but the sequence slot (sq_ass_item/mb_ass_item) was not updated. CPython routes del buf[n] through sq_ass_item with other=NULL. mb_ass_item immediately calls PyBytes_Check(other) — a NULL dereference.

// minibuffer.h:40 — fix:
static int mb_ass_item(MiniBufferObj *self, Py_ssize_t idx, PyObject *other)
{
    if (other == NULL) {           /* ADD: sequence protocol delete */
        PyErr_SetString(PyExc_TypeError,
            "minibuffer doesn't support item deletion");
        return -1;
    }
    if (idx < 0 || idx >= self->mb_size) { ... }
    if (PyBytes_Check(other) && PyBytes_GET_SIZE(other) == 1) {

Finding 8 Detail

FFIObject has init_once_cache (a PyDict mapping arbitrary tag objects to (True, func_result) tuples, set by ffi.init_once()). This dict is Py_XDECREF'd in ffi_dealloc (line 70) but is not visited in ffi_traverse. If user code passes an object referencing the FFI itself as a tag (or if func_result holds a back-reference), the GC cannot detect or break the cycle.

// ffi_obj.c ffi_traverse — fix:
static int ffi_traverse(FFIObject *ffi, visitproc visit, void *arg)
{
    Py_VISIT(ffi->types_builder.types_dict);
    Py_VISIT(ffi->types_builder.included_ffis);
    Py_VISIT(ffi->types_builder.included_libs);
    Py_VISIT(ffi->gc_wrefs);
    Py_VISIT(ffi->init_once_cache);    /* ADD */
    return 0;
}

Should Consider (CONSIDER) — 23

Resource Management

# Finding File:Line
10 data = PyMem_Malloc(ct->ct_size) in _CFFI_OP_CONSTANT branch is never freed, even on success — bounded leak (once per constant per first access) lib_obj.c:340–346
11 Module-init leaks: v (Unicode/capsule) not freed on PyModule_AddObject failure; tpo not freed on type-loop failure; m/ffi not freed in b_init_cffi_1_0_external_module — error-path-only, fatal paths _cffi_backend.c:7965–7989, cffi1_module.c:184–191

Finding 10 is distinct from Findings 1+2: even when realize_c_type succeeds and the constant is converted to a Python object, the intermediate data buffer is allocated via PyMem_Malloc but never freed. The convert_to_object call consumes its own copy; data should be freed after:

data = PyMem_Malloc(ct->ct_size);
// ...fill data...
x = convert_to_object(data, ct);
PyMem_Free(data);   /* ADD */

Finding 11 is module-init-only (runs once per interpreter, failure is fatal), so CONSIDER rather than FIX. Prefer PyModule_AddObjectRef (3.10+) to eliminate the steal-only-on-success ambiguity.

NULL Safety

# Finding File:Line
12 PyDict_GetItem in _get_interpstate_dict silently treats internal dict errors as "key not found" — use PyDict_GetItemWithError call_python.c:42
13 PyDict_GetItem in _update_cache_to_call_python treats internal errors as "not registered" with no exception propagation call_python.c:162

Both are non-crashing. Finding 12 fix:

d = PyDict_GetItemWithError(interpdict, attr_name);
if (d == NULL) {
    if (PyErr_Occurred()) goto error;
    /* key not found — create new dict */
    d = PyDict_New();
    // ...
}

GIL Discipline

# Finding File:Line
14 dlsym() called with GIL held — will block all Python threads during symbol resolution on slow filesystems/NFS cdlopen.c:15
15 dlclose() called with GIL held — same concern cdlopen.c:27, 32
16 Missing Py_IsInitialized() check before gil_ensure() in invoke_callback — undefined behavior if callback fires during interpreter teardown _cffi_backend.c:6289, call_python.c:249

The dlopen call at cdlopen.c already uses Py_BEGIN_ALLOW_THREADS. The dlsym/dlclose calls should match. Finding 16: add if (!Py_IsInitialized()) { /* write error to stderr; return */ } before the GIL-acquire path.

PyErr_Clear Safety

# Finding File:Line
17 lib_getattr calls PyErr_Clear() without checking PyErr_ExceptionMatches(PyExc_AttributeError) — swallows arbitrary exceptions including MemoryError lib_obj.c:528, 532, 536, 547, 552
18 call_python.c unguarded PyErr_Clear() in two places — same concern call_python.c:55, 180

Pattern to apply:

// Before:
PyErr_Clear();
// After:
if (PyErr_ExceptionMatches(PyExc_AttributeError))
    PyErr_Clear();
// (or check for the specific expected exception)

Version Compatibility

# Finding File:Line
19 PyErr_Fetch, PyErr_Restore, PyErr_NormalizeException deprecated in 3.12 — 8 sites; replacements: PyErr_GetRaisedException / PyErr_SetRaisedException (3.12+) _cffi_backend.c:2038, 2046, 2058, 6165, 6244, 6252, 6253, 6273
20 PyModule_AddObject soft-deprecated in 3.10; steals ref only on success causing the ref-leak pattern at Finding 11 — replace with PyModule_AddObjectRef _cffi_backend.c:7965–7989, cffi1_module.c:186, 190
21 #include "structmember.h" deprecated standalone header; all symbols available from Python.h on 3.12+ _cffi_backend.c:3
22 Dead #ifdef PyUnicode_KIND guard — always true on 3.9+; wchar_helper.h (pre-3.3 path) is dead code _cffi_backend.c:382–386
23 Dead Py_SET_REFCNT compat shim — guard < 0x030900a4 is always false on 3.9+; symbol never used in the file _cffi_backend.c:145–147
24 PyDict_GetItemRef shim uses <= while PyWeakref_GetRef shim uses < for the same version threshold 0x030d00a1 — inconsistency, recommend both use < 0x030d0000 _cffi_backend.c:149

Type Slot Correctness

# Finding File:Line
25 FFI_Type and Lib_Type have Py_TPFLAGS_HAVE_GC + tp_traverse but no tp_clear — GC can detect cycles but not break them ffi_obj.c:1162, lib_obj.c:612
26 _keepalive1/_keepalive2 fields in builder_c_t (embedded in FFIObject) are Py_XDECREF'd in ffi_dealloc but not visited in ffi_traverse realize_c_type.c:14–15, ffi_obj.c:78–85
27 CField_Type holds owned cf_type (CTypeDescrObject*) but has no Py_TPFLAGS_HAVE_GC / tp_traverse — breaks GC cycle detection for recursive struct types (e.g., struct Node { struct Node *next; }) _cffi_backend.c:790–820

C/Python Parity

# Finding File:Line
28 ffi.integer_const(name) exists in the C FFI but is absent from api.py's FFI class — raises AttributeError through the normal API path ffi_obj.c:879, src/cffi/api.py
29 backend_ctypes.py: callback(onerror=...) uses assert onerror is None — onerror silently ignored under python -O; should raise NotImplementedError explicitly src/cffi/backend_ctypes.py:1005–1007
30 backend_ctypes.py: sizeof, alignof, new_function_type use assert for type validation — wrong exception type (AssertionError vs TypeError) and silent failure under -O src/cffi/backend_ctypes.py:863, 990, 994

Complexity and Maintainability

# Finding File:Line
31 b_complete_struct_or_union_lock_held: 371 lines, CC≈45 — two interleaved bitfield algorithms (GCC + MSVC) in one function body; extract _layout_gcc_bitfield() and _layout_msvc_bitfield() _cffi_backend.c:5149–5520
32 LIB_GET_OR_CACHE_ADDR macro contains a goto error into the calling function's scope — hidden control flow at call sites; convert to inline function lib_obj.c:443–452
33 ALIGNMENT_CHECK macro contains hidden return NULL — same concern; inline explicitly _cffi_backend.c (used in b_unpack)
34 Magic number 512 in cdata_call alloca threshold (undocumented); 1000 recursion guard in realize_c_type_or_func_lock_held — name both _cffi_backend.c:~3175, realize_c_type.c:~708

Git History

# Finding File:Line
35 ct_under_construction flag is reset unconditionally on error in do_realize_lazy_struct_lock_held, but ct_lazy_field_list flag is NOT cleared — leaves type in inconsistent state where ct_lazy_field_list=1 but ct_under_construction=0; concurrent callers will retry indefinitely if the failure is deterministic realize_c_type.c:875–888

Tensions

Module init leaks vs. intentional immortalization (cffi1_module.c:185): Py_XINCREF(ffi) is intentional — it immortalizes the FFI object to prevent the static context_s from becoming dangling. Cleanup on PyModule_AddObject failure must account for this: releasing ffi correctly requires two Py_DECREF calls (one for the intentional immortal ref, one for the original). The error path cleanup is complicated by design.

gilstate_counter (Finding 6) vs. no public API: The ThreadCanary mechanism that keeps PyThreadState alive across C callbacks (see misc_thread_common.h:36–45) requires direct access to gilstate_counter. There is no public equivalent. The correct long-term fix is filing a CPython issue to add PyThreadState_IncrementGilstateCounter(). Short-term: add a compile-time assertion on the field's offset.

ctypes fallback parity (Findings 27–29): The pure-Python ctypes backend (backend_ctypes.py) is documented as unsupported. Full parity is not a goal. Finding 27 (integer_const) is the only gap reachable through the normal api.py FFI class. Findings 28–29 affect only callers who explicitly import and use CTypesBackend.


Policy Decisions (POLICY) — 6

# Finding
36 m_size = -1 (single-phase init) blocks isolated sub-interpreter loading — migration to multi-phase init (m_size = sizeof(module_state)) is a large refactor requiring all 13 static types to become heap types
37 13 static PyTypeObject instances are the primary blockers for both limited API adoption and multi-phase init — converting each requires PyType_Spec + PyType_FromModuleAndSpec
38 Limited API (Py_LIMITED_API) is not feasible for _cffi_backend due to static types, PyMemberDef usage, and internal macros. The architectural split (C backend = no limited API; generated user modules = Py_LIMITED_API via _cffi_include.h) is correct and intentional
39 Unity build: _cffi_backend.c #includes cffi1_module.c which #includes 8 .c files — a deliberate choice for inlining and build simplicity. Should be documented in a comment at the top of each included .c file
40 PyModuleDef dynamically allocated and leaked in cffi1_module.c:117–139 — intentional by design (the module cannot be unloaded; the allocation is bounded to one per interpreter)
41 wchar_t raises NotImplementedError in backend_ctypes.py:343–344 — intentional given the ctypes backend's unsupported status

Strengths

  • GIL discipline is correct throughout: libffi calls use Py_BEGIN_ALLOW_THREADS; mutex acquisition for PyThread_acquire_lock is wrapped in Py_BEGIN/END_ALLOW_THREADS; all callback GIL acquisition is via PyGILState_Ensure/Release.
  • Closure lifecycle is clean: cffi_closure_alloc error paths free the mmap'd memory; cffi_closure_free is always called from CDataOwningGC_Type dealloc; pool allocator is correct.
  • builder_c_t ownership pattern handles all sub-allocations in FFIObject consistently via free_builder_c.
  • dlopen handle lifecycle is correctly tracked via l_auto_close + lib_dealloc + error labels in lib_internal_new.
  • PyWeakref_GetRef and PyDict_GetItemRef compat shims are present and correctly structured.
  • new_*_type functions all have correct goto error cleanup; b_complete_struct_or_union_lock_held finally: label is sound.
  • ffi_cdef / type realization error paths are clean; _cffi_parse_info_s output buffer is always freed.
  • PyObject_Init(NULL, ...) pattern is the canonical CPython idiom — not UB as it might appear; CPython handles NULL gracefully and the == NULL check catches it.
  • suppressions_free_threading.txt and conftest.py GIL re-enable detection show active free-threading CI investment.
  • Generated user modules correctly use Py_LIMITED_API via _cffi_include.h — independent of the backend's internal API usage.

Code Removal Opportunities

Based on deprecated_apis.json and the declared minimum Python version (3.9):

PyModule_AddObjectRef (added 3.10)

Replaces the Py_INCREF(obj) + PyModule_AddObject(m, name, obj) + error-path Py_DECREF(obj) pattern. cffi has 5 call sites using PyModule_AddObject across _cffi_backend.c and cffi1_module.c. Switching would eliminate the steal-on-success-only ambiguity and fix the ref-leak pattern at Finding 10.

Savings estimate: ~3 lines per site × 5 sites = 15 lines removed (and bug fix at no cost). Guard needed for 3.9: #if PY_VERSION_HEX >= 0x030A0000.

Dead compat code (min version 3.9+)

  • Py_SET_REFCNT shim at _cffi_backend.c:145–147: 3 lines (also never called — safe to delete)
  • #ifdef PyUnicode_KIND guard at _cffi_backend.c:382–386 + wchar_helper.h (dead file): 5 lines + entire file
  • Outdated Python 2/3.x<3.9 comment block in get_current_ts() (misc_thread_common.h:316–330): 15 lines

PyDict_GetItemRef (added 3.13, shim at _cffi_backend.c:150)

The shim is already present; sites using PyDict_GetItemWithError + Py_INCREF pattern could be simplified. Not a priority given the shim works correctly.

PyImport_ImportModuleAttrString (added 3.14)

Replaces PyImport_ImportModule + PyObject_GetAttrString + Py_DECREF(module) in file_emulator.h:9 and cffi1_module.c:85. Only applicable when minimum is raised to 3.14.


Recommended Action Plan

Immediate — before 3.14 support (FIX items)

  1. Fix Finding 3 — Replace PyUnicode_KIND/DATA/READ family in wchar_helper_3.h before 3.14 headers land. Use PyUnicode_New + PyUnicode_WriteChar for write path, PyUnicode_AsUCS4 for read path. Also remove the dead wchar_helper.h include guard at _cffi_backend.c:382–386.

  2. Fix Finding 9 — Add other == NULL guard to mb_ass_item in minibuffer.h:40. Two-line fix; prevents crash on del buf[n].

  3. Fix Finding 8 — Add Py_VISIT(ffi->init_once_cache) to ffi_traverse in ffi_obj.c:83. One-line fix, prevents real memory leaks.

  4. Fix Findings 1+2 — Add three Py_DECREF(ct) calls in lib_build_and_cache_attr (constant branch, lib_obj.c:314, 321, 342) and one in the global-var branch (line 375). Also add PyMem_Free(data) after convert_to_object (Finding 10).

  5. Fix Finding 4 — Add PyLong_GetSign compat shim guarded < 0x030D0000, replace _PyLong_Sign at lines 878 and 3964.

  6. Fix Findings 5, 6, 7 — For _PyErr_WriteUnraisableMsg: add #ifdef with safe fallback for stripped builds. For gilstate_counter and _status.bound_gilstate: add compile-time offset assertions and file CPython issues requesting public APIs.

Short-term (CONSIDER items — correctness/safety)

  1. Finding 11 — Replace PyModule_AddObject with PyModule_AddObjectRef (gate on 3.10) at all 5 call sites. Fixes both the deprecation (Finding 20) and the error-path leak.

  2. Findings 17+18 — Guard PyErr_Clear() calls in lib_getattr with PyErr_ExceptionMatches(PyExc_AttributeError) check. Same for call_python.c:55, 180.

  3. Finding 16 — Add Py_IsInitialized() guard in invoke_callback and cffi_call_python before gil_ensure().

  4. Findings 14+15 — Wrap dlsym()/dlclose() calls in Py_BEGIN/END_ALLOW_THREADS to match dlopen().

  5. Finding 25 — Add tp_clear to FFI_Type and Lib_Type to allow GC to break cycles.

  6. Findings 22–24 — Remove dead compat code (Py_SET_REFCNT shim, PyUnicode_KIND guard). Fix structmember.h include. Fix PyDict_GetItemRef shim boundary condition.

  7. Finding 28 — Add integer_const(name) to api.py's FFI class wrapping self._parser._int_constants[name].

  8. Finding 35 — Investigate ct_lazy_field_list flag not cleared on error in do_realize_lazy_struct_lock_held; determine if deterministic failures can cause infinite retry under concurrent access.

Longer-term (POLICY items)

  1. Findings 36+37 — Multi-phase init + heap type migration is a major undertaking. Not recommended unless subinterpreter support becomes a requirement. Prerequisite: audit all uses of static type objects across the 13 types.

  2. Finding 31 — Refactor b_complete_struct_or_union_lock_held by extracting GCC and MSVC bitfield layout algorithms into separate helpers. Reduces 371 lines to ~150 lines with independent testability.

  3. Finding 32 — Convert LIB_GET_OR_CACHE_ADDR macro with hidden goto into an inline function.


Cross-Reference: Free-Threading Analysis

See FREE_THREADING_ANALYSIS.md for the 12 thread-safety findings (6 RACE, 2 UNSAFE, 1 PROTECT, 3 MIGRATE). The most critical overlap:

  • RACE: externpy->reserved1 pre-GIL read in cffi_call_python (call_python.c:249) — highest severity
  • UNSAFE: dlerror() static buffer in misc_win32.h — also an UNSAFE finding
  • PROTECT: do_realize_lazy_struct re-acquires CFFI_LOCK while outer lock held (suspended, not deadlocked under CPython critical sections)
  • MIGRATE: gilstate_counter (Finding 6 above) — additionally a MIGRATE finding in the FT analysis

Generated by cext-review-toolkit — all 13 agents completed. For guidance on sharing findings with maintainers, see WORKING_WITH_MAINTAINERS.md.

C Extension Analysis Report — cuda-python

Toolkit: cext-review-toolkit (all scanners executed with python3t.exe = CPython 3.15.0b3, free-threaded) Branch: 315-support  |  Date: 2026-07-19 Companion report: FT_THREAD_SAFETY_ANALYSIS.md (free-threading deep-dive, same day) — thread-safety findings are cross-referenced, not re-counted. Sharing note: internal review artifact. Before filing issues/PRs from it, agree with maintainers on shape and scope (see the toolkit's WORKING_WITH_MAINTAINERS.md).

Extension Profile

  • Modules: cuda.core + cuda.bindings (+ pure-Python cuda.pathfinder)
  • Code generation: mixed — ~99 % Cython (cuda_bindings largely generator-produced .pyx; cuda_core hand-written .pyx) over a small hand-written C/C++ core
  • Hand-written C/C++ analyzed (5 files, ~3,250 lines): cuda_core/cuda/core/_cpp/resource_handles.cpp (1,819 L) / .hpp (987 L), tensor_map.cpp (154 L), tensor_map_cccl.h, cuda_bindings/cuda/bindings/_lib/param_packer.h (146 L)
  • Init style: Cython-generated module init (no hand-written PyInit_*); multi-phase per Cython defaults; freethreading_compatible=True in both build hooks
  • Python targets: >=3.10 (both packages); free-threaded 3.15t wheels built on this branch
  • Limited API / abi3: not claimed (correct — see POLICY 7)
  • Types defined in hand-written C: 0 (all classes are Cython cdef class)
  • Type stubs: 45 .pyi files in cuda_core

Stages Run

Stage How it ran Result
Phase 0 discovery (discover_extension.py) python3t.exe mixed codegen, 1 raw C-API TU, 45 stubs
Phase 0.5 external tools (run_external_tools.py) python3t.exe; cppcheck ✓, clang-tidy present (no compile_commands.json) 5 findings → 4 triaged FP, 1 tool error
10 C scanners (refcounts, error-paths, null, GIL, resources, type-slots, module-state, pyerr-clear, version-compat, format-strings) python3t.exe, both hand-written targets _cpp/: 0 across the board; param_packer.h: 42 raw hits → deduped below
5 Cython scanners (scan_cython_*) python3t.exe — no-op: tree-sitter-cython wheel fails to build on 3.15.0b3 free-threaded replaced by targeted manual .pyx review (capsules, buffer protocol, __dealloc__, except+ discipline)
measure_c_complexity.py python3t.exe 152 fns, avg CC 1.8, 0 hotspots
analyze_history.py + manual git forensics python3t.exe fix-pattern & completeness analysis (#2162, #835, #2215, #2371)
context7 verification Cython docs (/cython/cython), free-threading guide except + semantics; PyMutex guidance

Execution note: stages were executed directly (scripts + manual triage) rather than via 13 subagent dispatches — the previous session's subagents were killed mid-run by a platform session limit, and per the "rerun all interrupted stages" instruction the pipeline was completed in-line with every stage's JSON persisted to the session scratchpad. The interrupted ft-history agent's partial output (113 KB JSON) was recovered from the scratchpad and folded in.

Executive Summary

The hand-written C/C++ core of cuda-python splits sharply in quality. cuda.core's resource_handles.cpp/.hpp is modern, disciplined, low-complexity C++ — locked registries, RAII GIL guards, except+-guarded Cython boundaries, unregister-before-destroy ordering that already anticipates the pointer-reuse race fixed on the Cython side in #2162 — with only one design-contract inconsistency worth attention (the _ref creators). cuda.bindings' 2021-era param_packer.h, by contrast, carries the project's two must-fix defects: a C++ exception escaping through a non-except+ Cython boundary (guaranteed std::terminate) and the free-threading data races detailed in the companion FT report — both live in shipped 3.15t wheels because the modules declare Py_MOD_GIL_NOT_USED. Complexity, NULL safety, version compatibility, and error-path hygiene elsewhere are clean. The dominant residual risk is the unscanned Cython layer (tooling gap) and the maintainers' own acknowledged TSan gap.

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety 🟡 0* 1* ctypes_module strong ref never released (folded into Finding 2 remediation)
Error Handling 🟡 1 0 feed() extern boundary not except +; C++ throw → std::terminate
NULL Safety 🟢 0 0 accessors uniformly guard h ? … : default
GIL Discipline 🟡 1† 0 † = FT-report races (Finding 2, cross-ref); GILReleaseGuard discipline itself is strong
Resource Lifecycle 🟢 0 0 deleters verified: unregister-before-destroy + GIL guards
Module State 🟡 0* 0 37 scanner hits on param_packer.h statics — all dedupe into Finding 2
Type Slots 🟢 0 0 no hand-written type objects
PyErr_Clear Safety 🟢 0 0 zero PyErr_Clear in hand-written code
ABI Compliance 🟢 0 0 abi3 not claimed; per-version wheels incl. 3.15t (POLICY 7)
Version Compat 🟢 0 0 only live 3.13-boundary guards (resource_handles.hpp:835/856)
C/Python Parity 🟡 0 1 Buffer.__buffer__ advertised but raises NotImplementedError (WIP)
Complexity 🟢 – 0 avg CC 1.8, max nesting 3, zero hotspots (top: CC 9)

🟢 = no FIX   🟡 = 1–3 FIX (or FIX-adjacent cross-ref)   * = deduplicated into Finding 2

Findings by Priority

(Global numbering; continues across sections.)

Must Fix (FIX) — 2

# Finding File:Line Agents/Evidence
1 C++ exception across a non-except+ Cython boundary → std::terminate. param_packer.pxd declares int feed(void*, object, object) with no except +; Cython docs (verified via context7): extern cdef functions are implicitly noexcept — no try/catch is generated. Yet fetch_ctypes() (called from feed() on first use) deliberately throws std::runtime_error when import ctypes or its dict lookup fails (param_packer.h:35,39), and m_feeders[...]/std::function construction can throw std::bad_alloc. Any such throw unwinds into Cython-generated frames with no handler → std::terminate() → hard interpreter abort (during e.g. interpreter finalization, embedded/restricted envs, or OOM) instead of a Python exception. The project's own _resource_handles.pxd:173,225 shows the correct in-repo idiom (except+ nogil). Fix (1 line): declare feed as except + — or make feed noexcept by catching in C++ and returning -1. Note: on the throw path the pending ImportError from PyImport_ImportModule is left set; with except + translation it would be replaced by RuntimeError — prefer catching in C++ and letting the original Python error propagate. param_packer.pxd:7; param_packer.h:33-39,62,70 error-path scanner (adjacent hit) + manual + context7 /cython/cython
2 param_packer.h shared-state races under free-threading (live in shipped 3.15t wheels): unsynchronized std::map m_feeders read/write and racy lazy-init of 18 static ctypes globals, plus the never-released ctypes_module strong ref and long-term caching of PyDict_GetItemString borrowed refs. This run's scanners re-confirmed via 37 scan_module_state hits + 3 scan_gil_usage hits + 1 scan_refcounts potential_leak (:33) — all deduplicated into this one finding. Full analysis, interleavings, and the PyMutex-vs-std::mutex deadlock discussion: FT report Findings 1–3. Fix: populate the ctypes pointers and the (finite, 6-pair) feeder table once at module import; feed()'s existing size == 0 ctypes fallback (utils.pxi:91-95) makes prepopulation behavior-neutral. param_packer.h:11-29,62,133-138 module-state (37) + gil (3) + refcounts (1) scanners + FT report

Should Consider (CONSIDER) — 4

# Finding File:Line
3 _ref creator asymmetry breaks the registry design contract on the miss path. create_context_handle_ref and create_stream_handle_ref register their boxes (with unregistering deleters); create_event_handle_ref and create_kernel_handle_ref look up but never register, and their miss-path boxes are dependency-less: KernelBox{kernel, {}} drops the library dependency REGISTRY_DESIGN.md says KernelBox exists to carry; EventBox{event, false,false,false,-1, {}} drops the context dependency and zeroes timing/IPC metadata. Reachable from graph round-trips (graph/_subclasses.pyx:121,511,549): after the original owned handle dies, a round-tripped event misreports timing_enabled=false, and a round-tripped kernel no longer keeps its library loaded. Mitigations found: registry-hit path preserves everything; Kernel.from_handle (_module.pyx:538-556) explicitly accepts mod to re-supply library lifetime — the API anticipates the gap. Recommendation: align the four creators (either register miss-boxes for identity dedup, or document why events/kernels deliberately don't cache degraded boxes) and state the degraded-metadata contract in REGISTRY_DESIGN.md. resource_handles.cpp:769-775, 1262-1268 vs :431-447, :622-635
4 Tautological PyCapsule validation in the driver-API capsule extraction. PyCapsule_GetPointer(capsule, PyCapsule_GetName(capsule)) passes the capsule's own name as the expected name — the check can never fail, so a cuda_core wheel paired with an ABI-drifted cuda_bindings wheel casts a mismatched function pointer silently (call-time UB instead of import-time error). DESIGN.md documents the convenience rationale ("query it rather than hardcoding signatures"), so this is a known trade-off — but comparing against the expected signature string (even just in debug builds) would turn silent drift into a clean import failure. Same pattern at the cynvrtc/cynvvm/cynvjitlink extraction sites. _resource_handles.pyx:352-353, 358-361, 472-473, 489-490, 506+
5 Advertised-but-unimplemented surfaces (parity). Buffer.__buffer__ / __release_buffer__ raise NotImplementedError("WIP…") — defining the slots makes memoryview(buf) fail with NotImplementedError rather than TypeError, and stubs/docs imply buffer support; also _memoryview.copy_from (:446), a texture path (texture/_texture.pyx:521), managed-memory ops on unsupported platforms, and win32 virtual memory (_virtual_memory_resource.py:135). Track as explicit WIP or remove the slots until implemented. _memory/_buffer.pyx:436,440 et al.
6 Maintainer-flagged open threads from #2162 (commit message, Sebastian Berg): (a) SMResource split-mutex — fix added self._split_mutex but the author records residual uncertainty ("This is thread-unsafe. But I am honestly not sure…") about CUDA-internal safety; (b) explicit TSan gap: "Testing with thread-sanitizer might flush out some…"; (c) cuda-pathfinder's load_nvidia_dynamic_lib functools.cache flagged for review. These are the maintainers' own TODOs — carrying them here so they don't get lost. _device_resources.pyx:498-505; pathfinder

Tensions

  • Register-on-miss vs. don't-cache-degraded-boxes (Finding 3): registering event/kernel miss-boxes would restore identity-map dedup but would cache wrong metadata that later lookups would trust; not registering (current behavior) creates a fresh degraded box per round-trip but leaves the registry clean for a future owned registration (map_[key] = h overwrites). Neither is strictly better — which is exactly why the choice should be written down; today the code silently disagrees with REGISTRY_DESIGN.md's stated purpose.
  • Capsule convenience vs. ABI safety (Finding 4): the tautological name check is what lets cuda_core avoid hardcoding cydriver's signature strings; fixing it costs a maintained signature table. Debug-only validation is the compromise.
  • cppcheck style vs. C++ idiom: cppcheck wants constructors on EventBox (4 warnings); the codebase's aggregate-init-only style is safer than a constructor here (verified all 3 sites fully initialize). Dismissed rather than "fixed."

Policy Decisions (POLICY) — 3

# Finding
7 abi3/Limited API is not claimed and should not be pursued now. The Cython layer + per-version wheels (incl. cp315t) are incompatible with today's limited API anyway (free-threaded builds have no stable ABI yet); revisit only if wheel-matrix cost forces it.
8 Subinterpreters are unsupported by design. Process-wide C++ singletons (six HandleRegistry instances, DeferredCleanupQueue, driver fn-pointer capsules) are shared across interpreters; per-interpreter isolation would require moving them into per-module state. No action unless subinterpreter support becomes a goal.
9 Code-removal opportunities: none at the current >=3.10 floor. When the floor reaches 3.13/3.14: PyDict_GetItemRef (3.13) and PyImport_ImportModuleAttrString (3.14) replace exactly param_packer.h's PyImport_ImportModule + PyDict_GetItemString-borrowed-ref chain (~5-8 lines saved and the borrowed-ref hazard of Finding 2 eliminated structurally). PyModule_AddType (3.10) is already available but the project defines no hand-written types.

Triaged False Positives (transparency)

Raw finding Verdict
cppcheck uninitMemberVarNoCtor ×4 (EventBox members, :704-707) FP — all 3 construction sites use full aggregate init (:752, :773, :786)
scan_error_paths missing_null_check on fetch_ctypes (:33) FP as stated (NULL check exists at :34 via throw) — but the throw itself escalated into Finding 1
cppcheck internalAstError (param_packer.h:62) tool parse limitation on the nested-template std::map declaration; no code defect implied
scan_module_state "19 static_type_object" labels mislabel — cached pointers to ctypes' types, not user static PyTypeObjects (see FT report dedup note)

Strengths

  • except+ nogil discipline on every C++ handle-API declaration in _resource_handles.pxd — the exact pattern Finding 1 asks param_packer.pxd to adopt.
  • Unregister-before-destroy deleter ordering in all owned-handle deleters — already immune to the CUDA pointer-reuse race that #2162 had to fix in the Cython Level-2 registry (_graph_node.pyx). Cross-layer fix-completeness: ✓.
  • Locked registries (all six), thread-local error state, RAII GIL guards with finalization handling, error-return-not-exception C++ API contract — see FT report's SAFE-patterns list.
  • Very low complexity: 152 functions, avg CC 1.8, zero hotspots; the single CC-9 function (cuda_core_cccl_make_tma_descriptor_tiled) is stateless and fully exception-guarded.
  • Documentation quality: DESIGN.md (static-init/GIL deadlock hazard, GIL management, deferred cleanup) and REGISTRY_DESIGN.md explain why — rare and valuable.
  • History hygiene: resource_handles.{cpp,hpp} + _resource_handles.pyx co-change 8/8 commits (atomic cross-layer edits); generated cuda_bindings symbol tables made thread-safe wholesale in #835; Cython accessor critical_section guards added in #2215.

Recommended Action Plan

Immediate (FIX)

  1. Finding 1 — add except + to feed in param_packer.pxd (or catch in C++ and return -1, preserving the pending Python error). One line; crash-class payoff.
  2. Finding 2 — restructure param_packer.h to import-time initialization (single change eliminates the race, the lazy-init race, the ref leak, and the borrowed-ref caching; details in FT report Recommendations 1-3).

Short-term (CONSIDER)

  1. Finding 3 — align the four _ref creators and document the miss-path metadata contract in REGISTRY_DESIGN.md.
  2. Finding 6 — schedule the maintainers' own open items: TSan pass (/ft-review-toolkit:explore . stress-test generates a starting script), split-mutex question, load_nvidia_dynamic_lib review.
  3. Finding 4 — add (debug-mode) capsule signature validation.
  4. Finding 5 — resolve or de-advertise the WIP buffer-protocol slots before 3.15 GA.

Longer-term (POLICY)

  1. Finding 9 — when the Python floor reaches 3.13+, adopt PyDict_GetItemRef / PyImport_ImportModuleAttrString in param_packer (or delete param_packer outright: the utils.pxi fallback already handles every case it covers, just slower — measure whether the fast path still pays its complexity rent).
  2. Findings 7/8 — record the abi3 and subinterpreter non-goals in SECURITY.md/docs so future contributors don't relitigate them.

Coverage Limitations

  • The Cython layer (~99 % of compiled code) was not scanner-analyzed: tree-sitter-cython fails to build against free-threaded 3.15.0b3, so all five scan_cython_* scanners no-op'd. Manual review substituted for the highest-value targets (capsule extraction, buffer protocol, __dealloc__ discipline, except+ usage, #2162 fix-pattern siblings), but a systematic Cython audit remains open — rerun the Cython scanners once the grammar builds (or under a GIL-build python3), and see the FT report for the free-threading-specific Cython review recommendation.
  • Generated cuda_bindings .pyx (cydriver/cyruntime et al.) treated as generated code per mixed-codegen policy; its thread-safety pass (#835) noted but not re-audited.
  • benchmarks/, vendored _include/ headers, and test binaries excluded as out of scope.

C Extension Analysis Report — Cython Utility Templates

Date: 2026-06-19
Scope: Cython/Utility/ (28 hand-written C template files, ~12 000 lines)
Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, c-complexity-analyzer, git-history-analyzer (12/13 — parity-checker skipped: no dual C/Python implementations)
Phase reports: reports/cext_v1/ directory

Note on code generation: discover_extension.py reported code_generation: "cython" because this repo IS Cython. The files analysed (Cython/Utility/*.c) are hand-written C templates embedded by the compiler into generated output — not generated code themselves. Cython-mode agent suppression was therefore overridden; all agents ran at full sensitivity.


Executive Summary

The Cython/Utility/ template layer is a mature, well-structured codebase. The majority of its ~355 functions are short, focused, and correctly handle error paths. However, five confirmed bugs and a cluster of systematic PyErr_Clear discipline gaps were found that require attention before a free-threading production release. The highest-impact bug is a reference leak in __Pyx_PEP560_update_bases (ObjectHandling.c) that affects all Cython modules using PEP 560 __mro_entries__, introduced when the function was ported from CPython's bltinmodule.c. A GIL leak in __Pyx_TraceReturnCValue (Profile.c) causes deadlock under CYTHON_TRACE_NOGIL when a BaseException is raised during a trace event. The missing Py_tp_clear registration on all four coroutine/generator/async-generator type specs (Coroutine.c, AsyncGen.c) means GC cannot break reference cycles through those types — a latent memory leak affecting every compiled Cython module that uses generators. All five FIX findings have clear, minimal fixes. The codebase has a strong ABI-discipline story (all 16 private _Py* API usages are correctly guarded) and zero file-scope static PyObject* leaks outside per-module state.


Extension Profile

  • Templates: 28 C files, ~12 000 lines, 355 analysed functions
  • Init style: N/A (utility templates — no PyInit_*; embedded into generated extensions)
  • Python targets: CPython 3.9+ (free-threading: 3.13+)
  • Limited API: dual-path (CYTHON_COMPILING_IN_LIMITED_API, mutually exclusive with FT)
  • Types defined: 10 (CythonFunction, FusedFunction, Coroutine, Generator, IterableCoroutine, CoroutineAwait, AsyncGen, AsyncGenASend, AsyncGenAThrow, AsyncGenValueWrapper)
  • External tools: cppcheck — 62 findings, all false positives (template syntax); clang-tidy — skipped (no compile_commands.json)
  • Prior FT review: reports/ft_review.md — 0 RACE, 3 UNSAFE, 5 PROTECT, 6 MIGRATE; findings not re-reported here

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 1 0 new_base leaked on 5 error paths in __Pyx_PEP560_update_bases
Error Handling Y 2 2 PyMem_Malloc not NULL-checked; PyDict_GetItemStringRef return discarded
NULL Safety Y 1 0 inspect import not NULL-checked before use (Limited API)
GIL Discipline Y 1 0 GIL leaked on BaseException path in __Pyx_TraceReturnCValue
Resource Lifecycle G 0 1 Missing Py_tp_clear in 4 coroutine/generator type specs
Module State G 0 2 am_send method tables at file scope; single-interpreter guard (POLICY)
Type Slots G 0 2 gi_frame not visited in Coroutine_traverse; FusedFunction partial-init gap
PyErr_Clear Safety G 0 9 Multiple unguarded clears in MatchCase.c, ExtensionTypes.c, Profile.c
ABI Compliance G 0 0 16 private _Py* usages, all correctly guarded by feature-flag master block
Version Compat G 0 8 PyEval_GetBuiltins() deprecated in 3.13 — 6 ungated call sites
Complexity G 0 5 4 of top-10 complex functions have confirmed bugs; complexity is the common driver

G = 0 FIX findings · Y = 1–3 FIX findings · R = 4+ FIX findings


Findings by Priority

Global numbering: FIX first (1–5), then CONSIDER (6–28), then POLICY (29–47).


Must Fix (FIX) — 5

# Finding File:Line Agents
1 new_base ref leaked on 5 goto error paths in __Pyx_PEP560_update_bases ObjectHandling.c:1244,1256,1258,1266,1269 refcount-auditor, git-history-analyzer
2 PyMem_Malloc not NULL-checked before immediate dereference in __Pyx_MergeVtables ImportExport.c:732-733 error-path-analyzer
3 PyDict_GetItemStringRef return discarded on Python 3.13+ path; real exception clobbered by misleading PyErr_Format in __Pyx_ImportFromPxd ImportExport.c:591 error-path-analyzer
4 GIL leaked via PyGILState_Ensure when BaseException raised before PyGILState_Release in __Pyx_TraceReturnCValue (both 3.13 and pre-3.12 variants) Profile.c:253-262, 486-498 gil-discipline-checker
5 PyImport_ImportModule("inspect") not NULL-checked before use; segfaults on import failure. Limited API only. Sister site at line 2574 correctly checks. ModuleSetupCode.c:1316-1346 null-safety-scanner

FIX 1 Detail — new_base reference leak

__Pyx_PEP560_update_bases is ported from CPython's bltinmodule.c::update_bases (comment: "Shamelessly adapted"). CPython's original has explicit Py_DECREF(new_base) before each goto error. The Cython port restructured control flow and lost those inline DECREFs.

new_base becomes a live owned reference at line 1229 (new_base = __Pyx_PyObject_CallOneArg(meth, bases)). Five goto error paths that execute after this point do not release it:

Line Condition Build guard
1244 PyList_New fails Always compiled
1256 PyTuple_GetItem fails !CYTHON_ASSUME_SAFE_MACROS
1258 PyList_SetItem fails !CYTHON_ASSUME_SAFE_MACROS
1266 PyList_Size fails !CYTHON_ASSUME_SAFE_SIZE
1269 PyList_SetSlice fails Always compiled

The error: label at line 1285 releases new_bases and base but never new_base. new_base is declared uninitialized (PyObject *base = NULL, *meth, *new_base, ...), so a naive Py_XDECREF at the label would crash early-exit paths that execute before line 1229.

Fix (two-part):

// 1. Initialize new_base at declaration (line 1182):
PyObject *base = NULL, *meth, *new_base = NULL, ...

// 2. Add to the error: label (line 1285 area):
Py_XDECREF(new_base);

Py_XDECREF(NULL) is a no-op, so early-exit paths before line 1229 are unaffected.

FIX 4 Detail — GIL leak in __Pyx_TraceReturnCValue

The macro acquires the GIL via PyGILState_Ensure(&state) to call a C→Python value converter and fire a trace event. If the converter raises a BaseException (not a subclass of Exception — so KeyboardInterrupt, SystemExit, GeneratorExit), the goto_error inside the macro body fires before PyGILState_Release(state). The local state variable is lost; the GIL is permanently held for that thread.

Every other tracing macro in Profile.c correctly calls PyGILState_Release(state) before goto_error.

Fix: In both the sys.monitoring variant (line 256) and the classic trace variant (line 491), add PyGILState_Release(state); immediately before goto_error:

// Before:
if (unlikely(exc_info.exc_value && !PyErr_GivenExceptionMatches(exc_info.exc_value, PyExc_Exception)))
    goto_error;
PyGILState_Release(state);

// After:
if (unlikely(exc_info.exc_value && !PyErr_GivenExceptionMatches(exc_info.exc_value, PyExc_Exception))) {
    PyGILState_Release(state);
    goto_error;
}

Should Consider (CONSIDER) — 23

# Finding File:Line
6 Missing {Py_tp_clear, (void *)__Pyx_Coroutine_clear} in 4 type specs; GC cannot break cycles through coroutines/generators/async-generators Coroutine.c:~1965,2062,2166; AsyncGen.c:~401
7 __Pyx_Coroutine_traverse does not visit gi_frame, but __Pyx_Coroutine_clear does Py_CLEAR(gi_frame) — asymmetric traverse/clear Coroutine.c:1393-1401
8 Unguarded PyErr_Clear() during ABC type disambiguation in __Pyx_MatchCase_ABCCheck; silences MemoryError, produces wrong structural match results MatchCase.c:140, 184
9 Unguarded PyErr_Clear() after PyObject_RichCompareBool returns -1 in __Pyx_setup_reduce_is_named; any error (incl. MemoryError) silently corrupts pickle reduce logic ExtensionTypes.c:378
10 Unconditional PyErr_Clear() after __Pyx_PyObject_GetAttrStrNoError for __setstate__; non-AttributeError exceptions silently discarded ExtensionTypes.c:449
11 Unconditional PyErr_Clear() on success path of __Pyx_MergeVtables loop; swallows MemoryError from __Pyx_GetVtable calls ImportExport.c:780
12 Unguarded PyErr_Clear() in __Pyx__IsSameCyOrCFunction after PyObject_GetAttr fails; swallows MemoryError. Fix: guard with ExceptionMatches(AttributeError) CythonFunction.c:151
13 Three unguarded PyErr_Clear() calls during array module detection in __Pyx_MatchCase_IsSequence; comment says "soft errors" but swallows ImportError/MemoryError MatchCase.c:279, 285, 294
14 Unguarded PyErr_Clear() in __Pyx_PyType_GetFullyQualifiedName (Limited API path); falls back to "?". Low risk (diagnostic helper inside error handlers) ObjectHandling.c:3183
15 PyErr_Clear() in __Pyx_TraceReturnCValue guards against non-Exception but swallows MemoryError which IS an Exception subclass Profile.c:257, 268, 492, 506
16 PyDict_New in else-branch not NULL-checked; NULL passed to PyDict_SetItemString. Limited API path Exceptions.c:917
17 Legacy PyDict_GetItem without PyErr_Occurred() check; hash errors silently treated as "key absent". Replace with PyDict_GetItemWithError Dataclasses.c:41
18 PyEval_GetBuiltins() deprecated in Python 3.13 (replacement PyEval_GetFrameBuiltins() returns new reference vs. borrowed); 6 ungated call sites. Recommend __Pyx_PyEval_GetBuiltins() wrapper Exceptions.c:29; Builtins.c:72,119,912; CythonFunction.c:1989; ObjectHandling.c:1532
19 Three file-scope mutable __pyx_*_as_async structs used as am_send backport method tables (Python < 3.10 only); safe but need clarifying comment Coroutine.c:1906, 1995, 2197
20 __pyx_FusedFunction_New sets extension fields after __Pyx_CyFunction_New; unreachable error branch could Py_CLEAR uninitialized memory (LOW risk; tp_alloc zeroes struct) CythonFunction.c:1497-1517
21 __Pyx_call_next_tp_traverse silently returns 0 when no distinct parent traverse found; FIXME comment in source ("really ignore?"); may miss GC objects in unusual inheritance hierarchies ExtensionTypes.c:332
22 FIXME comments: non-BaseException class used in a Cython except clause silently accepted instead of raising TypeError (Python semantics mismatch) ModuleSetupCode.c:1623, 1637
23 TODO comment flags uncertain error path in __Pyx_RemoveModuleStateLookup; analysis suggests NULL means "table empty" and goto done is correct, but intent is not documented ModuleSetupCode.c:3161
24 __Pyx_ValidateAndInit_memviewslice has 9 goto fail paths in 63 lines; all checked but one goto fail inside the axes loop may leak new_memview in some combinations (unaudited) MemoryView_C.c:224-316
25 __Pyx_BufFmt_CheckString CC ~30, 133 lines; struct-recursion inner loop and complex-number look-ahead could be extracted to reduce audit surface. No known bugs Buffer.c:604-743
26 __Pyx_setup_reduce has 16 goto-to-cleanup sites in 88 lines (highest goto density in codebase); the multiple CYTHON_USE_PYTYPE_LOOKUP duplicated branches make it hard to audit consistently ExtensionTypes.c:386-487
27 __Pyx__MatchCase_ClassPositional CC ~22, 117 lines; match_self == -1 detection block (PEP 634 flags) is a refactor target MatchCase.c:830-959
28 __Pyx_PEP560_update_bases CC ~20, 11 gotos; the three CYTHON_ASSUME_SAFE_* macro axes multiply error paths invisibly — the refcount leak (FIX 1) is a direct consequence ObjectHandling.c:1179-1291

CONSIDER 6 Detail — Missing Py_tp_clear in coroutine/generator type specs

__Pyx_Coroutine_clear is correctly implemented (clears closure, classobj, gi_exc_state, gi_code, gi_frame, gi_name, gi_qualname, gi_modulename) and is called from __Pyx_Coroutine_dealloc. However, it is NOT registered as Py_tp_clear in any of the four primary generator/coroutine type specs.

Without tp_clear, Python's cyclic GC can detect cycles through these types (via tp_traverse) but cannot break them. Cycles involving Cython generators are uncollectable; they accumulate in the GC's garbage list and are freed only at interpreter shutdown or __del__ time.

__pyx_CyFunctionType and __pyx_CoroutineAwaitType correctly register tp_clear, making this omission inconsistent. CPython's own genobject.c has registered tp_clear since Python 2.5 — the Cython port of genobject.c omitted it.

Fix (add one slot entry per type spec, 4 total):

// In __pyx_CoroutineType_slots (Coroutine.c ~1965):
{Py_tp_traverse, (void *)__Pyx_Coroutine_traverse},
{Py_tp_clear,    (void *)__Pyx_Coroutine_clear},   // ADD

// In __pyx_GeneratorType_slots (Coroutine.c ~2166):
{Py_tp_traverse, (void *)__Pyx_Coroutine_traverse},
{Py_tp_clear,    (void *)__Pyx_Coroutine_clear},   // ADD

// In __pyx_IterableCoroutineType_slots (Coroutine.c ~2062):
{Py_tp_traverse, (void *)__Pyx_Coroutine_traverse},
{Py_tp_clear,    (void *)__Pyx_Coroutine_clear},   // ADD

// In __pyx_AsyncGenType_slots (AsyncGen.c ~401) — use a wrapper:
{Py_tp_traverse, (void *)__Pyx_async_gen_traverse},
{Py_tp_clear,    (void *)__Pyx_async_gen_clear},   // ADD (new wrapper below)

For AsyncGen, a thin wrapper is preferred to also clear ag_finalizer:

static int __Pyx_async_gen_clear(PyObject *self) {
    Py_CLEAR(((__pyx_PyAsyncGenObject*)self)->ag_finalizer);
    return __Pyx_Coroutine_clear(self);
}

Tensions

Complexity vs. compatibility: The dominant complexity driver in this codebase is conditional compilation (#if CYTHON_ASSUME_SAFE_MACROS, #if CYTHON_AVOID_BORROWED_REFS, #if CYTHON_COMPILING_IN_LIMITED_API, #if PY_VERSION_HEX) — not algorithmic complexity. Functions that appear to have 5 branches have 10–20 real paths across all preprocessor combinations. This is inherent to Cython's mission (single codebase for CPython, PyPy, Limited API, free-threading), but it is the direct cause of 4 of the 5 FIX findings: a version-guarded #else branch adds a new error exit that the existing cleanup label does not handle.

PyErr_Clear in "soft" checks: Findings 8 and 13 (MatchCase.c array/ABC detection) use PyErr_Clear intentionally to implement "soft" best-effort type checks. The design intent is reasonable (a failed isinstance(x, array.array) should not abort pattern matching), but MemoryError should not be swallowed. The tension is between fail-safe fallback and hiding serious errors. The recommended resolution is to guard each clear with if (PyErr_ExceptionMatches(PyExc_AttributeError) || PyErr_ExceptionMatches(PyExc_ImportError)) rather than clearing unconditionally.


Policy Decisions (POLICY) — 19

# Finding
29–44 16 private _Py* API usages in non-Limited-API CPython paths (_PyErr_StackItem, _PyType_Lookup, _PyLong_Copy, _PyObject_GetDictPtr, PyDictObject.ma_version_tag, _PyUnicodeWriter family, _PyGen_Send, _PySet_NextEntry, _PyBytes_Join, _PyThreadState_UncheckedGet, others). All correctly guarded — ModuleSetupCode.c:211-283 zeroes all internal-access feature flags in Limited API builds (CYTHON_USE_TYPE_SLOTS=0, CYTHON_COMPILING_IN_CPYTHON=0, etc.). No action required.
45 CYTHON_USE_MODULE_STATE=0 default: all Python-object globals live in a process-wide struct; __Pyx_check_single_interpreter() enforces single-interpreter use via ImportError. Migration path: CYTHON_USE_MODULE_STATE=1.
46 __Pyx_FastGilFuncs process-wide vtable (ModuleSetupCode.c:2310-2368). Contains only C function pointers. Dead code under CPython 3.12+ free-threading.
47 Module-state lookup registry (ModuleSetupCode.c:2800-2875) is intentionally process-wide — the only feasible design for a per-interpreter module lookup table.

Strengths

  • Zero bare file-scope static PyObject*: Every Python-object global routes through CGLOBAL(x) → modulestateglobal->x (per-module-state struct). The module state architecture is sound.
  • Complete Limited API feature-flag discipline: The master block at ModuleSetupCode.c:211-283 correctly zeroes all 8 internal-access flags in Limited API builds. All 16 private _Py* API usages are cleanly gated.
  • Strong GIL accounting (non-profile code): All 14 non-TraceReturnCValue PyGILState_Ensure sites in Profile.c correctly release before goto_error. The 14 other PyGILState sites across Buffer.c, Builtins.c, Complex.c, Exceptions.c, MemoryView_C.c, ObjectHandling.c, StringTools.c are clean.
  • Correct CythonFunction GC protocol: __pyx_CyFunctionType and __pyx_FusedFunctionType correctly register tp_traverse, tp_clear, and tp_dealloc covering the same fields; PyObject_GC_Track is called only after full initialization.
  • All PyErr_Fetch/PyErr_Restore pairs balanced: All 12 pairs across the codebase are correctly balanced, including the subtle Coroutine.c exception chaining pair that correctly accounts for two steal-by-reference semantics.
  • PyMem_Malloc — single gap: Of 21+ PyImport_ImportModule call sites across the codebase, only 1 is unchecked. Of all PyMem_Malloc uses, only 1 is unchecked. Error-handling discipline is otherwise strong.
  • Memoryview decomposition: MemoryView_C.c decomposes memoryview validation into short, focused helpers (__pyx_check_strides, __pyx_check_suboffsets, __pyx_verify_contig) rather than one monolithic function. Structural quality is high.
  • ModuleSetupCode.c clean despite 3339 lines: No function in the largest file has CC ≥ 15. The one confirmed bug there (FIX 5) is in a 30-line function with CC ≈ 5 — a missed null-check, not a complexity failure.

Code Removal Opportunities

Based on deprecated_apis.json and version compat findings:

  • PyEval_GetBuiltins → PyEval_GetFrameBuiltins (3.13+): 6 call sites. A __Pyx_PyEval_GetBuiltins() wrapper (~8 lines) in ModuleSetupCode.c would centralize the version dispatch and handle the new-reference vs. borrowed-reference difference in one place. Est. net reduction: +8 lines wrapper, -6 #if PY_VERSION_HEX guards = ~flat.

  • PyErr_NormalizeException → PyErr_GetRaisedException (3.12+): Already tracked in reports/ft_review.md (MIGRATE finding). Coroutine.c:683. Est. 4-5 lines saved.

  • PyDict_GetItem → PyDict_GetItemWithError: 2 sites (Findings 17, 11-area). No line savings but removes PyErr_Occurred() check pattern.


Recommended Action Plan

Immediate (FIX items)

  1. Finding 1 — ObjectHandling.c:1182,1285: Initialize new_base = NULL; add Py_XDECREF(new_base) at error: label. Two-line fix.
  2. Finding 2 — ImportExport.c:732: Add if (!base_vtables) { PyErr_NoMemory(); return -1; } after PyMem_Malloc.
  3. Finding 3 — ImportExport.c:591: Check PyDict_GetItemStringRef return; goto bad on -1 before the !cobj test.
  4. Finding 4 — Profile.c:256, 491: Add PyGILState_Release(state); before goto_error in both __Pyx_TraceReturnCValue variants.
  5. Finding 5 — ModuleSetupCode.c:1319: Add if (unlikely(!inspect)) return -1; after the import call.

Short-term (High-priority CONSIDER)

  1. Finding 6 — Add {Py_tp_clear, (void *)__Pyx_Coroutine_clear} to all four coroutine/generator/async-generator type specs. Add __Pyx_async_gen_clear wrapper for AsyncGen. Four-line change across two files (plus the 6-line wrapper).
  2. Finding 7 — Add Py_VISIT(gen->gi_frame) to __Pyx_Coroutine_traverse (one line).
  3. Finding 9 — ExtensionTypes.c:378: Guard PyErr_Clear() with if (PyErr_ExceptionMatches(PyExc_TypeError) || PyErr_ExceptionMatches(PyExc_AttributeError)).
  4. Finding 11 — ImportExport.c:780: Guard the loop-end PyErr_Clear() with if (PyErr_ExceptionMatches(PyExc_KeyError) || PyErr_ExceptionMatches(PyExc_AttributeError)).
  5. Finding 8 — MatchCase.c:140, 184: Guard PyErr_Clear() with ExceptionMatches(AttributeError) / ExceptionMatches(TypeError) respectively.

Medium-term (CONSIDER cluster)

  1. Finding 18 — Add __Pyx_PyEval_GetBuiltins() wrapper and migrate all 6 call sites (centralized version dispatch for PyEval_GetBuiltins → PyEval_GetFrameBuiltins).
  2. Findings 10, 12, 13, 15 — Audit remaining PyErr_Clear() sites in ExtensionTypes.c:449, CythonFunction.c:151, MatchCase.c:279/285/294, Profile.c:257/268/492/506 and add appropriate ExceptionMatches guards.
  3. Finding 16 — Exceptions.c:917: Add if (unlikely(!dict)) goto bad; after PyDict_New() in the else-branch (Limited API path).
  4. Finding 17 — Dataclasses.c:41: Replace PyDict_GetItem with PyDict_GetItemWithError; add if (PyErr_Occurred()) return -1; guard.
  5. Finding 24 — Audit all 9 goto fail paths in __Pyx_ValidateAndInit_memviewslice to confirm new_memview is released on every path.

Longer-term (Complexity and Policy)

  1. Findings 26–28 — Refactor __Pyx_setup_reduce, __Pyx__MatchCase_ClassPositional, and __Pyx_PEP560_update_bases to reduce goto density and make version-guarded error paths more auditable. See reports/cext_v1/complexity.md for specific extraction suggestions.
  2. Finding 22 — ModuleSetupCode.c:1623, 1637: Implement the FIXME (raise TypeError for non-BaseException class in catch clause) to match Python semantics.
  3. Policy Finding 45 — Evaluate CYTHON_USE_MODULE_STATE=1 as the new default to enable full subinterpreter isolation.
  4. CONSIDER 19 — Add clarifying comments to the three __pyx_*_as_async structs in Coroutine.c explaining they are am_send backport method tables (Python < 3.10) and are intentionally shared.

Relationship to ft-review Findings

This report and the free-threading review (reports/ft_review.md) are complementary and non-overlapping:

Topic ft-review coverage cext-review coverage
Thread-safety races ✓ RACE findings —
Borrowed refs under FT ✓ Optimize.c:210,222 (UNSAFE) — (cross-validated)
Non-atomic shared state ✓ TString.c, CO_*, TPFLAGS —
Critical section correctness ✓ Lock discipline audit —
Refcount leaks (all builds) — ✓ FIX 1 (this report)
GIL discipline (classic) — ✓ FIX 4 (this report)
tp_clear missing — ✓ CONSIDER 6 (this report)
PyErr_Clear discipline — ✓ Findings 8–15 (this report)
Version deprecations — ✓ Findings 18, etc. (this report)

The three ft-review UNSAFE findings (Optimize.c:210,222; AsyncGen.c:231; CythonFunction.c:632) are not re-reported here; see reports/ft_review.md for fixes.


CPython-Ported Functions — Risk Register

Function File CPython source Port omission?
__Pyx_PEP560_update_bases ObjectHandling.c:1177 bltinmodule.c::update_bases ✓ FIX 1 (5 missing DECREFs)
__Pyx_CoroutineObject family Coroutine.c genobject.c (Py 3.5) ✓ CONSIDER 6 (missing tp_clear), CONSIDER 7 (gi_frame asymmetry)
__Pyx__MatchCase_Mapping_ExtractNonDict MatchCase.c:582 ceval.c::match_keys No confirmed omissions
__Pyx__MatchCase_ClassPositional MatchCase.c:824 ceval.c::match_class No confirmed omissions
__Pyx__PyObject_LookupSpecial ObjectHandling.c:1663 ceval.c::special_lookup Not audited
__Pyx_PyObject_FastCallDict ObjectHandling.c:2593 _PyObject_FastCallDict Not audited

This report was produced by the cext-review-toolkit. Before sharing externally, consult WORKING_WITH_MAINTAINERS.md — a large unsolicited findings report is a burden. Coordinate with the maintainer first, do not auto-file issues, and handle security findings via private disclosure.

C Extension Analysis Report

Extension: frozenlist

Scope: entire project

Agents Run: gil-discipline-checker, resource-lifecycle-checker, type-slot-checker,

module-state-checker, parity-checker, version-compat-scanner,

c-complexity-analyzer (git-history-analyzer: shallow clone, 1 commit — CHANGES/ used instead)


Executive Summary

frozenlist/_frozenlist.pyx is a small, well-structured Cython C++ extension (156 lines, 1 type) with no hand-written C and no memory-safety issues. The generated C is excluded from VCS; external tools (clang-tidy, cppcheck) found 0 findings. The dominant finding category is behavioral parity between the Cython and pure-Python implementations: index() accepts only one argument in Cython while inheriting a three-argument signature from MutableSequence on the Python path, causing TypeError on the Cython path in code that passes start/stop. The type stub incorrectly declares slice __getitem__ as returning FrozenList[_T] when both implementations return list. Secondary findings are lifecycle hygiene (__init__ instead of __cinit__) and free-threading readiness (see reports/ft_review.md for the complete TOCTOU analysis; this report surfaces only the GIL-discipline lens).


Extension Profile

  • Module: frozenlist._frozenlist (Cython source — 0 committed .c files; .c is a build artifact excluded from VCS via .gitignore)
  • Code generation: Cython 3.2.5 (# distutils: language = c++; C++ std::atomic<bool>)
  • Init style: multi-phase (Py_mod_exec slot)
  • Python targets: >=3.10
  • Limited API: no
  • Types defined: 1 (FrozenList — cdef class, GC-enabled)
  • Pure-Python fallback: frozenlist/__init__.py (93 lines, class FrozenList(MutableSequence))
  • Type stubs: frozenlist/__init__.pyi

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety G 0 0 N/A — Cython-managed; no manual Py_INCREF/Py_DECREF in source
Error Handling G 0 0 N/A — Cython error paths are mechanical (skipped per Cython strategy)
NULL Safety G 0 0 N/A — Cython null checks are mechanical (skipped per Cython strategy)
GIL Discipline G 0 1 freethreading_compatible = True overstated for concurrent mutate-and-freeze
Resource Lifecycle G 0 1 __init__ used instead of __cinit__; _items NULL after __new__ without __init__
Module State G 0 1 Py_mod_multiple_interpreters slot absent; blocked via Cython runtime guard
Type Slots G 0 1 new_missing_member_init — same root as Resource Lifecycle (merged)
PyErr_Clear Safety G 0 0 N/A — no maintainer-owned PyErr_Clear in source
ABI Compliance G 0 0 No limited API claim; no hand-written C; N/A
Version Compat G 0 1 Missing explicit language_level=3 directive
C/Python Parity G 0 3 index() arity mismatch; stub slice return type wrong; __deepcopy__ absent from Python path
Complexity G 0 2 __richcmp__ unnecessary list copy; _fast_len dead indirection

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 0

No FIX-class findings. No crash-inducing bugs, no memory corruption, no reference leaks in maintainer-authored code.

Should Consider (CONSIDER) — 9

# Finding File:Line Agents
1 __init__ used instead of __cinit__ for C-level member initialization. cdef list _items (line 16) and atomic[bint] _frozen (line 15) are only initialized inside __init__ (lines 19, 24). Cython's tp_new zeroes all struct fields before __init__ runs; if a Python subclass overrides __init__ without calling super().__init__(), or if the object is created via FrozenList.__new__(FrozenList), _items remains NULL. Every method that accesses _items (__len__, __getitem__, __iter__, __reversed__, __contains__, index, count, all mutation methods, __hash__, __copy__, __deepcopy__) then either raises SystemError: null argument to internal routine or exhibits undefined behavior. The pure-Python path raises a clean AttributeError (slots descriptor detects unset slot) — a parity divergence on failure mode. A partial-__init__ failure (e.g., list(items) raises) also leaves _items NULL, resolved by the same fix. Fix: move both initializations to __cinit__ — guaranteed to run even when __init__ is bypassed: def __cinit__(self): self._frozen.store(False); self._items = []. Then __init__ only sets self._items = list(items) when items is not None. _frozenlist.pyx:15-16,18-24 resource-lifecycle-checker, type-slot-checker
2 index() arity mismatch between Cython and pure-Python paths. Cython declares def index(self, item) (line 86), accepting only the value to search. The pure-Python PyFrozenList inherits MutableSequence.index, whose signature is index(value, start=0, stop=None). Calling frozen_list.index(value, 1) raises TypeError on the Cython path (wheels installed) but returns the correct result on the Python fallback path (no wheels, PyPy). This is a broken ABC contract: any code that uses the full start/stop interface will work in some environments and fail silently in others. The .pyi stub inherits index from MutableSequence[_T], implying three-argument support that the C path does not honour. Fix: def index(self, item, start=0, stop=None): return self._items.index(item, start) if stop is None else self._items.index(item, start, stop). _frozenlist.pyx:86 parity-checker
3 .pyi stub declares __getitem__(s: slice) -> FrozenList[_T] but both implementations return list. Both _frozenlist.pyx:40-41 and __init__.py:34-35 delegate slicing to self._items[index] where _items is a plain list; slicing a list returns a list. The stub overload at __init__.pyi:15 promises FrozenList[_T], causing mypy and type-aware IDEs to infer that fl[0:2] supports .freeze(), .frozen, .insert() etc., then failing at runtime with AttributeError. No test in the suite checks the return type of slice __getitem__. Fix: change __init__.pyi:15 to def __getitem__(self, s: slice) -> list[_T]: .... frozenlist/__init__.pyi:15 parity-checker
4 __deepcopy__ present in Cython but absent from __init__.py — parity gap. The Cython FrozenList implements __deepcopy__ (lines 133-152) with explicit circular-reference handling and _frozen propagation. PyFrozenList has no __deepcopy__. The default deepcopy protocol handles this correctly for __slots__ classes (all 14 deepcopy test cases pass including circular-reference cases), so there is no correctness gap today. The gap becomes relevant when the FT fix is applied to __deepcopy__ (adding a critical section on the Cython path), since AGENTS.md requires matching changes in both implementations. The .pyi stub does not declare __deepcopy__ (acceptable — it is a protocol hook, not a public API). Suggested action: add a matching __deepcopy__ to __init__.py at the same time the FT fix is applied to _frozenlist.pyx. __init__.py (absent) parity-checker, c-complexity-analyzer
5 freethreading_compatible = True partially justified. The declaration at _frozenlist.pyx:1 is correct for the primary use case: append items, call freeze(), share the frozen list across threads for concurrent read-only access. It is not correct for the concurrent mutate-and-freeze use case: _check_frozen() (line 30-32) atomically reads _frozen, then returns; the caller mutates _items in a separate unprotected step. A race between _check_frozen() passing and freeze() completing produces a mutation on a semantically frozen list. This is a logical invariant violation, not a crash (_items per-list locking prevents C-level corruption). See reports/ft_review.md Findings 1-4 for the complete line-level analysis and the @cython.critical_section fix. This entry provides the GIL-discipline perspective only. _frozenlist.pyx:1,30-38,43-111 gil-discipline-checker
6 Py_mod_multiple_interpreters slot absent from module definition. With CYTHON_USE_MODULE_STATE = 0 (Cython 3.2.5 default), the slot is conditionally emitted only when CYTHON_USE_MODULE_STATE is enabled. On Python 3.12+ with isolated subinterpreters, CPython sees no declaration and assumes Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED; the module is then blocked by __Pyx_check_single_interpreter() inside __pyx_pymod_create, which raises ImportError("Interpreter change detected") rather than the standard PermissionError. The protection is effective but non-standard. Fix (build-level, no .pyx change needed): add -DCYTHON_USE_MODULE_STATE=1 to CFLAGS in pyproject.toml's [tool.local.cythonize.env] section; Cython 3.2.5 already generates the correct Py_mod_multiple_interpreters = Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED slot under this flag. frozenlist/_frozenlist.pyx (generated slot) module-state-checker
7 Missing # cython: language_level=3 directive. _frozenlist.pyx opens with # cython: freethreading_compatible = True and # distutils: language = c++ but no language_level directive. Cython 3.x defaults to 3str; Cython 4.x is expected to default to 3. For this codebase the practical impact is zero (no cdef str, no byte-string handling), but relying on the default makes the Cython 4 migration silent rather than explicit. Fix: add # cython: language_level=3 as the first directive in _frozenlist.pyx. _frozenlist.pyx:1-2 version-compat-scanner
8 __richcmp__ creates a redundant list copy per comparison. Lines 60-72: every branch calls list(self) which iterates self._items via __iter__ and constructs a new O(n) list before performing the comparison. Since self._items is already a list, the copy is pure overhead — self._items < other delegates directly to CPython's list comparison without allocation. The two forms are semantically identical for standard list inputs. This also intersects with FT-RACE-3 in reports/ft_review.md: the FT fix will need to snapshot self._items inside a critical section (for unfrozen lists), so the list(self._items) inside the section replaces list(self) — but for frozen lists, self._items < other without any copy is valid. _frozenlist.pyx:60-72 c-complexity-analyzer
9 _fast_len is a dead indirection. cdef inline object _fast_len(self) (lines 34-35) wraps len(self._items) in a single line and is called only from __len__ (line 52). The cdef inline optimization benefit (C-level inlining) only applies when called from another cdef/cpdef function; from a def method, Python's slot dispatch applies anyway. The return type object boxes the result into a Python int unnecessarily. Fix: inline the body into __len__: def __len__(self): return len(self._items) and remove _fast_len. _frozenlist.pyx:34-35,52 c-complexity-analyzer

Tensions

Module state vs. simplicity (CYTHON_USE_MODULE_STATE):

  • module-state-checker flags the missing Py_mod_multiple_interpreters slot (Finding 6 above).
  • The extension is tiny (1 type, no mutable module-level state). The __Pyx_check_single_interpreter() guard already blocks multiple-interpreter use correctly, just with a less standard error.
  • Resolution: enabling -DCYTHON_USE_MODULE_STATE=1 requires zero .pyx changes and no functionality change. Low-effort, low-risk improvement if the project wants standard slot-based declarations.

__deepcopy__ parity vs. YAGNI:

  • parity-checker and complexity analyzer flag that __deepcopy__ is absent from __init__.py.
  • parity-checker confirmed all 14 deepcopy tests pass on PyFrozenList without it — Python's default protocol handles the cases correctly.
  • Resolution: leave it absent unless the FT fix modifies __deepcopy__ in the Cython path, at which point AGENTS.md requires a matching implementation in __init__.py.

Policy Decisions (POLICY) — 2

# Finding
10 CYTHON_USE_MODULE_STATE = 0 is Cython 3.2.5's standard default for all generated modules. The generated code already contains all the infrastructure for m_size-based per-module state (traverse, clear, Py_mod_multiple_interpreters slot); it is gated on the build-time macro. The project should decide whether to opt in to the proper slot declarations (low-effort via CFLAGS, see Finding 6) or document the current approach. This is not a bug — the module is functionally correct.
11 Cython >= 3.1.1 has no upper bound in get_requires_for_build_wheel (packaging/pep517_backend/_backend.py:378). CI pins cython==3.2.5 via requirements/cython.txt enforced by PIP_BUILD_CONSTRAINT/UV_BUILD_CONSTRAINT. Developer builds outside CI (e.g., pip install -e . without the constraint file) will receive whatever >= 3.1.1 resolves to, which will include Cython 4.x when released. The project should decide whether to add <4 to the dynamic requirement or accept that Cython 4 compatibility is the operator's responsibility.

Strengths

  • No hand-written C code: All C-level operations are Cython-generated, including reference counting, null checks, error cleanup, and GC traversal. The generator handles these correctly; maintainers do not need to reason about them.
  • tp_dealloc correctly absent: Cython's auto-generated dealloc calls ~atomic<bint>() (trivial destructor), issues Py_XDECREF(_items) with NULL-guard, and PyObject_GC_UnTrack. Nothing to maintain.
  • GC slots correctly generated: Py_TPFLAGS_HAVE_GC is set; tp_traverse visits _items; tp_clear clears it. test_deepcopy_circular confirms cycle detection works correctly.
  • tp_hash correctly defined: Explicit __hash__ overrides the __richcmp__-implies-no-hash rule; hash is only available on frozen lists; tests confirm correct behavior.
  • All 6 tp_richcompare ops handled: __richcmp__ dispatches ops 0-5 exhaustively. The implicit return None for op > 5 is dead code (CPython never passes values outside 0-5) and harmless.
  • atomic[bint] for _frozen: Uses C++ std::atomic<bool> via libcpp.atomic.atomic[bint] with .load()/.store() throughout — correct and GIL-independent for the frozen flag.
  • __copy__ correctly implemented: Creates a new instance, conditionally propagates _frozen state; confirmed that copy.copy() no longer shares the internal list (CHANGES/743.bugfix.rst).
  • Multi-phase init: Uses Py_mod_exec slot (not single-phase PyInit_* returning the module directly). Import is importable multiple times without module-state leaks.
  • No static PyObject* cached imports: import copy, import types, from collections.abc import MutableSequence are stored in the module dict (per-interpreter), not in file-scope static pointers.

Code Removal Opportunities

All four entries in deprecated_apis.json (PyModule_AddType, PyImport_ImportModuleAttrString, PyDict_GetItemRef, PyUnicode_AsUTF8AndSize) apply to hand-written C code that manually calls these patterns. Since frozenlist has no hand-written C — all C-level calls are Cython-generated — none of these apply to maintainer-authored source. The Cython generator may or may not use the newer APIs internally depending on its target Python version; upgrading the Cython build version would pick up any such improvements automatically.


Recommended Action Plan

Immediate (CONSIDER items with most user impact)

  1. Fix Finding 2 — index() arity mismatch (highest user impact). Add start=0, stop=None parameters to def index(self, item, ...) in _frozenlist.pyx and delegate to self._items.index(item, start) / self._items.index(item, start, stop). This is a silent environment-specific failure affecting any caller that uses list.index start/stop.

  2. Fix Finding 3 — stub return type for __getitem__(slice). Change frozenlist/__init__.pyi:15 from -> FrozenList[_T] to -> list[_T]. Pure annotation fix; no runtime change needed.

  3. Fix Finding 1 — __cinit__ for C-level initialization. Move self._frozen.store(False) and self._items = [] to a def __cinit__(self) method. This eliminates the NULL _items crash window and the Cython/Python parity divergence on __new__-without-__init__ paths. Also resolves the partial-init crash risk.

Short-term (CONSIDER items with lower immediate impact)

  1. Finding 7 — add # cython: language_level=3 as the first directive in _frozenlist.pyx. One-line change; no semantic impact today; prevents silent breakage on Cython 4 adoption.

  2. Finding 8 — replace list(self) with self._items in __richcmp__. Eliminates O(n) allocation on every comparison. Requires care when applying the FT fix (Finding 5): the snapshot must be taken inside the critical section for unfrozen lists.

  3. Finding 9 — remove _fast_len. Inline len(self._items) into __len__ directly. One-line change; removes dead indirection.

Apply with FT fix (CONSIDER items tied to reports/ft_review.md)

  1. Finding 4 — add __deepcopy__ to __init__.py at the same time __deepcopy__ in _frozenlist.pyx is modified to add FT locking. AGENTS.md requires both implementations to be updated in the same PR.

  2. Finding 5 — update freethreading_compatible declaration. Remove the declaration now; restore it after @cython.critical_section is applied to all mutation methods and freeze(), and concurrent stress tests pass. See reports/ft_review.md for the complete fix plan.

Longer-term (POLICY)

  1. Finding 6 / Finding 10 — enable CYTHON_USE_MODULE_STATE=1 in CFLAGS if standard Py_mod_multiple_interpreters slot declarations are desired. No .pyx changes required.

  2. Finding 11 — add <4 upper bound to the Cython >= 3.1.1 dynamic build requirement in packaging/pep517_backend/_backend.py:378 to protect out-of-CI developer builds from inadvertent Cython 4 adoption.


Historical Notes (from CHANGES/ — git history is a shallow clone with 1 commit)

  • CHANGES/743.bugfix.rst: Fixed copy.copy() sharing the internal items list between the original and the copy (mutations to one affected the other). The current __copy__ implementations in both Cython and Python correctly create independent lists.
  • CHANGES/765.breaking.rst: Python 3.9 support dropped; 3.10 is now the minimum.
  • CHANGES/775.breaking.rst: CPython 3.13t (free-threaded 3.13) wheels dropped; CI still tests on 3.14t.

Before sharing this report externally, see WORKING_WITH_MAINTAINERS.md in the toolkit. A report dropped on a maintainer who did not ask for it is a homework assignment. Reach out first; ask what shape they want it in; do not auto-file or auto-PR. Security findings need responsible disclosure.

C Extension Analysis Report

Extension: google_crc32c

Scope: packages/google-crc32c (the only C extension in the google-cloud-python monorepo — everywhere else is pure Python; scope narrowed from the repo default with prior agreement in this session)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer

Executive Summary

_crc32c.c (91 lines, hand-written, single-phase init) is a small, clean CPython C extension — ten of eleven applicable review dimensions came back with zero FIX findings, and the file has essentially no complexity, no unmanaged resources, and no unsafe error handling. The one substantive bug in this review isn't in the C extension at all: it's in the pure-Python fallback (python.py), which silently computes a different CRC32C checksum than the C extension for the same input when that input is an array.array with itemsize != 1 (or an equivalently-cast memoryview) — no exception is raised on either backend, so a caller has no way to detect the divergence. This bug has existed unchanged since the file was created in 2020; the package does have a real dual-backend (C vs. Python) parametrized test suite, but it has never been given a non-bytes buffer input, so it never caught this. Two structural opt-in opportunities (limited API / abi3, and free-threading support) are cheap and low-risk given the file's small, stateless design, but are maintainer calls, not bugs.

Extension Profile

  • Module: _crc32c (1 C file, 91 lines)
  • Init style: single-phase (PyModule_Create, m_size=-1)
  • Python targets: >=3.10 (tested clean through 3.15)
  • Limited API: no (not claimed; one non-limited macro would need to change to adopt it — see Finding 4)
  • Types defined: 0 (pure function-module — no PyTypeObject anywhere)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety G 0 0 Clean — borrowed references never outlive their scope
Error Handling G 0 0 Every failable call checked or correctly passed through
NULL Safety G 0 0 No dereference without a preceding check
GIL Discipline G 0 0 Save/restore pairing correct on every path (classic-GIL lane; see Tensions for the free-threading-specific gap found in a prior review)
Resource Lifecycle G 0 0 No manually-managed resources exist in this file
Module State G 0 1 Single-phase init has no real state hazard (zero mutable globals) but blocks a Py_mod_gil declaration
Type Slots N/A - - File defines no PyTypeObject — lane doesn't apply
PyErr_Clear Safety N/A - - No exception-clearing calls anywhere — lane doesn't apply
ABI Compliance G 0 1 PyTuple_GET_ITEM (2 sites) is the only symbol blocking abi3
Version Compat G 0 0 No deprecated/removed API usage anywhere
C/Python Parity Y 1 2 array.array(itemsize≠1) silently produces a different checksum in the Python fallback than in C
Complexity G - 0 All 4 functions score 1.0/5.0 — trivially simple

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings

Findings by Priority

Must Fix (FIX) — 1

# Finding File:Line Agents
1 python.py's pure-Python fallback (update(), both the module-level function and Checksum.update()) special-cases array.array input: when itemsize != 1, it does array.array("B", data), which iterates the source's logical element values, not its raw bytes. The C extension's "y#" format always reads the raw underlying buffer via the buffer protocol, regardless of the source's logical typecode. For the same input object — e.g. array.array('I', [1, 2, 3]), or a memoryview cast to a wide format — the two backends silently compute different CRC32C values with no exception on either path (verified empirically: 0xea4b121a from the raw-byte reading vs. 0xf130f21e from the element-value reading, for the identical 3-element array('I', ...)). This is a genuine, currently-undetectable data-integrity bug for any caller who might run on either backend (the package falls back to Python silently when the C extension isn't importable) and who ever passes a non-byte-itemsize buffer object directly rather than pre-converting to bytes. The bug has existed unchanged since the file's creation in 2020 and has never been reported. src/google_crc32c/python.py:73-76 (vs. src/google_crc32c/_crc32c.c:25,50) parity-checker (verified via git-history-analyzer: 6-year-old, untouched, no prior bug report)

Should Consider (CONSIDER) — 5

# Finding File:Line
2 python.py additionally accepts a bare list/tuple of ints (via the same array.array("B", data) conversion path), matching its own docstring's advertised Union[bytes, List[int], Tuple[int]] signature — but the C extension's "y#" format requires a genuine buffer-protocol object and raises TypeError for a bare list/tuple. Lower severity than Finding 1 because the input type is chosen by calling code rather than arriving from untrusted wire data, but it's a real cross-backend API-contract inconsistency. src/google_crc32c/python.py:20-50 (docstrings) vs. src/google_crc32c/_crc32c.c:25,50
3 tests/test___init__.py has genuine dual-backend test scaffolding (pytest_generate_tests parametrizes over ["python", "cext"], used by most tests) — the right shape to have caught Finding 1 — but every parametrized test input is a plain bytes literal; none exercise array.array or memoryview of any itemsize. Separately, the one existing list-of-ints test (test_value, added 2018, predating the Python backend) explicitly coerces its input to bytes before calling the API, neutering its original "test list-of-integers, too" intent, and isn't parametrized over both backends at all. Add array.array('I', ...)/wide-memoryview.cast(...)/bare-list cases to the dual-backend parametrization when fixing Finding 1, so a regression is actually caught. tests/test___init__.py (parametrization fixture + test_value)
4 PyTuple_GET_ITEM (used to recover the original argument object for a PyBytes_Check type test) is the only symbol in the file not available under the limited API — everything else (PyArg_ParseTuple, PyBytes_Check, PyEval_SaveThread/RestoreThread, PyLong_FromUnsignedLong, PyModule_Create/PyModuleDef) already resolves to a stable-ABI-safe form. A two-site swap to PyTuple_GetItem (bounds-checked, same semantics, stable since 3.2) removes the only code-level blocker to building this extension against the limited API. src/google_crc32c/_crc32c.c:28,53
5 Single-phase module init (PyModule_Create, m_size=-1) has no real module-state hazard today — the file's only static, gil_threshold, is a genuine immutable const and there are zero mutable globals — but it structurally blocks declaring a Py_mod_gil slot. This is the same finding an earlier free-threading-focused review of this file surfaced (see Tensions below); migrating to multi-phase init (or calling PyUnstable_Module_SetGIL from the existing single-phase PyInit__crc32c) is enablement-only, not a bug fix, since there's no state to migrate. src/google_crc32c/_crc32c.c:76-90
6 (Tooling note, not a code finding) scan_module_state.py produced a false-positive missing_module_traverse finding on this file, caused by mis-parsing the PyModuleDef struct's field positions (it read the doc-comment text as the m_size value). analyze_history.py's automatic project-root detection also broke on this monorepo layout — it stops walking upward at the package's own pyproject.toml/setup.py rather than the real git root, then silently analyzes the entire monorepo's last N commits instead of the requested package subtree (same class of bug found in a prior session's analyze_ft_history.py run on this same package). Worth reporting both to the toolkit maintainers. n/a (tooling)

Tensions

  • This review vs. the prior free-threading-focused review of the same file: a session earlier today ran ft-review-toolkit against this exact file and found (a) a RACE-classified finding — "y#"/"ky#" accepts bytearray/memoryview in addition to bytes, but only bytes gets the immutability check gating the GIL-release decision, which is safe today only because of (b); and (b) the same single-phase-init/Py_mod_gil structural gap as Finding 5 above. This review's gil-discipline-checker (standard, classic-GIL lane) correctly found 0 findings — the RACE finding is free-threading-specific and out of this toolkit's default lane, not a contradiction. If Finding 5/POLICY-8 (adopting Py_mod_gil) is ever acted on, the RACE finding from the prior review must be fixed in the same change — enabling free-threading without it would turn a currently-latent bug live. Full report: packages/google-crc32c/reports/ft_analysis.md (this same session, same repo).
  • Module-state-checker vs. version-compat-scanner: both flagged the single-phase-init gap from different angles (module-state hazard vs. free-threading-declaration opportunity) — merged into Finding 5 / POLICY 8 respectively rather than double-counted.

Policy Decisions (POLICY) — 2

# Finding
7 Adopting the limited API (abi3) is plausibly high-value here: the package currently ships 5 separate per-Python-version wheels (3.10–3.14) per platform, built from byte-identical C source (no version-conditional code exists in this file) across manylinux/osx/windows build scripts. Fixing Finding 4 is the only code change required; the rest is a setup.py/build-config change (py_limited_api=True, #define Py_LIMITED_API). Would collapse the wheel matrix roughly 5x. Maintainer call, not a defect.
8 Adopting a Py_mod_gil = Py_MOD_GIL_NOT_USED free-threading declaration is a reasonable low-effort target: the module has zero mutable state and already manually releases the GIL around the CRC computation. Requires the same structural change as Finding 5, and must be paired with fixing the RACE finding from the prior free-threading review (see Tensions) before it would be safe to ship. Maintainer call on priority, not a defect in the current (GIL-required) build.

Strengths

  • Refcounting, error handling, and NULL safety are all clean — every failable call (PyArg_ParseTuple, implicitly PyLong_FromUnsignedLong, PyModule_Create) is either explicitly checked or uses the correct return-value-passthrough idiom; no borrowed reference is ever held past its safe scope.
  • GIL save/restore discipline is textbook-correct under the classic (GIL-enabled) interpreter — verified independently across two separate reviews (this one and the earlier free-threading-focused one) with no C-API call ever occurring inside a released region.
  • Zero manually-managed resources — no malloc/PyMem_*/file handles/sockets anywhere; the one buffer touched is borrowed directly from the argument object via the buffer protocol, with lifetime correctly bounded by the caller.
  • The core CRC32C algorithm itself is correct and consistent between backends — the parity-checker confirmed the standard Castagnoli check vector (value(b"123456789") == 0xe3069283), the lookup table, the empty-input case, and multi-call chaining semantics (update() called twice matches update() called once on the concatenation) all check out. The bug found (Finding 1) is specifically in input-buffer interpretation for a narrow, rarely-used input shape, not in the checksum algorithm itself.
  • Trivially low complexity — all 4 functions score at the complexity floor (1.0/5.0), nothing to refactor.
  • No deprecated/removed API usage across the full 3.10–3.15 range, and no dead PY_VERSION_HEX compatibility cruft to clean up.

Code Removal Opportunities

None applicable. Checked data/deprecated_apis.json's code_removal_opportunities list (PyModule_AddType, PyImport_ImportModuleAttrString, PyDict_GetItemRef, PyObject_HasAttrStringWithError) against this file: it defines no types, performs no imports, and touches no dicts, so none of the listed replacement patterns apply.

Recommended Action Plan

Immediate (FIX items)

  1. Fix Finding 1 — make python.py's buffer handling read raw bytes for any buffer-protocol object regardless of logical itemsize (e.g., route non-bytes/non-1-byte-itemsize inputs through memoryview(data).cast("B") or an equivalent raw-byte view, matching what "y#" does in C), so both backends interpret the same input identically.

Short-term (CONSIDER items)

  1. Finding 3 — add array.array/wide-memoryview/bare-list cases to the existing dual-backend parametrized test suite alongside the Finding 1 fix, so this class of bug can't silently regress.
  2. Finding 2 — decide and document whether bare list/tuple input should be supported by both backends or neither; align the docstring with whichever is chosen.
  3. Finding 4 — swap PyTuple_GET_ITEM → PyTuple_GetItem (2 sites) if Policy 7 (abi3 adoption) moves forward.
  4. Finding 5 — no immediate action; revisit if Policy 8 (free-threading) moves forward.
  5. Finding 6 — file the two scanner/script bugs against the toolkit upstream.

Longer-term (POLICY)

  1. Policy 7 — evaluate adopting abi3 to collapse the 5-wheel-per-platform build matrix to 1.
  2. Policy 8 — evaluate adopting a free-threading declaration, contingent on first fixing the coupled RACE finding from the prior free-threading-focused review.

C Extension Analysis Report — greenlet

Extension: _greenlet

Scope: src/greenlet/ (entire project)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, c-complexity-analyzer, git-history-analyzer


Executive Summary

The greenlet C++ extension is a mature, carefully engineered codebase with strong RAII discipline (greenlet_refs.hpp) that eliminates entire classes of reference-counting and error-path bugs common in plain C extensions. The most critical finding cluster is a NULL pimpl guard gap: green_traverse (tp_traverse) has a documented shutdown guard (if (!self->pimpl) return 0), but three sibling slot functions — green_clear, green_dealloc, and green_is_gc — dereference pimpl before any such guard, creating GC-driven segfaults at interpreter shutdown. A second high-value finding is that SwitchingArgs._args/_kwargs have been invisible to the GC cycle detector for 55 months — a structural bug that prevents collection of reference cycles through suspended greenlets. The private _Py_NewReference() API used for object resurrection is a CPython-version-fragility risk, and three vestigial structmember.h includes need removal before Python 3.14 deprecation paths tighten further. The free-threading work is substantially complete but the codebase correctly claims this is experimental; the existing FREE_THREADING_ANALYSIS.md covers that dimension in full. The overall assessment is Yellow: a small number of targeted fixes would raise it to Green.


Extension Profile

  • Module: _greenlet (14 C++ source files, unity build via greenlet.cpp)
  • Init style: single-phase (m_size = -1, PyInit__greenlet in PyModule.cpp)
  • Python targets: >=3.10
  • Limited API: no
  • Types defined: 2 (PyGreenlet_Type, PyGreenletUnswitchable_Type)
  • Code generation: hand_written (C++)
  • External tools: cppcheck only — 2 portability findings in platform assembly headers (false positives)
  • Note: FREE_THREADING_ANALYSIS.md at repo root covers FT race conditions (7 RACE, 2 UNSAFE, 6 PROTECT, 6 MIGRATE) independently; this report focuses on the CEXT correctness angle

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety G 0 1 Py_CLEAR on BorrowedObject variable (fragile ownership)
Error Handling G 0 3 std::bad_alloc from new GreenletGlobals escapes noexcept → std::terminate
NULL Safety Y 2 6 green_clear + g_switch_finish NULL deref
GIL Discipline G 0 1 PyInterpreterState_Head() called without GIL in cleanup path
Resource Lifecycle G 0 2 PythonAllocator::allocate() returns NULL without throwing
Module State G 0 4 Static type objects share tp_dict across interpreters
Type Slots R 3 2 green_clear/green_dealloc/green_is_gc missing pimpl guard; SwitchingArgs GC invisible
PyErr_Clear Safety G 0 0 Both PyErr_Clear calls correctly preceded by PyErr_WriteUnraisable
ABI Compliance Y 2 1 _Py_NewReference() private API; structmember.h vestigial includes
Version Compat G 0 5 PyErr_Fetch/PyErr_Restore deprecated 3.12 (TODO exists); GREENLET_314 typo
C/Python Parity N/A — — No dual Python implementation
Complexity G — 3 ThreadState::~ThreadState CC 23; complexity does NOT correlate with bug density

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 7

# Finding File:Line Agents
1 green_clear dereferences pimpl without NULL guard — green_traverse has the guard and documents the shutdown path; green_clear is called by the same GC machinery PyGreenlet.cpp:183 null-safety, type-slot, git-history
2 green_dealloc uses BorrowedGreenlet me(self) and calls me->active(), me->started(), me->main() at line 308 — all route through the unguarded operator->() before the if (self->pimpl) guard appears at line 321 PyGreenlet.cpp:307–321 type-slot
3 green_is_gc calls self->main(), self->active(), and was_running_in_dead_thread() at lines 151/155 via unguarded BorrowedGreenlet::operator->() — tp_is_gc runs before tp_traverse in the GC pipeline, crashing before the traverse guard has a chance to fire (GIL builds only; slot absent under Py_GIL_DISABLED) PyGreenlet.cpp:142–169 type-slot, git-history
4 g_switch_finish unconditionally forms ThreadState& state = *this->thread_state() — MainGreenlet::thread_state() explicitly returns nullptr after thread death; dereferencing NULL to form a C++ reference is UB/crash TGreenlet.cpp:427 null-safety, complexity
5 SwitchingArgs._args and _kwargs (OwnedObject members holding live PyObject* refs) are never visited in Greenlet::tp_traverse and never cleared in Greenlet::tp_clear — a suspended mid-switch greenlet is invisible to the GC cycle detector; gap born 2021-11-06 in 8796b17, ~55 months old TGreenlet.cpp:584–611 type-slot, git-history
6 _Py_NewReference() — private CPython internal used for the tp_dealloc object-resurrection pattern (analogous to iobase_dealloc). CPython changed this function's behavior between 3.12 and 3.13. No public alternative exists; this is a latent break risk on CPython version upgrades PyGreenlet.cpp:267 stable-abi, version-compat
7 structmember.h vestigial includes in 3 files — PyMemberDef is provided by Python.h since 3.12; the green_members[] arrays are empty sentinel-only; removing the includes and empty arrays eliminates a 3.14 deprecation path greenlet.cpp:17, PyGreenlet.cpp:11, PyGreenletUnswitchable.cpp:12 stable-abi, version-compat

Must Fix — Detail

FIX 1 — green_clear missing pimpl guard

Fix:

static int
green_clear(PyGreenlet* self)
{
    Py_CLEAR(self->dict);
    if (!self->pimpl) {   // add this guard — mirrors green_traverse:130
        return 0;
    }
    return self->pimpl->tp_clear();
}

History: The guard was added to green_traverse in commit 8a6085c (2024-09-12) with a comment: "can be NULL at interpreter shutdown time". Neither green_clear, green_dealloc, nor green_is_gc were patched at the same time — a classic sibling-fix gap.


FIX 2 — green_dealloc pre-guard deref

Reachability: green_dealloc at line 326 sets self->pimpl = nullptr before calling delete p at line 327. ~Greenlet() (TGreenlet.cpp:40) also sets pimpl = nullptr, but more importantly it releases SwitchingArgs._args/_kwargs via RAII destructors. If those DECREFs trigger finalizers that INCREF/DECREF the same PyGreenlet, the refcount can drop to zero during delete p, causing a re-entry of green_dealloc. On re-entry, pimpl is already NULL (line 326 ran before delete p), so lines 308-310 crash. The maintainer added the guard at line 321 to handle exactly this re-entry — but placed it too late (after the unguarded dereferences at 308-310). The // In case deleting this... somehow winds up calling back into us comment at line 322 confirms the scenario is known.

Fix: Move the if (self->pimpl) check (currently at line 321) to before line 307, before BorrowedGreenlet me(self) is constructed. All me-> method calls must be inside the guarded block.


FIX 3 — green_is_gc missing pimpl guard

Reachability: tp_is_gc runs before tp_traverse in the GC pipeline — the collector calls tp_is_gc to decide whether to include an object in the candidate set, then calls tp_traverse on included objects. The maintainer documented observing NULL pimpl in tp_traverse at interpreter shutdown (green_traverse:130 comment: "I have seen this at interpreter shutdown time"). If the NULL condition can arise during tp_traverse, it can arise in the same collection pass during the earlier tp_is_gc call, which has no guard.

Fix:

static int
green_is_gc(PyObject* self)
{
    PyGreenlet* g = reinterpret_cast<PyGreenlet*>(self);
    if (!g->pimpl) {   // add guard — same shutdown window as green_traverse
        return 0;
    }
    BorrowedGreenlet me(g);
    // ... existing logic
}

FIX 4 — g_switch_finish NULL deref on thread_state()

Fix:

OwnedObject
Greenlet::g_switch_finish(const switchstack_result_t& err)
{
    assert(err.the_new_current_greenlet == this);
    ThreadState* ts = this->thread_state();
    if (!ts) {
        PyErr_SetString(mod_globs->PyExc_GreenletError,
            "Cannot finish switch: greenlet's thread is dead");
        return OwnedObject();
    }
    ThreadState& state = *ts;
    // ... rest unchanged

FIX 5 — SwitchingArgs GC blind spot

SwitchingArgs::_args and _kwargs are private — access them via the public args()/kwargs()/CLEAR() methods. Add visits to both Greenlet::tp_traverse and Greenlet::tp_clear in TGreenlet.cpp:

int
Greenlet::tp_traverse(visitproc visit, void* arg)
{
    // add: switch_args holds live refs during stack-switch and parent-chain resumption
    Py_VISIT(this->switch_args.args().borrow());
    Py_VISIT(this->switch_args.kwargs().borrow());
    int result;
    if ((result = this->exception_state.tp_traverse(visit, arg)) != 0) {
        return result;
    }
    bool visit_top_frame = this->was_running_in_dead_thread();
    if ((result = this->python_state.tp_traverse(visit, arg, visit_top_frame)) != 0) {
        return result;
    }
    return 0;
}

int
Greenlet::tp_clear()
{
    this->switch_args.CLEAR();  // add — clears both _args and _kwargs
    bool own_top_frame = this->was_running_in_dead_thread();
    this->exception_state.tp_clear();
    this->python_state.tp_clear(own_top_frame);
    return 0;
}

Liveness window: g_switchstack (TGreenlet.cpp:146) explicitly documents that GC can be triggered mid-switch (line 152-155 comment: "Switching away would happen if we trigger a garbage collection"). switch_args on the target is populated before the stack switch and consumed by g_switch_finish after it — during that window, GC can run and these refs are invisible to the cycle detector. A second window exists in the parent-chain resumption path (TUserGreenlet.cpp:553): parent->args() <<= result stores args in the parent before parent->g_switch(), and g_switch() runs Python trace functions while switch_args holds those refs.


FIX 6 — _Py_NewReference() private API

Context: Used for the object-resurrection pattern in tp_dealloc (PyGreenlet.cpp:267). This follows the iobase_dealloc idiom from CPython's own Modules/_io/iobase.c. No public API alternative currently exists. Options:

  • File a CPython issue requesting a public PyObject_Resurrect() function
  • Add a version guard (#if PY_VERSION_HEX < 0x030E0000) and assert-fail with a clear message on unknown versions to detect breakage early
  • Consider restructuring dealloc to avoid resurrection (use a flag instead of refcount manipulation)

FIX 7 — structmember.h vestigial includes

Remove the three #include "structmember.h" lines. Remove or NULL the tp_members slot in PyGreenlet_Type (the empty green_members[] array can be deleted). Verify the build compiles cleanly on Python 3.10–3.14.


Should Consider (CONSIDER) — 18

# Finding File:Line
8 std::bad_alloc from new GreenletGlobals escapes noexcept function greenlet_internal_mod_init() → std::terminate with no Python exception; fix: add catch (std::bad_alloc&) { PyErr_NoMemory(); return NULL; } greenlet.cpp:219
9 UserGreenlet::operator new / BrokenGreenlet::operator new call PythonAllocator<T>::allocate() which returns NULL without throwing — C++ new expression then passes NULL as this to constructor, undefined behaviour; fix: add if (!p) throw std::bad_alloc(); in allocator greenlet_allocator.hpp:36, TGreenlet.hpp
10 greenlet_exceptions.hpp:38-44 debug path: PyObject_Str(typ) can return NULL, then PyUnicode_AsUTF8(NULL) → std::string(nullptr) is UB; affects debug builds only greenlet_exceptions.hpp:38–44
11 dynamic_cast<BrokenGreenlet*>(self->pimpl) result used without NULL check in all 4 PyGreenletUnswitchable_Type getset handlers — a custom __new__ subclass can produce a pimpl that fails the cast PyGreenletUnswitchable.cpp:62,76,88,102
12 PyThreadState_GetDict() return passed unchecked to PyDict_SetItem — returns NULL on allocation failure or non-Python thread context PyModule.cpp:89–92
13 PyTuple_Pack OOM in SwitchingArgs::operator<<= stores NULL into OwnedObject without calling PyErr_NoMemory() — assert(PyErr_Occurred()) fires in debug builds TGreenlet.cpp:384
14 PyDict_Size/PySequence_Length return -1 on error; !(-1) is false in C — silently skips the branch and swallows the exception TGreenlet.cpp:375,378
15 Greenlet::g_switchstack calls thread_state() three times without NULL check; invariant that the thread is alive is implicit rather than enforced at this level TGreenlet.cpp:158,163,165
16 PyInterpreterState_Head() called in the no-GIL zone of the cleanup path; self-documented race (// XXX); fix: replace with greenlet::IsShuttingDown() / Py_IsFinalizing() (GIL-free, already used nearby) TThreadStateDestroy.cpp:53, TThreadState.hpp:465
17 Py_CLEAR applied to a BorrowedObject variable — semantically defined for owned refs; works today because p is the first member, but violates ownership contract and is fragile against layout changes TThreadState.hpp:559–569
18 GreenletGlobals constructor member-initializer list: if a late initializer throws after early ImmortalObject members are constructed, those immortal members leak their Python references (bounded one-time leak; very low risk) TGreenletGlobals.cpp:111 (MIL)
19 Static PyGreenlet_Type and PyGreenletUnswitchable_Type share tp_dict across all interpreter instances — shared-GIL subinterpreters see each other's runtime attribute mutations PyGreenlet.cpp:785, PyGreenletUnswitchable.cpp:128
20 PyErr_Fetch / PyErr_Restore / PyErr_NormalizeException deprecated in 3.12 — 5 call sites; existing TODO (GREENLET_312) comments acknowledge; migration is straightforward (PyErr_GetRaisedException / PyErr_SetRaisedException) but requires dropping Python 3.10/3.11 support greenlet_refs.hpp:1068,1079,1107, TGreenlet.cpp:395, greenlet_exceptions.hpp
21 tp_dictoffset soft-deprecated in 3.12 in favor of Py_TPFLAGS_MANAGED_DICT — still functional through 3.14 PyGreenlet.cpp:806
22 GREENLET_314 typo in version guard — undefined macro evaluates to 0; the c_recursion_depth = 0 branch at line 339 is dead code; delete the inner #if block TPythonState.cpp:339
23 MSVC compatibility header copies CPython static inline internals verbatim without an upper PY_VERSION_HEX bound — if CPython 3.15 changes the copied structs, the module loads but silently corrupts the interpreter greenlet_msvc_compat.hpp
24 check_switch_allowed() call in green_switch/green_throw is guarded by #ifdef Py_GIL_DISABLED; open TODO asks "should we always do this?" — removing the guard is zero-cost hardening PyGreenlet.cpp:390,481
25 Refactor: ThreadState::~ThreadState (CC 23) — extract the ~55-line gc.get_referrers() GC probe at TThreadState.hpp:520–576 into a private helper to make the destructor independently testable and reduce CC from 23 to ~10 TThreadState.hpp:520–576

Tensions

m_size = -1 vs Py_MOD_GIL_NOT_USED: The existing FREE_THREADING_ANALYSIS.md (finding M2) characterizes these as contradictory. The module-state-checker clarifies they are orthogonal: m_size = -1 declares subinterpreter incompatibility; Py_MOD_GIL_NOT_USED declares GIL-free thread safety. A module can have irreducible process-global state (requiring m_size = -1) while being safe without the GIL (using atomics and thread-locals). The pairing is architecturally coherent; M2 should be reframed as "multi-phase init is a prerequisite for subinterpreter support, not a prerequisite for Py_MOD_GIL_NOT_USED."

Complexity vs. bug density: c-complexity-analyzer found that the three FIX-severity NULL-guard bugs (green_clear, green_dealloc, green_is_gc) are in CC 2–7 functions, not in the CC hotspots. The complexity leader (ThreadState::~ThreadState, CC 23) has zero confirmed bugs. This is counterintuitive: in greenlet, bugs arise from subtle NULL/lifetime reasoning in simple functions, not from branching complexity. The implication is that code review effort should prioritize type-slot lifecycle functions over high-CC functions.


Policy Decisions (POLICY) — 14

# Finding
26 m_size = -1 single-phase init is intentional; CPython's import machinery blocks own-GIL subinterpreters at the check_multi_interp_extensions level; three structural obstacles make migration to multi-phase init HIGH difficulty
27 Py_MOD_GIL_NOT_USED with m_size = -1 is NOT a contradiction — orthogonal axes (see Tensions above)
28 mod_globs immortal singleton: ImmortalObject/ImmortalString/ImmortalException members intentionally never freed; teardown path accesses thread_states_to_destroy_lock without GIL from thread-local destructor — immortality is the deliberate solution
29 switching_thread_state dual-mode global: static volatile Greenlet* under GIL (10% speedup, GIL ensures exclusive access), thread_local under Py_GIL_DISABLED — both configurations correct
30 _PyGreenlet_API function-local static void* array: C++11 magic-statics guarantee one-time thread-safe initialization; import lock prevents any client from loading before init completes
31 Stable ABI not feasible: context switching requires writable access to PyThreadState internals that no public API exposes; CPython would need PyThreadState_SaveContext/RestoreContext before this changes
32 Platform switch headers (platform/switch_*.h): 24 files contain zero CPython API calls; all Python-level state access is in the C++ layer above the assembly boundary
33 PythonAllocator<T> stateless class statics (UserGreenlet::allocator, etc.) — stateless wrappers, no cross-interpreter concern
34 _green_dealloc_kill_started_non_main_greenlet at PyGreenlet.cpp:241: Py_INCREF(self.borrow_o()); /* leak! */ is intentional object-resurrection (CPython iobase_dealloc pattern) preventing use-after-free if the greenlet resumes after tp_dealloc
35 PyErr_Clear in clear_deleteme_list teardown loop — both call sites are preceded by PyErr_WriteUnraisable (reports exceptions including MemoryError to sys.unraisablehook before clearing); the pre-existing caller exception is saved/restored by the PyErrPieces RAII wrapper
36 PyErrPieces intentional destruction without restore() in g_calltrace (trace error replaces throw error) and inner_bootstrap (GreenletExit is converted to a result, guarded by PyExceptionMatches)
37 decref-then-steal pattern in TPythonState::operator<< (PyThreadState_GetFrame → Py_XDECREF(frame) → _top_frame.steal(frame)) — semantically borrows the thread state's natural reference; correct and tested, but counter-intuitive
38 CreatedModule (PyObjectPointer<>) has no destructor by design: in single-phase init CPython may already hold the module in sys.modules; a decrement in the exception path would create a double-free
39 G_TOTAL_MAIN_GREENLETS cross-interpreter counter: correct at the synchronization level (atomic under FT); cross-interpreter aggregation is a deliberate consequence of the single-phase init design, acceptable for its testing-utility purpose

Strengths

  • Comprehensive RAII via greenlet_refs.hpp: OwnedObject, BorrowedObject, and ~15 subclass wrappers eliminate virtually all manual INCREF/DECREF bookkeeping. The codebase has zero Py_BuildValue calls, zero PyTuple_SET_ITEM steal patterns, and uses PyModule_AddObjectRef (never-steals) exclusively.
  • Exception state discipline: TExceptionState.cpp save/restore pairs are perfectly symmetric around every slp_switch() call; PyErrPieces RAII is the consistent idiom.
  • GIL discipline: Zero Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS blocks (greenlet never releases the GIL voluntarily — cooperative switching is implicit). Zero PyGILState_Ensure / PyGILState_Release pairs. The cleanup queue defers all Python calls to Py_AddPendingCall callbacks that run with the GIL.
  • Stack lifecycle correctness: TStackState.cpp g_save() correctly uses the temp = PyMem_Realloc() idiom (old pointer preserved on NULL return); ~StackState() destructor correctly frees the stack copy on all paths via embedded RAII members.
  • Free-threading atomics: MainGreenlet::_thread_state as std::atomic<ThreadState*>, G_TOTAL_MAIN_GREENLETS atomic under Py_GIL_DISABLED, switching_thread_state as thread_local under Py_GIL_DISABLED — all correct.
  • Version-branching discipline: greenlet_cpython_compat.hpp defines GREENLET_PY310/311/312/313/314/315 chain that keeps version-specific CPython field access changes organized; no dead guards found below the 3.10 minimum.
  • PyErr_Clear safety: Both calls are in teardown paths preceded by PyErr_WriteUnraisable; neither silently swallows MemoryError or other critical exceptions.
  • Public C API stability: _PyGreenlet_API slot numbering is append-only; all 11 function pointer casts match their header declarations exactly; PyCapsule_New called with NULL destructor correctly (the BSS array needs no free).

Code Removal Opportunities

Checking the deprecated_apis.json code-removal opportunities against greenlet's >=3.10 minimum:

  • PyModule_AddObjectRef (3.10+): Already used exclusively — no PyModule_AddObject steal pattern to remove. ✓ Already modernized.
  • Py_TPFLAGS_MANAGED_DICT (3.12+): Could replace tp_dictoffset in PyGreenlet_Type (CONSIDER 21), saving ~5 lines. Requires #if GREENLET_PY312 guard or bump to 3.12 minimum.
  • PyErr_GetRaisedException/PyErr_SetRaisedException (3.12+): Would replace ~15 lines of PyErr_Fetch/PyErr_Restore/PyErr_NormalizeException across 5 sites (CONSIDER 20). Requires dropping 3.10/3.11 support.
  • PyImport_ImportModuleAttrString (3.14+): Not applicable — greenlet has no PyImport_ImportModule + PyObject_GetAttrString chains.

Relationship to FREE_THREADING_ANALYSIS.md

The existing FREE_THREADING_ANALYSIS.md (21 findings: 7 RACE, 2 UNSAFE, 6 PROTECT, 6 MIGRATE) and this report cover different surfaces with minimal overlap:

  • Overlap: CONSIDER 16 in this report (PyInterpreterState_Head without GIL) is the same as R2/R3 in FREE_THREADING_ANALYSIS.md. CONSIDER 24 (check_switch_allowed guard) is the same as P6 there.
  • This report adds: The NULL pimpl guard cluster (FIX 1-3), the SwitchingArgs GC blind spot (FIX 5), _Py_NewReference stability (FIX 6), the structmember.h removal (FIX 7), and all the RAII/error-path/complexity analysis.
  • Module-state clarification: This report corrects the M2 finding in FREE_THREADING_ANALYSIS.md — m_size = -1 combined with Py_MOD_GIL_NOT_USED is NOT a contradiction; they operate on orthogonal axes.

Both reports should be consulted for a complete picture of greenlet's current correctness posture.


Recommended Action Plan

Immediate (FIX items — 7)

  1. Finding 1 — Add if (!self->pimpl) return 0; guard to green_clear (PyGreenlet.cpp:183). One-line fix; mirrors the existing comment in green_traverse.
  2. Finding 2 — Move the if (self->pimpl) check in green_dealloc before the BorrowedGreenlet construction (PyGreenlet.cpp:307).
  3. Finding 3 — Add if (!g->pimpl) return 0; guard at the top of green_is_gc (PyGreenlet.cpp:142).
  4. Finding 4 — Add NULL check on this->thread_state() in g_switch_finish before forming the reference (TGreenlet.cpp:427).
  5. Finding 5 — Add Py_VISIT/CLEAR for switch_args._args and switch_args._kwargs in Greenlet::tp_traverse and Greenlet::tp_clear (TGreenlet.cpp:584–611). This fixes a 55-month-old GC cycle-detection blind spot.
  6. Finding 6 — Add upper-bound version guard around _Py_NewReference() and file a CPython issue for a public PyObject_Resurrect() API (PyGreenlet.cpp:267).
  7. Finding 7 — Remove three #include "structmember.h" lines; remove green_members[] empty arrays; set tp_members = NULL (greenlet.cpp:17, PyGreenlet.cpp:11, PyGreenletUnswitchable.cpp:12).

Short-term (top CONSIDER items)

  1. Finding 8 — Wrap new GreenletGlobals in a try/catch for std::bad_alloc in the noexcept module init function.
  2. Finding 9 — Add if (!p) { PyErr_NoMemory(); throw std::bad_alloc(); } in PythonAllocator<T>::allocate().
  3. Finding 11 — Add NULL checks on all four dynamic_cast<BrokenGreenlet*> results in PyGreenletUnswitchable.cpp.
  4. Finding 16 — Replace PyInterpreterState_Head() in TThreadStateDestroy.cpp:53 and TThreadState.hpp:465 with Py_IsFinalizing() (GIL-free, already used nearby).
  5. Finding 22 — Delete the dead #if GREENLET_314 inner block in TPythonState.cpp:339.
  6. Finding 24 — Remove the #ifdef Py_GIL_DISABLED guard around check_switch_allowed() in green_switch and green_throw — unconditional hardening with zero cost on GIL builds.

Longer-term (POLICY / structural)

  1. Finding 20 — Migrate PyErr_Fetch/PyErr_Restore/PyErr_Normalize to PyErr_GetRaisedException/PyErr_SetRaisedException when dropping Python 3.10/3.11 support. The existing TODO (GREENLET_312) annotations mark all 5 sites.
  2. Finding 19/26 — Multi-phase init migration (heap types, per-interpreter state) when subinterpreter support becomes a goal. HIGH difficulty due to the three structural obstacles documented in the module-state report. Not recommended without a clear use case.
  3. Finding 25 — Extract the GC referrers probe in ThreadState::~ThreadState into a private helper to reduce CC from 23 to ~10.

Before sharing this report externally: greenlet is an active project with a responsive maintainer. A report of this size should not be filed as a single issue. Reach out first to ask what format they prefer. The FIX 1-4 cluster (NULL guard siblings) is the most compelling single PR — small, safe, well-motivated by the existing green_traverse comment. FIX 5 (SwitchingArgs in GC) is independently motivatable with a reproduction test. Security findings (none found) would need private disclosure.

C Extension Analysis Report — grpc (cygrpc / _cyobservability)

Run: cext_v1 · completed 2026-07-04 Repo: grpc/grpc @ 3e1d845 (master; shallow clone deepened to 300 commits for history analysis)

Extension: grpc._cython.cygrpc (+ grpc_observability._cyobservability)

Scope: src/python/grpcio/grpc/_cython/, src/python/grpcio_observability/grpc_observability/, plus pure-Python grpc//grpc/aio/ consumers where relevant

Agents Run: type-slot-checker, resource-lifecycle-checker, gil-discipline-checker, parity-checker, module-state-checker, version-compat-scanner, refcount-auditor (scoped to hand-written C++), c-complexity-analyzer, stable-abi-checker, error-path-analyzer (scoped to hand-written C++), git-history-analyzer

Agents Skipped: null-safety-scanner, pyerr-clear-auditor (per Cython calibration — generator-emitted patterns; the sole hand-written C++ file was covered by the refcount and error-path agents; grep confirmed 0 maintainer-written PyErr_Clear in scope)

Before sharing externally: see WORKING_WITH_MAINTAINERS.md in the toolkit. Don't drop 46 findings on the grpc maintainers unsolicited — reach out first, and note that Findings 3–4 have security-adjacent impact (credential omission, memory corruption on TLS threads) and deserve responsible disclosure rather than a public issue.


Executive Summary

cygrpc is a mature, disciplined Cython extension: GIL handling around core calls is near-exemplary, __cinit__/__dealloc__ pairing is consistent, and interpreter-finalization hazards are explicitly engineered (often with issue references in comments). The FIX-class risk is concentrated in three zones, all confirmed by independent agents and by commit history:

  1. The new private_key_signing feature (TLS private-key offload, merged 2026-03-09 via PR #41701): the review bot and human reviewers flagged the exact bugs that shipped — a per-signer threading.Event leak, a destructor that can leave a pending exception on a gRPC-core thread and hang interpreter shutdown, and uninitialized result structs that can lead to a wild Py_DECREF on a TLS thread. Zero follow-up commits have touched the directory since merge.
  2. Silently-unauthenticated RPCs: _get_metadata's except -1 with gil declaration means a Python exception in a metadata-plugin callback is reported to gRPC core as successful synchronous completion with zero metadata — the RPC proceeds without credentials, and the error indicator is left set on a core thread.
  3. Abandoned twin sites: the aio server's sync-handler bridge (_SyncServicerContext) stopped tracking the sync context API in 2021-10 (commit-level proof) and its abort() doesn't raise; channelz/csds admin queries have leaked core allocations per call since 2018/2021.

Recommended order: patch the ~10-line private-key-signing cluster (Findings 1–3) and _get_metadata (4) immediately; then the one-token server error-path bug (5) and the aio bridge (6–7); batch the leak fixes (8–14) as one cleanup PR.

Extension Profile

  • Module: cygrpc — hand-written Cython: cygrpc.pyx + 61 .pxi + 1 .pxd (63 files, ~6,711 lines) + hand-written C++ CPython wrapper private_key_signing/private_key_signer_py_wrapper.cc/.h (~150 lines); secondary module _cyobservability (.pyx/.pxd, ~450 lines; its 17 C++ files don't touch the CPython API)
  • Init style: Cython 3.1.1-generated PEP 489 multi-phase structure with single-copy semantics; all state in C statics by design (process-wide C-core binding)
  • Python targets: >=3.10, <=3.14 · Limited API: no (50 per-version wheel targets)
  • Types defined: all Cython-generated (cdef class); 0 hand-written PyTypeObject
  • External tools: cppcheck ran (4 findings, all false positives — C++ headers parsed as C); clang-tidy skipped (no compile_commands.json)
  • History: local git truncated at 2026-04-24 (300 commits); pre-boundary history recovered via GitHub REST API

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 1 4 destroy_event_ INCREF'd at Create, never DECREF'd (leak per signer)
Error Handling Y 3 5 Pending exception across PyGILState_Release; garbage-struct Py_DECREF trigger
NULL Safety — (skipped) – – Cython-emitted checks; no hand-written surface beyond covered C++
GIL Discipline Y 3 9 _get_metadata except -1 → core sees success, sends RPC without credentials
Resource Lifecycle R 7 6 channelz leaks every query result string since 2018
Module State Y 1 6 Non-daemon destroy-event waiter can hang interpreter shutdown
Type Slots R 4 9 Error path raises AttributeError: c_call_error instead of InternalError (live-verified)
PyErr_Clear Safety — (skipped) – – 0 maintainer-written PyErr_Clear in scope
ABI Compliance G 0 2 No abi3 claim; hand-written C++ is 100% stable-ABI clean; 2 small blockers if ever wanted
Version Compat G 0 7 ~55 lines dead pre-3.10 compat code; asyncio-policy removal deadline ~3.16 (tracked)
C/Python Parity Y 2 3 _SyncServicerContext missing 6 sync-contract methods; abort() doesn't raise
Complexity G – 10 488 functions, avg CC 1.98, zero >100 LOC; risk concentrated in _call (13 params, CC 13)
Git History Y (3)* 3 PR #41701 review items dropped at merge; twin-site gap commit-proven

G = No FIX | Y = 1–3 FIX | R = 4+ FIX. *git-history FIX items corroborate Findings 1, 2, 6 below (merged, not double-counted).


Findings by Priority

Global numbering: FIX 1–14, CONSIDER 15–38, POLICY 39–46. Duplicate findings across agents are merged; the agent list shows independent confirmations.

Must Fix (FIX) — 14

# Finding File:Line Agents
1 ~PrivateKeySignerPyWrapper never DECREFs destroy_event_ (INCREF'd in Create()); one threading.Event + machinery leaked per signer, multiplied by SSLChannelCredentials.c() creating a fresh signer per call. PR #41701's reviewer requested exactly this DECREF; the suggestion was dropped and no commit has touched the directory since merge. _cygrpc/private_key_signing/private_key_signer_py_wrapper.cc:36,42-50 type-slot, resource-lifecycle, gil, refcount, error-path, git-history
2 Destructor failure path: PyObject_CallMethod(destroy_event_, "set", "()") result unchecked (// crash if result is nullptr? - discussing shipped to master) — on failure the pending exception survives PyGILState_Release on a gRPC-core thread (poisons the next Python entry there) and the non-daemon watchdog thread waiting on the never-set event (no timeout) hangs interpreter shutdown. Fix: PyErr_WriteUnraisable, guarantee the event is set (or signal via a C primitive), and fold in Finding 1's DECREF. private_key_signer_py_wrapper.cc:46-49 + _cygrpc/private_key_offload.pyx.pxi:106-114 error-path, gil, module-state, type-slot, git-history
3 PrivateKeySignerPyWrapperResult / AsyncResult have no default member initializers. Reachable trigger: the Cython sign wrapper is noexcept; an exception outside its try (e.g. MemoryError constructing OnCompleteWrapper, or a secondary exception in its except handler) returns a default-constructed struct — Sign() branches on indeterminate is_sync and the handle destructor can Py_DECREF a garbage pointer on a TLS thread (crash/heap corruption). Reviewers flagged the call-site half (fixed); the struct half shipped. Fix: is_sync = true, pointers = nullptr, validity guard in Sign(). private_key_signer_py_wrapper.h:43-54 (+ .cc:62) error-path, refcount, git-history
4 _get_metadata metadata-plugin callback is declared except -1 with gil, but gRPC core neither checks nor clears the Cython error protocol: a Python exception in the plugin leaves the thread's error indicator set on a core thread and the -1 return is read by core as "synchronous success, no metadata" — the RPC is silently sent without credentials. Dense fix history (2019/2021/2024) patched around this declaration without ever fixing the return protocol. _cygrpc/credentials.pyx.pxi:38-66 gil (git-history corroboration)
5 Server request-call error paths format the message with non-existent self.c_call_error (local is c_call_error) — live-verified to raise AttributeError: 'Server' object has no attribute 'c_call_error' instead of InternalError with the core diagnostic. One-token fix, two sites. _cygrpc/server.pyx.pxi:117,136 type-slot
6 _SyncServicerContext (context given to sync handlers on a grpc.aio server) is missing 6 methods of the sync grpc.ServicerContext/RpcContext contract: is_active, cancel, abort_with_status, trailing_metadata, code, details. Commit-level proof of abandonment: the bridge tracked the context API through 2021-03 (83b19b2e) and missed the very next addition 2021-10 (09a55f26). Idiomatic while context.is_active(): handlers crash with AttributeError on the aio server. All six have trivial delegation targets. _cygrpc/aio/server.pyx.pxi:292-356 parity, git-history
7 _SyncServicerContext.abort() calls future.exception(), which returns the AbortError instead of raising it — violating the documented guarantee (grpc/__init__.py:1233: "an exception is always raised"); handler code continues executing past context.abort(...). One-line fix: future.result() (matches the adjacent send_initial_metadata bridge). See POLICY 44 for the regression-test caution. _cygrpc/aio/server.pyx.pxi:301-309 parity
8 All 7 channelz query functions return core-allocated C strings that are never gpr_free'd — unbounded per-query leak on any process with channelz polling enabled. File untouched since 2018-12-11; never audited. _cygrpc/channelz.pyx.pxi:16-71 resource-lifecycle (git-history: original-sin age)
9 dump_xds_configs never unrefs the grpc_slice returned by core — per-CSDS-query leak (same class as Finding 8; file has one commit ever, 2021). _cygrpc/csds.pyx.pxi:16-21 resource-lifecycle
10 aio execute_batch on a failed/rejected batch skips tag.event() cleanup: the CallbackWrapper self-INCREF leaks and op buffers never get un_c()/gpr_free. Routine on cancelled RPCs. History shows no leak fix ever landed on either the sync or aio side — original defect, not a fix gap. _cygrpc/aio/callback_common.pyx.pxi:77-103 type-slot, resource-lifecycle, git-history
11 aio call credentials are never released after grpc_call_set_credentials (per secure RPC), and method_slice leaks on the error path of call creation. _cygrpc/aio/call.pyx.pxi:129-134 resource-lifecycle
12 Sync-side manually INCREF'd tags leak whenever core rejects grpc_call_start_batch / grpc_server_request_*_call (e.g. invalid metadata) — the DECREF normally happens in _interpret_event, which never sees the tag. Same fix pattern as Finding 10. (Type-slot rated this CONSIDER on frequency grounds — see Tensions.) _cygrpc/channel.pyx.pxi:128-136, call.pyx.pxi:24-36, server.pyx.pxi:97-137 resource-lifecycle, type-slot
13 _store_c_metadata / tag.prepare leak partially-built C metadata/op arrays when a mid-loop TypeError occurs (common user error: non-bytes metadata value). _cygrpc/metadata.pyx.pxi:27-50, tag.pyx.pxi:58-65 resource-lifecycle
14 ComputeEngineChannelCredentials has no __dealloc__ — the owned _call_creds core handle is never released; and a second .c() call double-consumes released credentials (use-after-free risk). _cygrpc/credentials.pyx.pxi:507-524 type-slot, resource-lifecycle

Should Consider (CONSIDER) — 24

# Finding File:Line
15 CompletionQueue.__dealloc__ pumps grpc_completion_queue_next with an infinite deadline while holding the GIL (only such site) — deadlock-capable if a pending tag needs a GIL-requiring core callback (metadata plugin, cert fetcher, key signer). History: this exact zone had 5 GIL-policy adjustments 2016→2025 and remains fragile. Complexity: extract the duplicated _next poll loop into _poll_once first, then fix once (see Tensions). _cygrpc/completion_queue.pyx.pxi:145-158
16 ~AsyncSigningHandlePyWrapper runs PyGILState_Ensure + Py_DECREF on foreign threads with no finalization guard and is not covered by the destroy-event interpreter-pinning design; its safety w.r.t. the signer currently depends on tsi_ssl_handshaker member declaration order; the inherited enable_shared_from_this (the intended pin) is never used. private_key_signer_py_wrapper.cc:80-85, .h
17 Redundant manual Py_INCREF/Py_DECREF pin of _private_key_signer: the balancing DECREF lives in __dealloc__, which the INCREF itself prevents from running — any signer→credentials reference cycle becomes permanently uncollectable. History shows the pin was kept deliberately ("This gets passed around C++…"); the C++ side already owns its references, so the pair should be deleted. _cygrpc/credentials.pyx.pxi:199-205
18 CompletionContext::OnComplete has no exactly-once guard (user double-completion, or completion racing Cancel(), invokes core's AnyInvocable twice); on_sign_complete can fire synchronously inside Sign(), violating the contract at include/grpc/private_key_signer.h:79. private_key_signer_py_wrapper.cc
19 C++ exceptions can cross noexcept/no-except+ FFI boundaries in Create/Sign/MakeStringForCython (std::terminate; plus a 2-ref INCREF leak if make_shared throws in Create); py_user_cancel_fn_ is Py_DECREF'd unguarded in one path but NULL-checked in Cancel() — use Py_XDECREF consistently. private_key_signer_py_wrapper.cc
20 Channel.__cinit__ partial-init: a TypeError from _ChannelArgs leaks two core completion queues and one grpc-init count. _cygrpc/channel.pyx.pxi:510-529
21 Sync _call leaks the grpc_call handle if a Python exception hits mid-setup; the function is also the codebase's top complexity target (13 params, CC 13, its own TODO #14569: could be a lot simpler) — a _CallSpec bundle + consolidated try/except cleanup fixes both structurally. _cygrpc/channel.pyx.pxi:220-295
22 Composite credentials leak earlier core handles when a later constituent's .c() raises. _cygrpc/credentials.pyx.pxi:142-156,257-267
23 Server-side SSL credential constructors (grpc_ssl_server_credentials_create_with_options et al., plus the new TLS-provider calls at :217/:234) run with the GIL held, while the client-side analogs received with nogil deadlock fixes (#34712, 2023; #36266, 2024). Verify against the #34712 root cause; the fix train stopped before the server side. _cygrpc/credentials.pyx.pxi:346-393
24 Incomplete 2026-06 fix 6152023d (custom interceptor exceptions): the five async accessors of InterceptedUnaryUnaryCall (initial_metadata, trailing_metadata, code, details, debug_error_string) retain the exact pre-fix pattern the commit eliminated from the sync accessors. grpc/aio/_interceptor.py:427-475
25 _has_fd_monitoring is a process-wide latch for a per-event-loop property, written under the GIL and read in the GIL-less poller loop (present-day data race, wrong under multiple loops). _cygrpc/aio/completion_queue.pyx.pxi:22,52-58
26 Observability: _flush_census_data builds Python objects while holding the core-shared g_census_data_buffer_mutex (blocks core telemetry hot path); the CV predicate is set without the mutex at :358-363 (missed-wakeup + C-level race); the non-daemon export thread hangs process exit if observability_deinit is skipped. _cyobservability.pyx:330-363, 95-103
27 gevent: INCREF'd tuples sit in a file-scope C++ queue — invisible to GC and abandoned (leaked) at shutdown. _cygrpc/grpc_gevent.pyx.pxi:31-46,84-112
28 aio execute_batch failure path passes an uninitialized grpc_event struct to tag.event() (fold into Finding 10's fix). _cygrpc/aio/callback_common.pyx.pxi:101-103
29 PollerCompletionQueue.__cinit__ starts the poller thread before member initialization completes. _cygrpc/aio/completion_queue.pyx.pxi:68-83
30 Unconditional Py_DECREF(self.method) in __dealloc__ is unbalanced if __cinit__ argument coercion failed (and the manual pair is redundant — Cython manages cdef object attributes); SSLChannelCredentials.__dealloc__'s guarded pattern is the correct template. _cygrpc/channel.pyx.pxi:119-120, server.pyx.pxi:30-35
31 ~55 lines of dead pre-3.10 compat code: hasattr(os, 'register_at_fork') guard (3.7+), the whole contextvars import-probe fallback (thread.pyx.pxi), try: import asyncio whose fallback can't work anyway. _cygrpc/fork.pyx.pxi:99, thread.pyx.pxi, cygrpc.pyx:26-32
32 Cython pin cython==3.1.1 (6 files) predates Python 3.14.0 final — bump to latest 3.1.x. setup.py:586 et al.
33 The only two limited-API blockers, worth fixing on their own merits: PyBytes_GET_SIZE struct-macro (→ len()), and cdef class AioRpcStatus(Exception) builtin base. _cygrpc/private_key_offload.pyx.pxi:16,38,77; aio/rpc_status.pxd.pxi:17
34 Typing surface: cygrpc.pyi is a catch-all __getattr__ -> Any (typo-transparent), no py.typed marker anywhere, _cyobservability has no stub at all. _cython/cygrpc.pyi
35 AioRpcError breaks under type(e)(str(e)) reconstruction (the pytorch pattern, grpc #38713); the guard added to sync _InactiveRpcError in f3291e01 has no aio twin. LOW confidence. grpc/aio/_call.py:73-93
36 Complexity refactor targets beyond Finding 21: _get_metadata (closure over C locals at the site of Finding 4), duplicated _next poll loops (extract before Finding 15's fix), aio _poll (hosts the _shutdown-flag race), _close/Server.destroy sole-teardown paths. Codebase-wide complexity is otherwise excellent (avg CC 1.98). see c-complexity report
37 Fork support: aio channels don't participate in fork handling at all (documented-unsupported, but silently divergent from the newly-hardened sync path); the new _cancel_all_calls drain loop and _close share the unbounded-pump shape of Finding 15. _cygrpc/fork.pyx.pxi, channel.pyx.pxi:520
38 CallbackWrapper keeps the event loop alive only transitively via future._loop; module-state hygiene: executable assignments duplicated between grpc_gevent.pxd.pxi and .pyx.pxi; g_interrupt_check_period_ms C global mutated by gevent init. _cygrpc/aio/callback_common.pyx.pxi:38; grpc_gevent.pxd.pxi:16-17

Tensions

  • Finding 12 severity — resource-lifecycle: FIX (leak on a user-triggerable path); type-slot: CONSIDER (core rejections are rare). Resolution: keep FIX — the fix is the same cheap pattern as Finding 10, so the frequency debate doesn't change the action.
  • Finding 15 fix shape — gil-discipline wants the one-line with nogil on the dealloc pump; type-slot warns against running arbitrary Python under tp_dealloc at all; complexity notes the poll loop is already duplicated verbatim and a third copy would ossify it. Resolution: extract _poll_once, then apply the nogil fix in one place.
  • Module state vs. migration — module-state confirms 13 mutable PyObject globals + 11 C statics and single-copy init, but recommends keeping the architecture: gRPC C-core is process-scoped, so per-module state would be a fiction (POLICY 41). No agent disagrees.
  • Finding 17 — history shows the INCREF pin was kept deliberately at review time; refcount analysis shows it is both redundant (C++ owns its refs) and actively harmful (uncollectable cycles). The in-code comment should not survive the fix.

Policy Decisions (POLICY) — 8

# Finding
39 Deallocs for Server/AioServer/Channel intentionally do not destroy core objects (documented TODOs); teardown correctness rests entirely on users calling close()/destroy(). Acceptable contract, but it's why Findings 8–14 leak classes matter.
40 The grpc_init/grpc_shutdown refcount pairing is distributed across every core-owning __cinit__/__dealloc__ by convention. It holds today (audited, including exception paths), but each new lifecycle class re-implements it; an RAII-style shared base would concentrate the invariant.
41 Process-singleton architecture (_global_aio_state, _fork_state, gevent globals, import-time SSL-roots/absl-log registration) and no-subinterpreter posture are correct by design for a process-wide C-core binding; imports in a second interpreter fail cleanly. Keep; don't attempt PyModule_GetState migration.
42 Stable ABI: no abi3 claim exists; hand-written C++ is 100% stable-ABI clean; only 2 source blockers (Finding 33). Recommendation: defer abi3 (Cython limited-API immaturity, hot-path perf risk, 3.12 vectorcall floor vs 3.10 minimum, free-threaded wheels outside abi3 anyway); land Finding 33 + an allowed-to-fail limited-API canary CI build; re-evaluate at the 3.12 floor (~2027), where abi3 collapses ~50 wheel targets to ~10.
43 asyncio policy APIs used in aio/common.pyx.pxi:177-237 are deprecated in 3.14, removal ~3.16 — already version-guarded and tracked (#39518, #40748); treat as a scheduled deadline, not a defect.
44 The aio abort() zone has a two-time revert history (#34525 applied → reverted → reapplied → re-reverted, 2024-03). Any fix to Finding 7 must ship with the regression test that killed #34525.
45 The SSL roots-override callback returns GRPC_SSL_ROOTS_OVERRIDE_OK with an unset out-pointer on internal error (security.pyx.pxi:20-33) — define and document the failure contract.
46 Review-deferral hygiene: PR #41701 merged with a literal // crash if result is nullptr? - discussing comment and two dropped reviewer-requested fixes, then received zero follow-ups in 4 months. Process fix: convert in-review deferrals into tracked issues before merge.

Strengths

  • GIL discipline is genuinely strong (gil-checker's words): every core→Python callback is with gil, every blocking core call releases the GIL (Finding 15 is the only exception found), and _next's signal handling was verified correct against Cython 3.2.5's declaration of PyErr_CheckSignals.
  • Finalization hardening with receipts: shutdown_grpc_aio checks sys.is_finalizing() citing #38679/#33342/#36655 and already handles Python 3.14's PythonFinalizationError; the metadata-plugin _destroy uses a C mutex + atexit sentinel to never take the GIL after teardown.
  • Guarded deallocs are the norm — partial-init tolerance in Call, SSLSessionCacheLRU, ServerCredentials, RPCState, _AioCall, _ChannelArgs (frees its gpr_malloc array even when the fill loop raised mid-way).
  • Coherent tag-ownership protocol applied consistently (INCREF before handing to core, DECREF in _interpret_event) — Findings 10/12 are gaps in an otherwise sound protocol, not an unsound protocol.
  • Zero dangling API references: all ~90 consumer-imported cygrpc names verified present; all 33 sync-side .pxi includes wired; StatusCode tables bijective; the aio-native _ServicerContext fully implements its 21-method ABC.
  • Version hygiene: no private _Py* APIs anywhere, no dead PY_VERSION_HEX guards in C++, deprecations pre-tracked with issue numbers; hand-written C++ is stable-ABI clean.
  • Exceptionally trim code: 488 functions, average complexity 1.98, zero functions over 100 lines; fixes dominate features 9:6 in recent history with focused (non-shotgun) churn.
  • The private-key-offload interpreter-pinning idea (non-daemon thread + destroy event keeping Python alive for C++-side destruction) is a thoughtful solution to the Py_DECREF-after-Py_Finalize crash class — Findings 1–3 are implementation gaps in a sound design, and OnCompleteWrapper's weak_ptr guard correctly prevents use-after-free on late sign completions.

Code Removal Opportunities

The deprecated_apis.json code_removal_opportunities entries (PyModule_AddType, PyImport_ImportModuleAttrString chains, etc.) are N/A — cygrpc has no hand-written module-init C at all. Real removal opportunities found instead:

  • ~55 lines of dead pre-3.10 compat code (Finding 31): register_at_fork hasattr guard, the entire thread.pyx.pxi contextvars fallback, the asyncio import-probe.
  • ~7 lines: the redundant Py_INCREF/Py_DECREF signer pin (Finding 17) — removal is also the bug fix.
  • 2 duplicated executable assignments in grpc_gevent.pxd.pxi (Finding 38).

Recommended Action Plan

Immediate (FIX items)

  1. Private-key-signing cluster — Findings 1, 2, 3 (~10-line diff in one file pair): DECREF destroy_event_; PyErr_WriteUnraisable + guaranteed event-set in the destructor; default member initializers + Sign() validity guard. Fold in CONSIDER 16/18/19 while the file is open. Security-adjacent (TLS-thread memory corruption) — coordinate disclosure.
  2. Finding 4 — make _get_metadata report failure to core correctly (call the callback with an error status instead of relying on except -1); security-adjacent (silent credential omission).
  3. Finding 5 — one-token fix (self.c_call_error → c_call_error), two sites.
  4. Findings 6–7 — complete the _SyncServicerContext bridge (6 delegating methods) and change future.exception() → future.result(); include the #34525 regression test (POLICY 44).
  5. Findings 8–14 — one "extension leak cleanup" PR: channelz gpr_free, csds slice unref, failed-batch cleanup (aio + sync twins together, per git-history's negative result), aio call-creds release, metadata partial-build cleanup, ComputeEngineChannelCredentials.__dealloc__.

Short-term (CONSIDER items)

  1. Finding 15 via Finding 36: extract _poll_once, then release the GIL in the dealloc pump.
  2. Findings 16–19: harden the C++ wrapper (finalization guard, wire the shared_from_this pin, exactly-once completion, except + / catch-all at FFI boundaries).
  3. Finding 23: verify server-side credential constructors against the #34712 deadlock root cause; apply with nogil if implicated.
  4. Findings 20–22, 25–30: partial-init and race cleanups as opportunistic fixes.
  5. Findings 24, 35: finish the interceptor fix's async half; add the AioRpcError reconstruction guard.
  6. Findings 31–34: dead-code removal, Cython pin bump, the two limited-API blockers, py.typed decision.

Longer-term (POLICY)

  1. POLICY 42: limited-API canary CI now; abi3 re-evaluation at the 3.12 floor (~2027).
  2. POLICY 40/41: RAII-style base for init pairing if new lifecycle classes keep appearing; document the no-subinterpreter posture.
  3. POLICY 43: schedule the asyncio-policy migration before Python 3.16.
  4. POLICY 45/46: define the roots-override failure contract; convert in-review deferrals to tracked issues before merge.

Per-Agent Reports

Full detail (code excerpts, per-path accounting, verification artifacts) in reports/cext_v1/agents/: type-slot-checker.md · resource-lifecycle-checker.md · gil-discipline-checker.md · parity-checker.md · module-state-checker.md · version-compat-scanner.md · refcount-auditor.md · c-complexity-analyzer.md · stable-abi-checker.md · error-path-analyzer.md · git-history-analyzer.md — preflight profile and external-tool baseline in reports/cext_v1/preflight/.

Run notes: agents ran with Cython-mode triage; one finding (5) was verified with a live Cython build; git history beyond 2026-04-24 was recovered via the GitHub REST API due to the shallow clone; several agents were interrupted by session limits and resumed from transcript without loss.

C Extension Analysis Report

matplotlib native extensions · branch freethreaded_fixes · generated by cext-review-toolkit:explore All scanners run under python3t.exe (CPython 3.15.0b3, free-threaded). Companion free-threading deep-dive: FREETHREADING_ANALYSIS.md.

Extension: matplotlib native extensions

Scope: src/ (27 C/C++ source & header files, 209 functions)

Agents Run: extension-discovery + central scanners (refcounts, error-paths, null-checks, gil-usage, module-state, type-slots, pyerr-clear, resource-lifecycle, version-compat, format-strings, complexity, history) + external tools (cppcheck) + deep agents: parity-checker, type-slot-checker, resource-lifecycle-checker + targeted manual verification

Executive Summary

matplotlib's native layer is a pybind11 codebase (8 modules: ft2font, _backend_agg, _image, _path, _tri, _qhull, _tkagg, _c_internal_utils, plus the Obj-C _macosx). Because pybind11 manages reference counts and error propagation, the classic C-API defect classes are essentially absent — refcount, NULL-check, type-slot, PyErr_Clear, format-string and version-compat scanners all returned zero, and their few non-zero hits proved to be false positives (documented below). The real defects live where pybind11's guarantees stop: C++ constructors/factories that new before they can throw. The headline bug — independently found by two agents — is that PyFT2Font_init leaks the object and an open file on any failed FT2Font(...) construction (e.g. a missing or corrupt font file), a per-call leak on a routine error path. A second real defect is an unchecked calloc with a possible integer overflow in FT2Image. Beyond those, findings are OOM-only leaks, subinterpreter blockers (global _ft2Library, static caches, single-phase init), notable .pyi stub drift (four stubs are empty; one advertises the wrong type), and the dominant systemic risk — thread-safety under the project-wide py::mod_gil_not_used() opt-in, analyzed in the companion report. The extension is otherwise healthy and well-factored (avg cyclomatic 3.9, no complexity hotspots; correct FreeType/qhull/buffer RAII), so the actionable list is short and specific.

Extension Profile

  • Modules: 8 pybind11 + 1 Obj-C (_macosx, outside the C/C++ scanners)
  • Init style: single-phase (PYBIND11_MODULE), all declaring py::mod_gil_not_used()
  • Python targets: >=3.12
  • Limited API / abi3: no (not claimed)
  • Code generation: mixed (hand-written pybind11 wrappers + vendored C/C++ — Agg, qhull libqhull_r)
  • Type stubs: .pyi present for every module (several are empty — see Finding 7)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety 🟢 G 0 0 pybind11 RAII manages all refcounts
Error Handling 🟢 G 0 0 4 scanner hits are internal _tri sentinels (false positives)
NULL Safety 🟡 Y 1 0 FT2Image calloc unchecked + width*height overflow (ft2font.cpp:22)
GIL Discipline 🟡 Y 0* 1 All modules assert mod_gil_not_used() unsynchronized — see companion report (1 CRITICAL race)
Resource Lifecycle 🟡 Y 1 3 PyFT2Font_init factory leaks object + fd on failed construction (ft2font_wrapper.cpp:521)
Module State 🟡 Y 0 2 Global _ft2Library + write-once statics block subinterpreters
Type Slots 🟢 G 0 1 pybind11 slots correct; Python refs in C++ members are GC-invisible
PyErr_Clear Safety 🟢 G 0 0 No masked exceptions
ABI Compliance 🟢 G 0 0 abi3 not claimed; infeasible with FreeType/Agg/qhull coupling (intentional)
Version Compat 🟢 G 0 0 Modern (>=3.12); no deprecated C-API in use
C/Python Parity 🟡 Y 0 1 4 empty .pyi stubs; count_bboxes_overlapping_bbox mistyped
Complexity 🟢 G – 0 No hotspots; avg cyclomatic 3.9, max nesting 5

* The free-threading data race is a genuine must-fix but is scoped to, and detailed in, FREETHREADING_ANALYSIS.md (Finding 1, CRITICAL) to avoid double-reporting; counted here as CONSIDER (Finding 3) with a cross-reference.

G = no FIX findings · Y = 1–3 FIX · R = 4+ FIX

Findings by Priority

Global, non-restarting numbering. "Agents" notes cross-validation.

Must Fix (FIX) — 2

# Finding File:Line Agents
1 PyFT2Font_init is a pybind11 factory that news a PyFT2Font (line 521) but can throw before returning it, so pybind11 never adopts the pointer and it leaks — on every failed construction. Reachable triggers: io.open fails on a bad path (:546), non-binary file-like arg (:556), or FT_Open_Face rejects a corrupt/truncated/unsupported font (:563 via FT_CHECK). On the :563 path an already-open file object (py_file) is stranded too (memory always; fd on the invalid-font path). Fix: hold it in std::make_unique<PyFT2Font>(...) and return self.release(); only on success (unwind is then clean — ~PyFT2Font→close() no-ops on face==nullptr). ft2font_wrapper.cpp:521-563 (registered :1589) type-slot-checker + resource-lifecycle-checker (independent cross-validation)
2 FT2Image constructor calls calloc(width * height, 1) with no NULL check, and width * height (both unsigned long) can integer-overflow → under-allocation then out-of-bounds writes in draw_bitmap/draw_rect_filled, or a NULL-deref on allocation failure. Fix: overflow-check the product and verify the result, throwing instead of returning a NULL buffer. ft2font.cpp:21-24 (writes :50-64, :79-83) null-safety (manual — scanner missed the C++ member-init calloc)

Should Consider (CONSIDER) — 11

# Finding File:Line
3 Shared global FreeType library _ft2Library (non-static, so the module-state scanner missed it): every FT_Open_Face/FT_Done_Face mutates it unlocked → a data race under mod_gil_not_used() and a subinterpreter blocker. Full analysis + fix in FREETHREADING_ANALYSIS.md (Finding 1, CRITICAL). ft2font.cpp:19, 200, 223
4 RendererAgg constructor leaks pixBuffer (and alphaBuffer) if a later new[] throws (bad_alloc, reachable via a pathological large dpi → hatch_size²). A throwing ctor means ~RendererAgg never runs. Fix: hold the buffers in std::vector/unique_ptr. (OOM-only → LOW.) _backend_agg.cpp:44, 52, 66
5 FT_Glyph handle leak between FT_Get_Glyph and glyphs.push_back — if the push_back (or an earlier xys.push_back) throws bad_alloc, the in-flight thisGlyph is never FT_Done_Glyph'd. Fix: RAII-guard thisGlyph, or glyphs.reserve() before the loop. (OOM-only → LOW.) ft2font.cpp:482→506, 558→559
6 Reference cycles through Python objects stored as C++ members are invisible to the cyclic GC. pybind11 installs no tp_traverse/tp_clear for C++-member py::objects, so a cycle through FT2Font.py_file/fallbacks/image (or the persistent numpy arrays in _tri types) leaks the whole cycle. Lifetime itself is safe (the py::list fallbacks owns the children; the raw FT2Font*/char_to_font are borrowed views — no use-after-free). Inherent pybind11 limitation (pybind/pybind11#1029). ft2font_wrapper.cpp:359-361; ft2font.h:185,189-190; tri/_tri.h:306-316
7 .pyi stub drift. (a) Wrong types that break type-checkers/keyword use: _path.count_bboxes_overlapping_bbox is typed Sequence[BboxBase] but the C++ takes an ndarray (real caller passes np.atleast_3d(...)); _tri.Triangulation.__init__ param correct_triangle_orientation vs C++ ...orientations; FT2Font.draw_glyph(s)_to_bitmap/_layout/load_glyph have kw-only / renamed params vs the stub; Win32 HWND capsules typed as int. (b) Empty/partial stubs (ship despite py.typed): _qhull.pyi, _image.pyi, backends/_backend_agg.pyi, backends/_macosx.pyi are 0 bytes; _path.pyi types only 2 of 14 functions. Runtime-safe, but degrades the advertised typed API. _path.pyi:8; _tri.pyi:31; ft2font.pyi:245-291; _qhull.pyi/_image.pyi/backends/_backend_agg.pyi/backends/_macosx.pyi (empty)
8 Write-once static mutable state (subinterpreter blockers). _tkagg caches Tk/Tcl entry points in file-scope statics via dlsym; ft2font has a static FT_Outline_Funcs table. Safe for free-threading (write-once) but process-global — must move to per-module state for PEP 489 / subinterpreter support. _tkagg.cpp:90,91,94,95; ft2font.cpp:149
9 Single-phase init across all modules — combined with the globals above, blocks subinterpreter support. all _wrapper.cpp PYBIND11_MODULE lines
10 QhullInfo manages resources (qhT*, FILE*) but declares no copy-constructor / operator= (Rule of Three). Never copied today, so no live bug, but an accidental copy would double-fclose/double-free. Recommend = delete on both. (Destructor itself is correct — see Strengths.) _qhull_wrapper.cpp:112-136
11 appid CoTaskMemFree leak if py::cast throws before the free (Windows-only, one-shot, OOM). Move the free ahead of the cast or use a scope guard. _c_internal_utils.cpp:108-109
12 LayoutItem::ft_object is a borrowed PyFT2Font* exposed via def_readonly with no keep-alive — a LayoutItem outliving its font would dangle. Transient, private (_layout) API → safe today; guard only if LayoutItems are ever stored. ft2font_wrapper.cpp:1382, 1566
13 mod_gil_not_used() asserted before it is enforced — no std::mutex/py::scoped_critical_section anywhere in src/. Until Finding 3 is guarded, the thread-safety claim on all 8 modules is unsound. project-wide

Tensions

  • Subinterpreters vs. simplicity (Findings 8–9): module-state-checker flags single-phase init and process-global state, but the codebase is small and low-complexity and there is no stated subinterpreter goal. Free-threading (the active effort) does not require multi-phase init — only synchronization of the shared state.
  • mod_gil_not_used() vs. current reality (Finding 13): the C-level thread-safety claim is ahead of the C-level guarantees; the Python-layer thread-keyed font cache only partially compensates.
  • Ownership vs. GC-visibility (Finding 6): the double bookkeeping of fallbacks (owning py::list + borrowed FT2Font*) is correct for lifetime but is exactly what makes cycles GC-invisible; fixing GC-visibility would mean custom type slots, adding complexity this otherwise-clean code avoids.

Policy Decisions (POLICY) — 2

# Finding
14 Subinterpreter support is not currently feasible (global _ft2Library, static caches, single-phase init). python_requires >=3.12 provides the needed APIs if this becomes a goal. Decide explicitly whether to pursue it.
15 Stable ABI (abi3) is not claimed and not feasible given pybind11 + FreeType/Agg/qhull coupling. Confirm this is intentional (matplotlib ships version-specific wheels).

Documented False Positives (do not chase)

Source Location Why it's a false positive
error-path return_without_exception ×4 tri/_tri.cpp:483, 1030, 1037, 1038 Internal C++ helpers (get_edge_in_triangle, get_exit_edge) return -1 as a "not found" sentinel to other C++ code, not as a Python error indicator.
gil callback_without_gil ×5 ft2font_wrapper.cpp:490, 759, 797; _c_internal_utils.cpp:98, 116 pybind11-exposed methods, not C callbacks. Under mod_gil_not_used() there is no GIL to acquire.
resource resource_never_freed ×1 _qhull_wrapper.cpp:179 The FILE* error_file is owned by the QhullInfo RAII guard, whose destructor fcloses it (skipping stderr) on both normal return and exception unwind. Not leaked.
cppcheck uninitvar ×2 _qhull_wrapper.cpp:74, 211 facet/neighbor are initialized by qhull's FORALLfacets/FOREACHneighbor macros, invisible to cppcheck.
cppcheck unknownMacro ×2 ft2font.cpp:342, ft2font_wrapper.cpp:1581 pybind11 _a/_s string-literal operators; cppcheck lacks include paths (no compile_commands.json).

Strengths

  • pybind11 RAII discipline — refcounts and error propagation are framework-managed; the classic C-API bug classes simply don't appear.
  • Correct FreeType handle lifecycle — ~FT2Font()→close() releases every FT_Glyph and the FT_Face, is idempotent, and ~PyFT2Font deliberately calls close() before member destruction so the stream-close callback can still use py_file (ft2font.cpp:193-226, ft2font_wrapper.cpp:363-369).
  • QhullInfo RAII — frees qhull memory (qh_freeqhull+qh_memfreeshort, warns on residue) and closes the error file (_qhull_wrapper.cpp:119-131); _qhull is reentrant (stack-local qhT per call, :145).
  • Buffer-protocol exporters are safe — FT2Image, RendererAgg, BufferRegion each own their backing store, expose no resize/realloc to Python, and pybind11's bf_getbuffer increfs the exporter for the view's lifetime → no use-after-free (_backend_agg_wrapper.cpp:237,256).
  • Dynamic-library handles are balanced on all paths — _tkagg (FreeLibrary/dlclose on every return incl. errors, :213/223/240/318/342) and _c_internal_utils (dlclose :53/85, FreeLibrary :183).
  • _image resampling uses agg RAII allocators / py::array_t — no raw new/malloc.
  • Low complexity throughout — 209 functions, avg cyclomatic 3.9, max nesting 5, zero hotspots.
  • Modern baseline — python_requires >=3.12, no deprecated C-API, no limited-API contortions.

Code Removal Opportunities

Module and type creation are handled by pybind11 macros, so the usual C-API modernization removals (PyModule_AddType, PyImport_ImportModuleAttrString, etc.) do not apply — there is no hand-written PyType_FromSpec/PyModule_AddObject chain to collapse. The only raw-C-API surface is _macosx.m (out of scope) and minimal Windows/X11 helpers in _c_internal_utils.cpp. No material dead-code/removal opportunities identified.

Recommended Action Plan

Immediate (FIX)

  1. Finding 1 — Convert PyFT2Font_init to std::make_unique<PyFT2Font>(...) + return self.release(); on success, so a throw on a bad/corrupt font path frees the object and its file. (Highest priority — a per-call leak on a routine, user-reachable error path.)
  2. Finding 2 — Overflow-check width * height and check the calloc result in FT2Image.

Short-term (CONSIDER)

  1. Finding 3 / 13 — Guard all _ft2Library access with a dedicated std::mutex (see FREETHREADING_ANALYSIS.md) and keep mod_gil_not_used() only where the code is actually synchronized.
  2. Findings 4, 5, 11 — Make the raw new[]/FT_Get_Glyph/CoTaskMem sites RAII so OOM/late-throw paths don't leak.
  3. Finding 7 — Fix the mistyped .pyi signatures and fill in the four empty stubs (_qhull, _image, _backend_agg, _macosx) — cheap, improves the typed API that py.typed promises.
  4. Finding 10 — = delete the QhullInfo copy-ctor/assignment.

Longer-term (POLICY)

  1. Findings 6, 8, 9, 14 — If subinterpreter support becomes a goal, migrate to multi-phase init + module state and relocate _ft2Library/caches; document the GC-cycle constraint for user-supplied py::object members.
  2. Finding 15 — Record the (correct) decision that abi3 is out of scope.

For a phased free-threading migration plan:

/ft-review-toolkit:plan src

Methodology & Caveats

  • Interpreter: every scanner was run with python3t.exe (3.15.0b3 free-threaded), the only interpreter present, as requested. A native cp315t tree-sitter-cpp wheel had to be built locally (no free-threaded wheel exists; its setup.py forces Py_LIMITED_API, which breaks under Py_GIL_DISABLED) — otherwise the scanners silently skip all .cpp files. Details in FREETHREADING_ANALYSIS.md.
  • Two-stage analysis: all 12 central scanners + cppcheck ran first (reliable, consistent inputs); the deep agents parity-checker, type-slot-checker, and resource-lifecycle-checker were then dispatched and cross-validated the results. Findings 1 and 4 were each reported independently by two agents. The remaining pybind11-classic dimensions (refcounts, null, pyerr-clear, type-slots-proper, version-compat) were near-zero at the scanner stage and confirmed clean; a full 13-agent fan-out was consolidated into these three highest-value deep agents after an earlier fan-out hit a session limit.
  • Scanner posture on pybind11: the cext scanners are tuned for hand-written C-API code; on pybind11 they correctly report near-zero but structurally miss C++-level issues (non-static globals, member-init calloc, throwing constructors/factories, RAII ownership). Findings 1–6 came from reading the code.
  • External tools: cppcheck ran (findings triaged above); clang-tidy is installed but unusable without a compile_commands.json (no build tree) — it would only emit missing-include noise.
  • Out of scope: _macosx.m (Objective-C) is not covered by the C/C++ scanners and warrants a separate manual pass.
  • Docs consulted: pybind11 semantics via Context7 (/pybind/pybind11).
  • Sharing: this is an internal artifact. Before filing issues/PRs upstream, check with the maintainers on the shape they want rather than dropping the full list on them unsolicited.
C Extension Analysis Report
Extension: msgpack._cmsgpack
Scope: entire project
Agents Run: type-slot-checker, gil-discipline-checker, resource-lifecycle-checker,
module-state-checker, parity-checker, version-compat-scanner, c-complexity-analyzer,
stable-abi-checker, git-history-analyzer
(refcount-auditor, error-path-analyzer, null-safety-scanner, pyerr-clear-auditor
skipped — Cython-mode, validated zero FIX-class loss)
External Tools: cppcheck (7 .h files scanned), clang-tidy (skipped — no
compile_commands.json; requires `make cython && python setup.py build_ext -i` first)
---
Executive Summary
The extension is broadly well-implemented for a Cython project: per-instance buffer
allocation, comprehensive @cython.critical_section coverage on public methods, and a
solid exception-handling architecture. Three areas need immediate attention. First,
the recent free-threading commit (#641) missed Packer.__getbuffer__ and
__releasebuffer__ — the two buffer-protocol methods that directly protect the C
buffer from concurrent reallocation — creating a use-after-free window under
Py_GIL_DISABLED. Second, Unpacker.__init__ has a pre-existing re-initialization leak
pattern (both the C buffer and the in-progress parse stack) that has never been fixed
upstream. Third, the pure-Python fallback diverges from the Cython implementation in
three user-visible ways: read_bytes() return type, object_pairs_hook contract, and
skip() exception translation. The git history also reveals the fork is 7+ bug-fix
commits behind the upstream canonical repo.
---
Extension Profile
- Module: msgpack._cmsgpack (3 .pyx files, 940 lines; 7 hand-written .h files, 1,830
lines)
- Init style: Multi-phase (structural), but with process-global module state
(CYTHON_USE_MODULE_STATE=0, m_size=0)
- Python targets: ≥3.10
- Limited API: not claimed
- Code generation: Cython 3.2.1
- freethreading_compatible = True declared
---
External Tool Baseline
cppcheck ran against all 7 hand-written .h files (pack.h, pack_template.h,
sysdep.h, unpack.h, unpack_container_header.h, unpack_define.h,
unpack_template.h). One finding:
▎ FALSE POSITIVE — pack_template.h:256 (cppcheck: syntaxError)
▎ Root cause: cppcheck's preprocessor evaluator cannot resolve the
▎ three-branch portability idiom used in msgpack_pack_short (and
▎ msgpack_pack_int, msgpack_pack_long, etc.):
▎
▎ #if defined(SIZEOF_SHORT)
▎ #if SIZEOF_SHORT == 2 ... #endif
▎ #elif defined(SHRT_MAX)
▎ #if SHRT_MAX == 0x7fff ... #endif ← nested #if inside #elif
▎ #else
▎ if(sizeof(short) == 2) { ... } ← plain if, not #if
▎ #endif
▎
▎ The code is valid C. cppcheck bails at line 256 and does not analyze
▎ the remainder of pack_template.h (~350 lines including _PyFloat_* guards
▎ and the unsigned-int packing logic). See also finding 23 (complexity).
▎
▎ Workaround: build the extension with `make cython && python setup.py
▎ build_ext -i`, then pass `compile_commands.json` to cppcheck via
▎ `--project=`. clang-tidy with compile_commands.json would also
▎ resolve the macro definitions and produce richer analysis.
---
Key Metrics
┌──────────────┬────────┬─────┬──────────┬───────────────────────────────────────┐
│ Dimension │ Status │ FIX │ CONSIDER │ Top Finding │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Refcount │ G │ 0 │ 0 │ Cython-mode skip │
│ Safety │ │ │ │ │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Error │ G │ 0 │ 0 │ Cython-mode skip │
│ Handling │ │ │ │ │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ NULL Safety │ G │ 0 │ 0 │ Cython-mode skip │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ GIL │ Y │ 1 │ 0 │ __getbuffer__/__releasebuffer__ │
│ Discipline │ │ │ │ missing @cython.critical_section │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Resource │ G │ 0 │ 1 │ Partial parse-stack retained on │
│ Lifecycle │ │ │ │ _unpack error paths │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Module State │ Y │ 0 │ 3 │ 5 process-global PyObject* statics; │
│ │ │ │ │ PyDateTimeAPI cross-interpreter │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Type Slots │ Y │ 2 │ 0 │ Unpacker.__init__ re-init leaks │
│ │ │ │ │ self.buf and parse stack │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ PyErr_Clear │ G │ 0 │ 0 │ Cython-mode skip │
│ Safety │ │ │ │ │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ ABI │ G │ 0 │ 6 │ Not claimed; datetime C API is hard │
│ Compliance │ │ │ │ blocker │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Version │ G │ 0 │ 5 │ Dead sysdep.h guards; _PyFloat_* │
│ Compat │ │ │ │ correctly guarded to 3.10-only │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ C/Python │ Y │ 3 │ 5 │ read_bytes type, object_pairs_hook │
│ Parity │ │ │ │ contract, skip() exception │
├──────────────┼────────┼─────┼──────────┼───────────────────────────────────────┤
│ Complexity │ G │ 0 │ 3 │ unpack_execute CC ~60+, macro/goto │
│ │ │ │ │ contract undocumented │
└──────────────┴────────┴─────┴──────────┴───────────────────────────────────────┘
G = 0 FIX | Y = 1–3 FIX | R = 4+ FIX
---
Findings by Priority
Must Fix (FIX) — 6
#: 1
Finding: Packer.__getbuffer__ and __releasebuffer__ missing @cython.critical_section
—
concurrent pack() can call PyMem_Realloc on self.pk.buf while another thread's
memoryview(packer) is calling PyBuffer_FillInfo on the same pointer;
use-after-free.
Also self.exports read-modify-write is non-atomic.
File:Line: _packer.pyx:363-368
Agents: gil-discipline, git-history
────────────────────────────────────────
#: 2
Finding: Unpacker.__init__ overwrites self.buf via PyMem_Malloc without first freeing
the existing buffer — any re-call to __init__ (or subclass super().__init__(...))
leaks the prior allocation
File:Line: _unpacker.pyx:373
Agents: type-slot, git-history
────────────────────────────────────────
#: 3
Finding: Unpacker.__init__ calls unpack_init (not unpack_clear) before
re-initializing
the parse context — live PyObject* references in ctx->stack[1..top] are leaked if
called on a mid-stream Unpacker
File:Line: _unpacker.pyx:385
Agents: type-slot, git-history
────────────────────────────────────────
#: 4
Finding: fallback.Unpacker.read_bytes() returns bytearray (mutable, unhashable);
Cython returns bytes — dict-key use, isinstance checks, and hashability all diverge
File:Line: fallback.py:346-348
Agents: parity, git-history
────────────────────────────────────────
#: 5
Finding: fallback.Unpacker passes a generator to object_pairs_hook; Cython passes a
fully-materialised list — len(pairs), re-iteration, and subscript access all break;
additionally, strict_map_key validation is lazy in the fallback and can be bypassed
by a hook that doesn't fully consume the generator
File:Line: fallback.py:520-529
Agents: parity, git-history
────────────────────────────────────────
#: 6
Finding: fallback.Unpacker.skip() lets RecursionError propagate raw; every other
unpack method converts it to StackError — callers catching StackError/ValueError to
guard against adversarial nested data miss the exception on this path
File:Line: fallback.py:584-587
Agents: parity, git-history
---
Should Consider (CONSIDER) — 20
#: 7
Finding: Unpacker._unpack error paths (FormatError, StackError, PyErr_Occurred) do
not
call unpack_clear before raising — partially-built container objects in
ctx->stack[1..top] are retained until __dealloc__. Bounded at 1024 slots; contrast
with unpackb which already calls unpack_clear on failure
File:Line: _unpacker.pyx:484-491
────────────────────────────────────────
#: 8
Finding: utc, epoch, giga, ExtType, Timestamp compile to file-scope static PyObject*
process globals (CYTHON_USE_MODULE_STATE=0). Re-importing in a second
subinterpreter
clobbers the shared pointer, leaving the first interpreter's live Packer/Unpacker
instances with stale or cross-interpreter object references
File:Line: _cmsgpack.pyx:8-9, _unpacker.pyx:20
────────────────────────────────────────
#: 9
Finding: PyDateTimeAPI is a per-translation-unit static pointer (from datetime.h).
The
module-exec function calls import_datetime() per interpreter import, clobbering the
global with a capsule from the new interpreter. Cross-interpreter datetime
timestamp
decode (timestamp=3) will call function pointers from the wrong interpreter
File:Line: _cmsgpack.pyx:5, unpack.h:339-344
────────────────────────────────────────
#: 10
Finding: Py_mod_multiple_interpreters: NOT_SUPPORTED slot is guarded by #if
CYTHON_USE_MODULE_STATE (always 0 in shipped builds) — the multi-interpreter
posture
is absent from the compiled binary, leaving CPython to apply an implicit default
rather than an explicit declaration
File:Line: generated C
────────────────────────────────────────
#: 11
Finding: Fork is 7+ upstream bug-fix commits behind: missing pack_timestamp
return-code checking (378edc6), missing pack_ext_type autoreset (284782d), missing
unpack_callback_int64 null check (95c8be5), missing unpack_container_header return
check (156bb05)
File:Line: various
────────────────────────────────────────
#: 12
Finding: unpackb accepts max_buffer_size and read_size silently in the fallback (via
**kwargs to Unpacker.__init__) but raises TypeError in Cython — a max_buffer_size
limit passed to unpackb is silently ignored by the Cython path, breaking any caller
relying on it as a security cap
File:Line: _unpacker.pyx:141-149, fallback.py:72
────────────────────────────────────────
#: 13
Finding: pack_ext_type in Cython passes typecode directly to a C char — values
outside
0–127 silently overflow or encode the reserved timestamp type (-1); the fallback
validates and raises ValueError for out-of-range codes
File:Line: _packer.pyx:288-297
────────────────────────────────────────
#: 14
Finding: Multi-byte buffer input raises BufferError in Cython, ValueError in fallback
— callers catching ValueError to handle all invalid input miss the exception on the
Cython path
File:Line: _unpacker.pyx:127, fallback.py:68
────────────────────────────────────────
#: 15
Finding: read_size default is 1 MiB in Cython vs 16 KiB in fallback and docs — Cython
issues 64× larger reads than documented from file_like sources
File:Line: _unpacker.pyx:369
────────────────────────────────────────
#: 16
Finding: datetime subclass packing: Cython uses PyDateTime_CheckExact (excludes
subclasses like pendulum.DateTime), fallback uses isinstance (includes them) —
undocumented behavioral split
File:Line: _packer.pyx:243
────────────────────────────────────────
#: 17
Finding: Dead MSVC pre-2010 integer typedef block (_MSC_VER < 1600) — unreachable
since VS 2010; Python ≥3.10 requires VS 2019+
File:Line: sysdep.h:23-31
────────────────────────────────────────
#: 18
Finding: Dead GCC <4.1 atomic macro branch + unused
_msgpack_atomic_counter_t/_msgpack_sync_* infrastructure — defined but never called
anywhere in the codebase; likely a leftover from an earlier refcount-based design
File:Line: sysdep.h:39-49
────────────────────────────────────────
#: 19
Finding: pyproject.toml declares cython>=3.2.5 with no upper bound; CI pins
Cython==3.2.1 via requirements.txt — developer environments can pick up Cython 4
before any migration review
File:Line: pyproject.toml:52, requirements.txt:1
────────────────────────────────────────
#: 20
Finding: PyUnicode_InternInPlace on every map key from untrusted sources — in Python
3.12+ mortal interned strings persist in the intern table; adversarial input can
hold arbitrarily many unique strings in the table, a memory amplification vector
File:Line: unpack.h:191-192
────────────────────────────────────────
#: 21
Finding: Py_SIZE(o) used on bytes/bytearray/list — soft-deprecated direct struct
access; stable-ABI-safe alternatives (PyBytes_Size, PyList_Size, etc.) are drop-in
replacements
File:Line: _packer.pyx:186,199,226
────────────────────────────────────────
#: 22
Finding: Py_TYPE(o)->tp_name direct PyTypeObject field dereference in error-path
format strings — not available under Limited API; use %T (3.12+) or PyType_GetName
(3.11+)
File:Line: _packer.pyx:188,254,256, unpack.h:188
────────────────────────────────────────
#: 23
Finding: unpack_execute (~308 lines, CC ~60+, 10 goto targets) — the macro/goto
contract is undocumented; adding a new msgpack type without knowing that every case
arm must terminate with a push_*/again_* macro will silently fall through to
_failed. Also: MSVC and GCC code paths for SWITCH_RANGE diverge without CI coverage
of the MSVC path
File:Line: unpack_template.h:86-394
────────────────────────────────────────
#: 24
Finding: _pack_inner (~105 lines, CC ~25) — unicode encoding and datetime arithmetic
are inlined; extracting _pack_unicode and _pack_datetime helpers would reduce
cognitive load without performance cost
File:Line: _packer.pyx:152-256
────────────────────────────────────────
#: 25
Finding: unpack_callback_ext (~90 lines, CC ~20) — four timestamp-mode arms each
manage 2–5 manual refcount variables with early-return error paths; variable
aliasing in mode-1 (float path: a is reassigned after Py_DECREF calls); extracting
per-mode helpers would remove this as the highest-risk refcount surface in the
codebase
File:Line: unpack.h:292-381
────────────────────────────────────────
#: 26
Finding: PyList_SET_ITEM/PyTuple_SET_ITEM are non-Limited-API macros (direct ob_item
access); PyList_SetItem/PyTuple_SetItem are stable-ABI drop-ins (add return-value
check)
File:Line: unpack.h:148,150,200
---
Tensions
- Free-threading vs. subinterpreters: The extension declared freethreading_compatible
= True in the last commit but has process-global PyObject* state (findings 8–10)
that makes it subinterpreter-unsafe. These goals are not contradictory —
free-threading safety is per-instance lock discipline, subinterpreter safety is
per-interpreter state isolation — but shipping freethreading_compatible without the
matching NOT_SUPPORTED slot risks user expectation mismatch.
- strict_map_key security in object_pairs_hook: The Cython path enforces key
validation eagerly in C before the hook is called (safe). The fallback enforces it
lazily inside a generator (finding 5). Upstream committed a partial fix (cd81370) but
it still passes a generator, not a list.
---
Policy Decisions (POLICY) — 3
#: P1
Finding: _PyFloat_Pack4/8/_PyFloat_Unpack4/8 private API usage is correctly guarded
to
Python 3.10 only via #if PY_VERSION_HEX >= 0x030B00A7. When Python 3.10 reaches EOL
(Oct 2026), remove the #else branches entirely. Alternatively, replace with manual
IEEE 754 packing using the existing _msgpack_store32/64/_msgpack_load32/64
byte-swap
infrastructure — this eliminates the version fork and the CPython dependency from
the float path.
────────────────────────────────────────
#: P2
Finding: Stable ABI (abi3) is not feasible at Python 3.10 without rework — the hard
blockers are the datetime C API (PyDelta_FromDSU not in stable ABI, PyDateTimeAPI
capsule not in Limited API) and _PyFloat_* on Python 3.10. Raising the minimum to
3.11 fixes the float issue; the datetime path requires replacing capsule calls with
PyObject_CallMethod/PyObject_IsInstance. Benefit: single abi3 wheel replacing
per-minor wheels. Assess when preparing the 3.10-EOL release.
────────────────────────────────────────
#: P3
Finding: Add an unconditional {Py_mod_multiple_interpreters,
Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED} slot to the module def (not guarded by
CYTHON_USE_MODULE_STATE) to make the multi-interpreter posture explicit. This is a
near-zero-cost protective declaration that blocks CPython from loading the module
into a second subinterpreter until the process-global state (findings 8–9) is
resolved.
---
Strengths
- Per-instance locking discipline is excellent: Every public Python method on Packer
and Unpacker carries @cython.critical_section, correctly serialising per-object state
for free-threading. The two missing methods (finding 1) are a narrow, fixable gap in
an otherwise complete implementation.
- C buffer management is correct on the happy path: msgpack_pack_write uses the
classic local-pointer-then-commit realloc pattern (pack.h:47-51) that correctly
preserves pk->buf on allocation failure. All 4 tracked heap allocations across the
extension have matching frees.
- Exception propagation is thorough: unpackb and Unpacker._unpack both catch and
correctly convert RecursionError, OutOfData, FormatError, and re-raise existing
exceptions — a complete chain validated by the recent upstream sweep.
- No _Py* private API beyond the correctly-version-guarded _PyFloat_* fallback: The
extension uses only stable, documented CPython APIs in production code paths.
- unpack_callback_ext and the streaming FSM are functionally correct: Despite their
complexity, no correctness bugs were found in unpack_execute or the timestamp decode
path during this review.
- freethreading_compatible declaration is substantively justified: Module-level
globals are write-once at import and read-only thereafter; per-instance C buffers are
strictly isolated. The declaration is correct subject to fixing finding 1.
---
Code Removal Opportunities
Based on data/deprecated_apis.json patterns applicable at Python ≥3.10:
- Py_SIZE → PyBytes_Size/PyList_Size/PyTuple_Size: 3 call sites
(_packer.pyx:186,199,226) — mechanical one-line substitution each.
- Py_TYPE()->tp_name → %T format (Python ≥3.12): 4 sites (_packer.pyx:188,254,256,
unpack.h:188) — simplifies error formatting, removes struct access.
- Dead sysdep.h blocks (findings 17–18): Removing the MSVC pre-2010 typedefs and the
unused atomic counter infrastructure eliminates ~25 lines of dead code.
- _PyFloat_* fallback (when 3.10 EOL): Removing the #else branches in pack_template.h
and unpack_template.h eliminates ~8 lines and the last private-API usage.
---
Recommended Action Plan
Immediate (FIX items)
1. Fix Finding 1 — Add @cython.critical_section to Packer.__getbuffer__ and
__releasebuffer__ (_packer.pyx:363-368). Two lines.
2. Fix Findings 2 & 3 — At the top of Unpacker.__init__, add a guard: if self.buf !=
NULL: unpack_clear(&self.ctx); PyMem_Free(self.buf); self.buf = NULL before the
PyMem_Malloc call (_unpacker.pyx:373).
3. Fix Finding 4 — Change fallback.Unpacker.read_bytes to wrap result in bytes(...)
before returning (fallback.py:348).
4. Fix Finding 5 — Materialise the key-value list before calling object_pairs_hook in
the fallback (fallback.py:520-529), and enforce strict_map_key eagerly on each key.
5. Fix Finding 6 — Wrap fallback.Unpacker.skip() body in try/except RecursionError:
raise StackError (fallback.py:584-587).
6. Finding 11 — Sync with upstream canonical repo to pick up the 7 post-#641 bug-fix
commits; priority: 378edc6 (pack_timestamp return codes), 95c8be5
(unpack_callback_int64 null check), 284782d (pack_ext_type autoreset).
Short-term (CONSIDER items)
7. Finding 7 — Add unpack_clear(&self.ctx) before each raise in Unpacker._unpack
(mirroring unpackb).
8. Finding 13 — Add typecode range validation (0–127) to Packer.pack_ext_type in the
Cython path.
9. Finding 15 — Fix read_size default in _unpacker.pyx:369 to min(max_buffer_size,
16*1024).
10. Finding 23 — Add a block comment to unpack_execute documenting the macro/goto
contract ("every case arm must terminate with a push_*/again_* macro"). Add a CI
build that tests the MSVC code path for SWITCH_RANGE parity.
11. Findings 17–18 — Remove dead MSVC <1600 typedef block and unused atomic counter
infrastructure from sysdep.h.
12. Finding 19 — Add <4 upper bound to Cython dev dependency in pyproject.toml; align
with requirements.txt.
13. Finding P3 — Add unconditional Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED slot.
Longer-term (POLICY)
14. Findings 8–10 / P3 — Build with -DCYTHON_USE_MODULE_STATE=1 to activate Cython
3.2+'s per-interpreter module state, migrating the 5 cdef object globals into
__pyx_mstatetype. This is the path to genuine subinterpreter safety.
15. Finding 25 / P1 — When Python 3.10 reaches EOL: collapse _PyFloat_* #else
branches; extract unpack_callback_ext timestamp modes into per-mode helpers.
16. Finding P2 — When preparing the 3.10-EOL release: assess abi3 migration; the
datetime path rework is the main investment.
---
▎ Note before sharing externally: This report is an internal artifact. Before filing
▎ issues or opening PRs against the upstream msgpack/msgpack-python repo, review
▎ WORKING_WITH_MAINTAINERS.md (../../../WORKING_WITH_MAINTAINERS.md). The upstream
▎ maintainer is actively fixing bugs (7 commits since #641); reach out first to
▎ understand preferred contribution format. Security-relevant findings (finding 5's
▎ strict_map_key bypass via generator) should go through responsible disclosure if
▎ reported publicly.

C Extension Analysis Report

Extension: multidict

Scope: multidict/_multidict.c, multidict/_multilib/*.h

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner,

gil-discipline-checker, resource-lifecycle-checker, module-state-checker,

type-slot-checker, pyerr-clear-auditor, stable-abi-checker,

version-compat-scanner, parity-checker, c-complexity-analyzer,

git-history-analyzer

Date: 2026-06-26


Executive Summary

The multidict C extension is a mature, well-structured codebase with modern practices (multi-phase init, heap types, pythoncapi_compat, PyModule_AddType). However the analysis surfaced 19 FIX-class bugs across all major safety dimensions. The highest-priority issues are: six unguarded PyErr_Clear calls on hot paths (two of which swallow MemoryError and KeyboardInterrupt in the in operator of CIMultiDict); reference leaks in the set-operation views that fire every iteration; and missing Py_DECREF(tp) in both non-proxy tp_dealloc functions (a heap-type contract violation that accumulates type-object leaks). The declaration of Py_MOD_GIL_NOT_USED at _multidict.c:1571 makes the two free-threaded (3.13t) bugs live today: use-after-free on resize and hash-table corruption from the hash=-1 marker pattern. All 19 FIX items are localized and individually patchable.


Extension Profile

  • Module: multidict._multidict (1 .c file / 1,592 lines; 6 .h files / ~7,500 lines)
  • Init style: Multi-phase (PEP 489, Py_mod_exec slot, PyModuleDef_Init)
  • Python targets: >=3.10 (classifiers up to 3.14)
  • Free-threaded: Py_MOD_GIL_NOT_USED declared (_multidict.c:1571)
  • Limited API: No
  • Types defined: 9 (MultiDict, CIMultiDict, MultiDictProxy, CIMultiDictProxy, istr, ItemsView, KeysView, ValuesView, plus iterator variants)
  • Code generation: Hand-written
  • Dual implementation: C + pure-Python (_multidict_py.py), parametrized test suite covers both

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety R 9 2 PyUnicodeWriter leaked in md_repr on concurrent-mod early return (#1)
Error Handling Y 3 1 PyErr_Clear after PyMapping_Items swallows MemoryError (#14)
NULL Safety Y 4 4 self->md deref without NULL guard in view/iter len (#20-22)
GIL Discipline Y 2 3 Use-after-free when _md_resize frees oldkeys (#12)
Resource Lifecycle Y 2 0 Missing Py_DECREF(tp) in both non-proxy tp_dealloc functions (#7, #8)
Module State G 0 2 Non-atomic ++state->global_version in NEXT_VERSION macro (#28)
Type Slots R 5 0 Missing Py_DECREF(tp) + re-init hazards in all four tp_init functions (#7-11)
PyErr_Clear Safety R 5 2 Hot-path md_calc_identity clear swallows KeyboardInterrupt (#17)
ABI Compliance G 0 2 Not Limited API; istr layout blocks any future migration (#32)
Version Compat Y 1 5 _default (Py_None new ref) leaked on error path on 3.10/3.11 (#18)
C/Python Parity G 0 5 richcompare returns False instead of NotImplemented (#24)
Complexity G - 3 ~400-line set-op duplication (12 functions reducible to 2 helpers) (#43)

G = no FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings

FIX counts per row reflect primary attribution to that dimension's agent; individual findings may be confirmed by multiple agents. 19 unique FIX findings total.


Findings by Priority

Must Fix (FIX) — 19

# Finding File:Line Confirmed By
1 md_repr: early return NULL on version change bypasses fail: label, leaking the PyUnicodeWriter hashtable.h:~1793 refcount-auditor, error-path-analyzer, null-safety-scanner
2 md_next: *pidentity = Py_NewRef(...) then cleanup label sets *pidentity = NULL without Py_DECREF hashtable.h:~636 refcount-auditor
3 multidict_itemsview_or2: PyTuple_Pack result tpl never Py_DECREF'd inside loop — leaks every iteration views.h:~571 refcount-auditor, null-safety-scanner
4 multidict_itemsview_sub1: same tpl leak pattern — leaks every iteration views.h:~682 refcount-auditor, null-safety-scanner
5 multidict_itemsview_sub1: arg not in fail: cleanup, leaks on all error paths views.h:~697 refcount-auditor, error-path-analyzer
6 multidict_itemsview_contains: key leaked when PySequence_GetItem(obj, 1) fails views.h:~908 refcount-auditor, error-path-analyzer, null-safety-scanner
7 multidict_tp_dealloc: missing Py_DECREF(tp) after tp->tp_free(self) — heap-type contract violation; type-object leaks accumulate _multidict.c:~502 resource-lifecycle-checker, module-state-checker, type-slot-checker
8 multidict_proxy_tp_dealloc: same missing Py_DECREF(tp) _multidict.c:~1185 resource-lifecycle-checker, module-state-checker, type-slot-checker
9 multidict_proxy_tp_init and cimultidict_proxy_tp_init: Py_INCREF(md); self->md = md; overwrites previous self->md without Py_XDECREF — re-init leaks old reference _multidict.c:~1087, ~1341 refcount-auditor, type-slot-checker
10 multidict_tp_init and cimultidict_tp_init: call md_init without md_clear(self) first — re-init leaks old key-storage block _multidict.c:~554, ~1012 refcount-auditor, type-slot-checker
11 multidict_tp_init and cimultidict_tp_init: pass args (the full args tuple) instead of arg (the parsed argument) to _multidict_clone_fast — fast-clone optimization is permanently dead _multidict.c:~548, ~1006 type-slot-checker
12 _md_resize: frees oldkeys while concurrent readers may hold entry pointers into it — use-after-free on 3.13t (Py_MOD_GIL_NOT_USED is declared) hashtable.h:~226 gil-discipline-checker
13 md_find_next: marks and restores entry->hash = -1 as a multi-key sentinel — concurrent reads see a corrupt hash, racing restores lose entries on 3.13t hashtable.h:~729, ~761 gil-discipline-checker
14 _multidict_extend: unguarded PyErr_Clear after PyMapping_Items — swallows MemoryError and any non-TypeError exception _multidict.c:~104 pyerr-clear-auditor
15 multidict_tp_richcompare: unguarded PyErr_Clear after PyMapping_Keys — swallows all exceptions from key enumeration _multidict.c:~474 pyerr-clear-auditor
16 multidict_itemsview_contains: unguarded PyErr_Clear after PyObject_Length on hot path — swallows MemoryError on every x in view call views.h:~898 pyerr-clear-auditor
17 multidict_itemsview_contains: unguarded PyErr_Clear after md_calc_identity on hot path — for CIMultiDict, this swallows exceptions from .lower() including KeyboardInterrupt views.h:~916 pyerr-clear-auditor
18 multidict_setdefault: _default = Py_GetConstant(Py_CONSTANT_NONE) (new ref) leaked when md_set_default returns -1 — version-dependent: only leaks on Python 3.10/3.11 where None is not immortal _multidict.c:~651 refcount-auditor, version-compat-scanner
19 _md_parse_item: calls _err_cannot_fetch / _err_bad_length while an exception from PySequence_GetItem/PySequence_Size is already active — PyErr_SetString overwrites the original exception; triggers assertions in debug builds hashtable.h:~1504 pyerr-clear-auditor

Should Consider (CONSIDER) — 25

# Finding File:Line
20 multidict_view_len: dereferences self->md without NULL guard — tp_clear can set it to NULL before sq_length is called views.h:~59
21 multidict_iter_len: same self->md NULL-deref risk after tp_clear iter.h:~162
22 Iterator iternext functions (items_iter, values_iter, keys_iter): all dereference self->md without NULL guard after tp_clear iter.h:~79, ~105, ~122
23 multidict_view_richcompare: unguarded PyErr_Clear after PyObject_Length views.h:~70
24 multidict_tp_richcompare: returns Py_False for non-mapping types instead of Py_NotImplemented — breaks the comparison protocol and diverges from the Python implementation _multidict.c:~478
25 multidict_itemsview_or2 first loop: identity and value refs not explicitly cleared between iterations — a subsequent error path may double-decref a stale pointer views.h:~567
26 multidict_itemsview_sub1: Py_CLEAR(ret) called at two adjacent lines (701 and 704) — the second call is a no-op after the first, indicating cut-and-paste error views.h:~701, ~704
27 _err_cannot_fetch: intentionally clears a preceding MemoryError to raise its own message — hides OOM from callers; consider documenting or restructuring hashtable.h:~1504
28 NEXT_VERSION macro: ++state->global_version is a non-atomic read-modify-write on a field shared across threads — concurrent mutations on 3.13t can produce duplicate version numbers, breaking the mutation-during-iteration guard state.h
29 Local PyType_GetModuleByDef backport accesses tp->tp_mro directly via PyTuple_GET_ITEM — relies on CPython internal layout; could be replaced with pythoncapi_compat.h if that header ever adds this function state.h:~48
30 _unicode_hash: reads PyASCIIObject->hash without an atomic load — formal data-race undefined behavior under the C11 memory model on 3.13t; use _Py_atomic_load_ssize_relaxed or PyObject_Hash htkeys.h:~318
31 empty_htkeys static struct not marked const — a write through a non-const pointer to this sentinel would silently corrupt all empty-dict behavior htkeys.h
32 istr embeds PyUnicodeObject by value (PyUnicodeObject super;) — any future Limited API migration would require replacing this with a pointer-based layout istr.h
33 istr::state field (unsigned int state) is set in istr_new but never read anywhere — dead code; remove or document intent istr.h
34 11 dead Py_TPFLAGS_IMMUTABLETYPE version guards are always-taken given python_requires >= 3.10; remove the #if/#endif wrappers unconditionally _multidict.c:984, 1041, 1292, 1376; istr.h:110; iter.h:192, 213, 234; views.h:1066, 1622, 1691
35 #include <structmember.h> is soft-deprecated since 3.12 and already dead: pythoncapi_compat.h provides Py_T_PYSSIZET/Py_READONLY for 3.10/3.11; the multidict_members block is compiled out entirely on 3.12+ via #ifndef MANAGED_WEAKREFS _multidict.c:2
36 Three Py_TPFLAGS_IMMUTABLETYPE guards in views.h use threshold 0x030a0000 (pre-release) while the rest of the codebase uses 0x030a00f0 (final) — inconsistency; resolve when removing guards (#34) views.h:1066, 1622, 1691
37 Local PyType_GetModuleByDef backport (50 lines) is live for Python 3.10; can be deleted entirely when minimum version is raised to 3.11 state.h:~48
38 getone/getall: C raises KeyError(key) (bare key as arg), Python raises KeyError("Key not found: {key!r}") (string message) — different exception values for the same error _multidict.c; _multidict_py.py:~120
39 _ItemsView.__contains__: Python accepts only tuple/list as the needle; C accepts any 2-element sequence — behavior diverges for range(2), custom sequences, etc. views.h; _multidict_py.py
40 extend/update: Python uses if arg: (falsy check) to skip an empty-but-nonempty mapping; C uses if arg is not None: — a falsy non-empty mapping is silently dropped in Python only _multidict_py.py:~796
41 _ItemsView sequence-item errors: Python raises TypeError; C raises ValueError — minor divergence in exception type views.h; _multidict_py.py
42 parse2 in parser.h: cyclomatic complexity ~13, 9 parameters, nesting depth 5 — refactor candidate if this function needs to grow parser.h
43 Set-operation logic in views.h is duplicated across ~12 functions for or/and/sub/xor across items/keys/values views; reducible to 2 generic helpers with ~400 lines saved views.h
44 _md_resize: newkeys is leaked if htkeys_build_indices fails — practically dead path (allocation of indices table after key-storage allocation), but a latent leak hashtable.h:~202

Tensions

FIX-18 severity depends on Python version: refcount-auditor rated the _default leak a CONSIDER because None is immortal on Python 3.12+. version-compat-scanner upgraded it to FIX because the declared minimum is 3.10, where None has a real reference count and the leak is observable. Resolution: FIX applies; fix it now and delete decref_default bookkeeping entirely once the minimum is raised to 3.12.

CONSIDER-30 vs POLICY-47 on _unicode_hash: gil-discipline-checker flags the non-atomic PyASCIIObject->hash read as formal UB under 3.13t (CONSIDER). version-compat-scanner flags the private struct access as a portability policy decision (POLICY). Both concerns are valid and orthogonal; fixing the atomic access does not resolve the private-struct dependency.

CONSIDER-28 vs FIX-12/13 on FT readiness: The NEXT_VERSION non-atomic counter (#28) and the _unicode_hash non-atomic read (#30) are CONSIDER because they are data-race undefined behavior but have no known crash path today. FIX-12 and FIX-13 are FIX because they have concrete crash paths (dangling pointer after free, corrupt hash slot). None of the FT issues can be resolved by removing the Py_MOD_GIL_NOT_USED declaration alone; the extension would need per-object locking throughout.


Policy Decisions (POLICY) — 3

# Finding
45 Py_MOD_GIL_NOT_USED is declared at _multidict.c:1571 but the extension is not FT-safe: FIX items #12 and #13 are concrete crash paths under 3.13t, and CONSIDER items #28 and #30 add more UB. Either add Py_BEGIN_CRITICAL_SECTION / Py_END_CRITICAL_SECTION guards throughout (significant work) or remove the declaration and document the intent to add it later.
46 Limited API migration is not feasible without major refactoring: istr embeds PyUnicodeObject by value, _unicode_hash requires PyASCIIObject, PyUnicodeWriter is only stable since 3.14, and heap-type PyType_FromModuleAndSpec requires the full non-stable API. Minimum floor for a credible abi3 claim is Python 3.14.
47 _unicode_hash (htkeys.h:~318) accesses the private PyASCIIObject->hash field directly, bypassing PyObject_Hash. The field layout has been stable since Python 3.3, but this blocks any future abi3 adoption and would break if CPython reorganizes the Unicode object header (an active area of work in 3.14+). The tradeoff is performance (avoids a virtual dispatch on the cached-hash fast path) vs portability.

Strengths

The extension gets the following things right, which is non-trivial to maintain:

  • Multi-phase init (PEP 489): Clean Py_mod_exec slot with well-structured state allocation and goto fail cleanup.
  • Heap types via PyType_FromModuleAndSpec: Correct pattern for subinterpreter isolation; types are not registered as global statics.
  • PyModule_AddType already in use: All 9 types are registered with the modern API that handles Py_DECREF internally.
  • pythoncapi_compat.h vendored and correct: All three post-3.10 APIs used (PyUnicodeWriter, Py_GetConstant, PyMapping_GetOptionalItem) are properly backported; no unguarded new-API calls.
  • Version-dependent mutation guard: The md_version counter + tp_hash_version correctly detects dict mutation during iteration across all supported Python versions.
  • Comprehensive parametrized test suite: Tests run against both C and Python backends; any behavioral divergence surfaces immediately.
  • Clean module state struct: No static PyObject* globals for extension state; all module state goes through the multidictstate struct.
  • multidict_get handles _default correctly: The paired function to multidict_setdefault already calls Py_CLEAR(_default) unconditionally after the operation — providing the correct pattern for fixing FIX-18.

Code Removal Opportunities

All entries from deprecated_apis.json that apply at Python >=3.10:

  • PyModule_AddType (3.10+): Already in use. No action needed.
  • PyImport_ImportModuleAttrString (3.14+): No PyImport_ImportModule + PyObject_GetAttr pattern found. Not applicable.
  • PyDict_GetItemRef (3.13+, also via pythoncapi-compat): No PyDict_GetItemWithError calls found. Not applicable.
  • PyObject_HasAttrStringWithError (3.13+): No PyObject_HasAttrString calls found. Not applicable.

The most impactful code removal opportunity is internal to the codebase: removing the 11 dead Py_TPFLAGS_IMMUTABLETYPE version guards (#34) and the deprecated structmember.h include (#35) is a pure cleanup with no behavior change. Removing the PyType_GetModuleByDef backport in state.h (~50 lines, #37) becomes available when the minimum version is bumped to 3.11.


Recommended Action Plan

Immediate (FIX items — fix before next release)

  1. #17 + #16: Add PyErr_ExceptionMatches guards before both PyErr_Clear calls in multidict_itemsview_contains (views.h:~898, ~916). The md_calc_identity clear is the most severe — it swallows KeyboardInterrupt on the x in ci_multidict hot path.
  2. #14 + #15: Same fix for _multidict_extend (_multidict.c:~104) and multidict_tp_richcompare (_multidict.c:~474). For richcompare, also change the non-mapping fallthrough to Py_RETURN_NOTIMPLEMENTED (fixes #24 simultaneously).
  3. #7 + #8: Add Py_DECREF(tp) before return in multidict_tp_dealloc and multidict_proxy_tp_dealloc after the tp->tp_free(self) call. Pattern: type = Py_TYPE(self); ...; tp->tp_free(self); Py_DECREF(type);.
  4. #3 + #4: Add Py_DECREF(tpl) inside the loops in multidict_itemsview_or2 and multidict_itemsview_sub1 after the set operation that consumes tpl.
  5. #5: Add arg to the fail: label cleanup in multidict_itemsview_sub1.
  6. #1: In md_repr, change the early return NULL on version change to a goto fail so the writer is cleaned up.
  7. #9: Add Py_XDECREF(self->md) before the Py_INCREF(md); self->md = md; assignment in multidict_proxy_tp_init and cimultidict_proxy_tp_init.
  8. #10: Call md_clear(self) at the top of multidict_tp_init and cimultidict_tp_init when self is being re-initialized (check for existing keys before calling md_init).
  9. #6: Add Py_XDECREF(key) before return NULL on the PySequence_GetItem(obj, 1) failure branch in multidict_itemsview_contains.
  10. #2: In md_next cleanup label, change *pidentity = NULL to Py_DECREF(*pidentity); *pidentity = NULL.
  11. #11: Change _multidict_clone_fast(self, args, ...) to _multidict_clone_fast(self, arg, ...) in both multidict_tp_init and cimultidict_tp_init.
  12. #18: Follow the pattern of multidict_get: move if (decref_default) { Py_CLEAR(_default); } before the return NULL on the md_set_default < 0 path in multidict_setdefault.
  13. #19: In _md_parse_item, call PyErr_Clear() before calling _err_cannot_fetch or _err_bad_length when those helpers are invoked after a failed PySequence_GetItem/PySequence_Size.
  14. #12 + #13 (3.13t): These require the most design work. Options: (a) add Py_BEGIN_CRITICAL_SECTION(self) / Py_END_CRITICAL_SECTION around the resize and mark/restore operations, (b) replace the hash=-1 marker with a separate bit-flag field, or (c) remove Py_MOD_GIL_NOT_USED and file an issue for a future FT-safe redesign (#45).

Short-term (CONSIDER items — good for next maintenance window)

  1. #20-22: Add NULL guards for self->md in all len and iternext functions in views.h and iter.h.
  2. #28: Replace ++state->global_version in NEXT_VERSION with _Py_atomic_add_uint64(&state->global_version, 1) (or the compat equivalent) so the counter is correct under concurrent mutation on 3.13t.
  3. #24: Change Py_RETURN_FALSE to Py_RETURN_NOTIMPLEMENTED in multidict_tp_richcompare for non-mapping operands (also fixes parity gap with Python impl).
  4. #34 + #35 + #36: Remove the 11 dead Py_TPFLAGS_IMMUTABLETYPE guards, the structmember.h include, and normalize the threshold inconsistency in views.h — pure cleanup, no behavior change.
  5. #38 + #39 + #40: Align getone/getall error message format, _ItemsView.__contains__ sequence acceptance, and if arg is not None: check between C and Python backends.
  6. #23: Add PyErr_ExceptionMatches guard before PyErr_Clear in multidict_view_richcompare.
  7. #30: Replace the PyASCIIObject->hash read in _unicode_hash with _Py_atomic_load_ssize_relaxed(&ascii->hash) to eliminate the formal data race under 3.13t.
  8. #31: Mark empty_htkeys as static const to prevent accidental writes.
  9. #43: Refactor the ~12 set-operation functions in views.h into 2 generic helpers (one for iterating self, one for membership testing against the other operand). Estimated savings: ~400 lines.

Longer-term (POLICY decisions — require maintainer judgment)

  1. #45: Decide on the FT story: add Py_BEGIN_CRITICAL_SECTION / Py_END_CRITICAL_SECTION throughout, or remove Py_MOD_GIL_NOT_USED and file a tracking issue. The extension declares GIL-not-used today, so this is a commitment question.
  2. #46: If Limited API adoption is ever desired, the minimum floor is Python 3.14. The istr layout, _unicode_hash, and PyUnicodeWriter dependency all require 3.14+ stable-API replacements.
  3. #47: Consider replacing PyASCIIObject->hash with PyObject_Hash once the immortal-object fast path (Python 3.12+) makes the virtual dispatch cheaper; benchmark before changing.


Git History Context

The git-history-analyzer surfaced three findings that add important context to this report.

FIX #7 and #8 — Confirmed incomplete fix from commit b9bf25c

Commit b9bf25c ("Fix ref bug with iter, views and istr") applied the Py_DECREF(tp) + Py_VISIT(Py_TYPE(self)) heap-type dealloc pattern to istr, the three iterator types, and the three view types. However multidict_tp_dealloc and multidict_proxy_tp_dealloc in _multidict.c were not touched. Both multidict_tp_traverse and multidict_proxy_tp_traverse in that same file already have Py_VISIT(Py_TYPE(self)) — confirming the traverse half was applied but the dealloc half was missed. FIX #7 and #8 are therefore not new bugs; they are the tail of a partially-applied fix.

POLICY #45 — Py_MOD_GIL_NOT_USED added without per-object locking (commit 888553e)

Commit 888553e ("Implement support for the free-threaded build of CPython 3.13") added Py_MOD_GIL_NOT_USED but no Py_BEGIN_CRITICAL_SECTION call sites. The hashtable rewrite (9d3c53f) that followed also contains no critical sections. The extension currently declares itself GIL-free with no locking mechanism protecting mutation — making FIX #12 and #13 live bugs today on any 3.13t workload.

Churn concentration

hashtable.h was introduced in 9d3c53f and has already accumulated at least 7 fix commits — the highest defect density of any file in the extension. views.h underwent a 1,372-line rewrite in 5d022e9 with no follow-up fix commits yet. Both files should receive extra scrutiny during review.


Before sharing this report externally: the multidict maintainers did not request this review. A 47-finding report dropped as a single issue is a 47-hour homework assignment. Reach out first, ask what shape they want it in, and do not auto-file or auto-PR. Security-sensitive findings (KeyboardInterrupt suppression in the hot path, use-after-free on 3.13t) may warrant private disclosure before a public issue. See WORKING_WITH_MAINTAINERS.md in the plugin root for guidance on next steps.

C Extension Analysis Report — numba

Date: 2026-06-20
Scope: Entire project (C:\Users\שזדס\repos\numba)
Toolkit: cext-review-toolkit


Executive Summary

Numba's 31 C/C++ extension files contain significant correctness issues across every dimension analyzed. The two most critical crashers are in _typeof.cpp::string_writer_ensure: memcpy before malloc NULL check (produces a write-to-address-0 on OOM) and realloc-into-self (silently loses the old heap buffer on OOM). A third CRITICAL bug is in modulemixin.c::pycc_init_: Py_DECREF drops envobj to refcount zero after its raw address was stored in a global, creating a dangling pointer that causes UAF in any compiled pycc module.

GC cycle detection is broken for two of the most-used types: DispatcherType has an incomplete tp_traverse (misses argnames and the entire functions vector), no tp_clear, and no PyObject_GC_UnTrack in its dealloc; DUFuncType participates in GC not at all despite holding owned references to Dispatcher objects that cycle back through Python-side attributes. Five module init routines use PyModule_AddObject patterns that silently swallow failures or leak references.

The extension has begun free-threading migration (Py_MOD_GIL_NOT_USED declared in all 14 modules) but relies extensively on private CPython internal headers and struct fields in _dispatcher.cpp, requiring per-version maintenance on every CPython release.


Extension Profile

Property Value
Modules 14 C/C++ modules, 31 source files
Init style Single-phase (m_size = -1 in all modules via MOD_DEF macro)
Python targets >=3.10, <3.15
Limited API No
Free-threading Py_MOD_GIL_NOT_USED declared (EARLY/PAUSED migration)
Types defined 10 PyTypeObject instances (8 static, 2 external-linkage)
Stub files 13 .pyi files
External tools cppcheck — 4 format-string issues in debug/test functions

Agents Run

Agent Status Notes
error-path-analyzer ✅ 12 findings
refcount-auditor ⚠️ Prompt-length limit hit; coverage partially overlaps null-safety + git-history
null-safety-scanner ✅ 16 findings
gil-discipline-checker ✅ 10 findings
resource-lifecycle-checker ✅ 8 findings
module-state-checker ✅ 20 findings
type-slot-checker ✅ 7 findings
pyerr-clear-auditor ✅ 12 findings
stable-abi-checker ✅ 21 findings
version-compat-scanner ✅ 20 findings
parity-checker ⚠️ Session limit — .pyi parity not assessed
c-complexity-analyzer ✅ 18 findings
git-history-analyzer ✅ 8 sibling findings + asymmetric-fix analysis

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety 🔴 12 4 compute_fingerprint leaks 3 new-refs via TRY macro; ndarray_key DECREF(NULL)
Error Handling 🔴 6 5 Dispatcher_call early goto → unbalanced DECREF on borrowed args
NULL Safety 🔴 8 4 string_writer_ensure: memcpy before malloc NULL check (CRITICAL)
GIL Discipline 🟡 1 7 rnd_implicit_init: PyGILState_STATE leaked on all error paths
Resource Lifecycle 🟡 3 5 string_writer_ensure realloc-into-self loses buffer on OOM
Module State 🔴 5 8 PyModule_AddObject w/o Py_INCREF + no error check in 5 module inits
Type Slots 🔴 5 2 DispatcherType traverse misses argnames/functions; no tp_clear; no GC_UnTrack
PyErr_Clear Safety 🟡 3 6 Dispatcher_call unguarded clear swallows MemoryError and fatal exceptions
ABI Compliance 🔴 10 7 internal/pycore_frame.h + pycore_instruments.h + multiple private _Py* APIs
Version Compat 🟡 5 7 PyEval_GetLocals() deprecated in 3.13 (changed from borrowed to new ref)
C/Python Parity — — — Agent unavailable (session limit)
Complexity 🟡 — 8 compute_fingerprint: 3 ref-leak patterns, cyclomatic ~22; call_cfunc triplicated

🔴 = 4+ FIX findings  |  🟡 = 1–3 FIX findings  |  🟢 = No FIX findings


Findings by Priority

All findings are numbered globally. FIX (1–51) → CONSIDER (52–92) → POLICY (93–100). Reference findings by number in issue trackers and code review.


Must Fix (FIX) — 51 findings

Memory Safety / Reference Counting (1–20)

# Severity Finding File:Line Agents
1 CRITICAL string_writer_ensure: memcpy(w->buf, …) called before NULL check after malloc — writes to address 0 on OOM _typeof.cpp:114 EP, NS
2 HIGH string_writer_ensure: w->buf = realloc(w->buf, …) overwrites pointer before checking result — old heap buffer permanently unreachable on OOM; string_writer_clear cannot free it _typeof.cpp:118 RL
3 HIGH typecode_using_fingerprint: after string_writer_move(key, &w) transfers ownership, _Numba_HASHTABLE_SET failure returns -1 without freeing key struct or its heap buffer _typeof.cpp:655 RL
4 HIGH compute_fingerprint (namedtuple path): _fields new ref from PyObject_GetAttrString is never DECREF'd on 10+ early return -1 paths before line 352 _typeof.cpp:300 EP, NS
5 HIGH compute_fingerprint (OmittedArg path): TRY macro expands to if (func()) return -1, bypassing Py_DECREF(default_val) — new ref from PyObject_GetAttr leaked _typeof.cpp:364 GH
6 HIGH compute_fingerprint (numpy scalar path): descr new ref from PyArray_DescrFromScalar leaked via same TRY-macro early-return pattern _typeof.cpp:375 GH
7 HIGH ndarray_key: three PyLong_FromLong results used unchecked — if any returns NULL on OOM, PyTuple_Pack receives NULL args and subsequent Py_DECREF(tmpX) is UB (crash) _typeof.cpp:743 NS, GH
8 HIGH cache_ndarray_typecode: result of ndarray_key() and PyLong_FromLong() passed directly to PyDict_SetItem without NULL check _typeof.cpp:764 NS
9 MEDIUM get_cached_ndarray_typecode: (a) NULL key from ndarray_key passed to PyDict_GetItem; (b) key leaked on cache miss (no DECREF before return -1) _typeof.cpp:752 EP, NS
10 MEDIUM cache_typecode: PyLong_FromLong result not checked; Py_DECREF(value) with potentially-NULL value is UB _typeof.cpp:733 EP
11 CRITICAL pycc_init_ loop 2: stores envobj raw pointer into *envgvs[i], then Py_DECREF(envobj) drops refcount 1→0 and frees the object — global becomes a dangling pointer, UAF in any compiled pycc module modulemixin.c:210 EP
12 HIGH NRT_adapt_ndarray_to_python_acqref: PyObject_New and PyTuple_New unchecked; PyTuple_SET_ITEM(args, …) dereferences potentially-NULL args; NRT_MemInfo_acquire not reversed on failure; miobj leaked if args is NULL _nrt_python.c:396 EP, NS
13 HIGH Dispatcher_call: goto CLEANUP at line 1337 runs before Py_INCREF(args) is established, then CLEANUP unconditionally Py_DECREF(args) — unbalanced DECREF on borrowed reference _dispatcher.cpp:1337 EP
14 HIGH Dispatcher_cuda_call: same early-goto/unbalanced-DECREF pattern as Finding 13 _dispatcher.cpp:~1453 EP
15 HIGH call_cfunc (3.10/3.11 profiling path): PyDict_New() result stored in globals used unchecked at line 650 PyDict_SetItemString(globals, …) — crash if OOM _dispatcher.cpp:633 EP, NS
16 MEDIUM Dispatcher_Insert: PyLong_AsLong errors silently swallowed (sets exception, returns -1, but code continues); Py_RETURN_NONE executes with active exception _dispatcher.cpp:502 EP
17 MEDIUM import_meminfo_release: PyLong_AsVoidPtr error not checked — returns (void*)-1 (non-NULL) on failure; garbage function pointer installed as global MemInfo_release → UAF crash in box_dealloc _box.c:147 EP
18 MEDIUM traceback_add_loc: PyString_AsString (= PyUnicode_AsUTF8) result not checked for NULL before passing to traceback_add and printf-style calls _helperlib.c:888 NS
19 HIGH rnd_implicit_init: goto error at lines 200, 207, 210, 213 bypasses the sole PyGILState_Release(gilstate) at line 216 — GIL state leaked on every error path _random.c:219 GIL
20 MEDIUM numba_new_thread (Windows): HeapAlloc(obj) at line 234 not freed when _beginthreadex fails; also wrong failure sentinel (-1 checked but _beginthreadex returns 0 on failure) workqueue.c:241 RL

Type Slot Bugs (21–25)

# Severity Finding File:Line Agent
21 HIGH Dispatcher_traverse (line 411–415) visits only self->defargs; misses self->argnames (owned PyObject*) and all callables in self->functions (std::vector<PyObject*>) — cycles through these fields invisible to GC _dispatcher.cpp:411 TS
22 HIGH Dispatcher_dealloc does not call PyObject_GC_UnTrack(self) before releasing members — GC may call Dispatcher_traverse on a half-freed Dispatcher if a Py_XDECREF inside dealloc triggers collection _dispatcher.cpp:418 TS
23 HIGH DispatcherType.tp_clear = NULL despite Py_TPFLAGS_HAVE_GC — GC can detect cycles through Dispatcher but cannot break them; any cycle involving a Dispatcher is permanently uncollectable _dispatcher.cpp:1594 TS
24 HIGH PyDUFuncType has no Py_TPFLAGS_HAVE_GC, no tp_traverse, no tp_clear, despite owning dispatcher (a Dispatcher object), ufunc (numpy PyUFuncObject), and keepalive — dispatcher cycles back to _DUFunc via Python attributes; these cycles are never collected _internal.c:722 TS
25 HIGH closure_dealloc does not call PyObject_GC_UnTrack before PyObject_ClearWeakRefs or member DECREFs — ClosureType has Py_TPFLAGS_HAVE_GC; a GC collection triggered during dealloc may call closure_traverse on a partially-freed object _dynfunc.c:207 TS

Module Init — PyModule_AddObject Patterns (26–30)

All five sites should use PyModule_AddObjectRef (available via the already-vendored numba/pythoncapi_compat.h) which does not steal the reference and is safe to check.

# Severity Finding File:Line Agent
26 HIGH PyInit__helperlib: PyModule_AddObject(m, "c_helpers", build_c_helpers_dict()) — new ref passed without Py_INCREF, return value not checked → ref leaked and error silently swallowed _helpermod.c:276 MS
27 HIGH PyInit__dynfunc: 5+ PyModule_AddObject calls — impl_info and c_helpers dict are new refs passed without Py_INCREF; no return-value check on any call → leaks + silent module init failure _dynfuncmod.c:81 MS
28 HIGH PyInit__nrt_python: PyModule_AddObject with c_helpers new ref, no Py_INCREF, no error check _nrt_pythonmod.c:212 MS
29 HIGH PyInit__box: PyLong_FromSsize_t results (new refs) passed to PyModule_AddObject without Py_INCREF or error check at lines 207 and 209 _box.c:204 MS
30 MEDIUM PyInit_mviewbuf: PyModule_AddObject return value not checked — on failure Py_INCREF is not undone, permanently leaking a refcount on MemAllocType mviewbuf.c:385 MS

Exception Handling (31–35)

# Severity Finding File:Line Agent
31 HIGH Dispatcher_call: PyErr_Clear() called unconditionally when typeof_typecode fails and can_fallback=True — clears MemoryError, KeyboardInterrupt, SystemExit as silently as TypeError _dispatcher.cpp:1361 EC
32 HIGH Dispatcher_cuda_call: same unguarded PyErr_Clear() on typeof failure _dispatcher.cpp:1475 EC
33 MEDIUM import_cython_function: PyErr_Clear() after PyMapping_GetItemString failure — intends to swallow KeyError but also swallows MemoryError from __hash__/__eq__; replaces it with ValueError _helperlib.c:577 EC
34 MEDIUM numba_reset_pyobject_private_data: unconditional PyErr_Clear() after PyDict_DelItem — intent is KeyError-ignore but MemoryError silently swallowed on a cleanup path _helperlib.c:765 EC
35 MEDIUM typecode_devicendarray: 5 unguarded PyErr_Clear() calls (lines 873, 889, 899, 909, 918) on optional attribute lookups — MemoryError from __getattr__ treated as cache miss and silently discarded _typeof.cpp:873 EC

Deprecated and Private APIs (36–51)

Items marked [no public alternative] have no stable public API replacement in the guarded Python version range; they require per-version maintenance until those versions are dropped.

# Sev Finding File:Line Agent
36 HIGH _PySet_NextEntry: private API, never public — accessed via internal/pycore_setobject.h with Py_BUILD_CORE bypass; replace with PyObject_GetIter/PyIter_Next or PySet_Pop+re-add _typeof.cpp:416 ABI
37 HIGH internal/pycore_frame.h included directly with #define Py_BUILD_CORE; also pulls in pycore_interp.h, pycore_call.h, pycore_atomic.h — internal ABI not guaranteed stable across minor versions _dispatcher.cpp:35 ABI
38 HIGH _PyErr_Fetch/_PyErr_Restore (tstate-local variant): private API, never public — use public PyErr_Fetch/PyErr_Restore or PyErr_GetRaisedException/PyErr_SetRaisedException on 3.12+ _dispatcher.cpp:99 ABI
39 HIGH co->co_linetable, _PyCode_CheckLineNumber, _Py_CODEUNIT: private struct fields + internal function — replace with PyCode_Addr2Line() or PyFrame_GetLineNumber() _dispatcher.cpp:184 ABI
40 HIGH internal/pycore_instruments.h + tstate->interp->monitors.tools[] / tstate->interp->monitoring_callables[][]: direct internal struct access for 3.12 monitoring [no public alternative in 3.12]; already migrated to public PyMonitoring_* on 3.13+ _dispatcher.cpp:706 ABI
41 HIGH PyEval_GetLocals() deprecated in 3.13 and returns new reference (not borrowed as before) — unguarded across all supported versions; use PyEval_GetFrameLocals() on 3.13+ _dispatcher.cpp:1335 VC
42 HIGH PyEval_GetLocals() same issue in Dispatcher_cuda_call _dispatcher.cpp:1449 VC
43 HIGH _PyErr_ChainExceptions1: private API — use PyException_SetContext() or restructure error handling to avoid C-layer chaining _helperlib.c:862 ABI
44 HIGH _PyErr_ChainExceptions (3-argument form): moved to internal API in 3.12; same fix _helperlib.c:865 ABI
45 HIGH _PyErr_StackItem/tstate->exc_info private struct access — replace with public PyErr_GetExcInfo()/PyErr_SetExcInfo() _helperlib.c:912 ABI
46 MEDIUM PyErr_Fetch/PyErr_Restore deprecated in 3.12; called unconditionally across 3.10–3.14 — use PyErr_GetRaisedException/PyErr_SetRaisedException on 3.12+ _helperlib.c:826 VC
47 MEDIUM PyErr_Restore deprecated in 3.12 in reraise_exc_is_none _helperlib.c:922 VC
48 MEDIUM PyUnicode_READY() deprecated since 3.10, removed in 3.12 — called unguarded; add #if PY_VERSION_HEX < 0x030C0000 guard _helperlib.c:1145 VC
49 MEDIUM _Py_c_pow: private CPython complex-math function — replace with cpow() from <complex.h> _helperlib.c:137 ABI
50 MEDIUM _Py_HashSecret private struct fields (.siphash.k0, .siphash.k1, .djbx33a.suffix) accessed directly — access via sys.hash_info or ctypes.pythonapi instead modulemixin.c:130 ABI
51 MEDIUM tstate->tracing / tstate->cframe->use_tracing direct struct access (3.10/3.11 only) — replace with PyThreadState_EnterTracing/PyThreadState_LeaveTracing from the already-vendored pythoncapi_compat.h _dispatcher.cpp:215 ABI

Should Consider (CONSIDER) — 41 findings

Reference Counting / NULL Safety (52–57)

# Finding File:Line
52 ndarray_key: use Py_XDECREF instead of Py_DECREF for tmpndim/tmplayout/tmpreadonly as defense-in-depth even after fixing Finding 7 _typeof.cpp:744
53 get_cached_typecode/get_cached_ndarray_typecode: replace PyDict_GetItem with PyDict_GetItemWithError to distinguish cache-miss from genuine dict error _typeof.cpp:724,753
54 import_meminfo_release: after fixing Finding 17, also add NULL check on return from PyDict_GetItemString before goto cleanup so module init surfaces a useful error message _box.c:145
55 DispatcherType.Dispatcher_traverse: guard Py_VISIT(self->argnames) with a NULL check since argnames may be NULL before __init__ completes _dispatcher.cpp:413
56 ClosureType.tp_clear is NULL despite Py_TPFLAGS_HAVE_GC — keepalive is mutable and can hold arbitrary Python objects participating in cycles; add closure_clear _dynfunc.c:240
57 PyUFuncCleaner_Type holds owned PyObject* object without GC participation — short-lived but could cycle back through object's attributes _internal.c:43

GIL Discipline / Free-Threading (58–65)

# Finding File:Line
58 pyobject_dtor: PyGILState_Ensure() called without Py_IsInitialized() guard — if destructor fires after interpreter shutdown, PyGILState_Ensure may deadlock _nrt_python.c:26
59 numba_fatal_error: discards PyGILState_Ensure() return value — GIL bookkeeping entry leaked (mitigated by Py_FatalError never returning; also lacks Py_IsInitialized() guard) _helperlib.c:806
60 _get_private_data_dict: lazy-init with non-atomic if (private_data_dict == NULL) — data race under free-threading; two threads could race to call PyDict_New() and one dict leaks _helperlib.c:700
61 workqueue.c globals _nesting_level, queue_count, queue_pivot, NUM_THREADS: non-atomic read-modify-write — _nesting_level check+increment at lines 377/394 is a race under free-threading workqueue.c:59
62 tbbpool.cpp: tg (task_group*) guarded by non-atomic if(tg) return in launch_threads — double-init race under free-threading np/ufunc/tbbpool.cpp:32
63 thread_worker (workqueue): worker threads have no PyThreadState attached; if any JIT-compiled task function reaches Python objects (e.g., via destructors), it does so without GIL workqueue.c:553
64 omppool.cpp: PyString_FromString(_OMP_VENDOR) result not checked for NULL before PyObject_SetAttrString + Py_DECREF in module init np/ufunc/omppool.cpp:358
65 tbbpool.cpp: init_thread_id write in set_main_thread() races with concurrent launch_threads calls under free-threading np/ufunc/tbbpool.cpp:208

Resource Lifecycle (66–70)

# Finding File:Line
66 fingerprint_hashtable: created at module init, never freed — no m_free registered; each entry holds a malloc'd string_writer_t* with potentially heap-allocated buffer _typeof.cpp:1139
67 launch_threads (workqueue): queues malloc acknowledged with comment /* this memory will leak */; POSIX pthread_cond_t/pthread_mutex_t inside each Queue never destroyed workqueue.c:587
68 private_data_dict static global never Py_DECREF'd at module teardown; no m_free registered _helperlib.c:700
69 NRT_MemSys_shutdown sets TheMSys.shutting = 1 only; no stats check for outstanding live MemInfo objects at shutdown (mismatched mi_alloc/mi_free in stats silently ignored) nrt.cpp:85
70 numba_new_thread (Windows): even after fixing Finding 20, confirm that all Windows code paths check _beginthreadex returns 0 (not -1) workqueue.c:234

Module State (71–78)

# Finding File:Line
71 omittedarg_type static global populated in typeof_init() without double-init guard — if called twice, old object leaked _typeof.cpp:47
72 typecache, ndarray_typecache, structured_dtypes: process-global dicts holding references to Python type objects; prevent GC finalization of those types _typeof.cpp:49
73 str_typeof_pyval, str_value, str_numba_type: interned string globals set in typeof_init() without NULL-before-assignment guard against double-init _typeof.cpp:53
74 static void (*fnclex)(void) mutable function pointer set from Python without synchronization — write race under free-threading _helperlib.c:53
75 rnd_globally_initialized non-atomic check-and-set — two concurrent threads can both call pthread_atfork registration; THREAD_LOCAL PRNG state is per-OS-thread, not per-interpreter _random.c:112
76 DispatcherType static type: tp_dict shared across subinterpreters; mutating it from one subinterpreter affects all others _dispatcher.cpp:1571
77 MemInfo_release function pointer global in _box.c — set once at module import; if NRT module unloads first, this pointer becomes dangling _box.c:16
78 PyModule_AddObjectRef is already available via numba/pythoncapi_compat.h for all supported Python versions; audit all 14 module init functions and replace every PyModule_AddObject call Multiple

PyErr_Clear / Exception Handling (79–83)

# Finding File:Line
79 compute_fingerprint writable-buffer fallback: PyErr_Clear() after PyObject_GetBuffer may swallow MemoryError; guard with PyErr_ExceptionMatches(PyExc_BufferError) _typeof.cpp:435
80 _numba_rnd_seed: PyErr_Clear() after PyArg_ParseTuple failure without exception-type guard — MemoryError could be swallowed _random.c:357
81 typeof_typecode (32-bit path): PyErr_Clear() after PyLong_AsLongLong overflow without checking PyErr_ExceptionMatches(PyExc_OverflowError) _typeof.cpp:975
82 MOD_INIT(_dispatcher): PyErr_Print() then PyErr_SetString() — if PyErr_Print() itself raises (OOM in formatting), the subsequent SetString clobbers that error _dispatcher.cpp:1673
83 mviewbuf.c:get_writable_buffer: PyErr_Clear() after writable buffer fail — intends BufferError only; MemoryError from __getbuffer__ silently discarded mviewbuf.c:16

Version Compatibility (84–88)

# Finding File:Line
84 traceback_add error path uses minor-version enumeration `(PY_MINOR_VERSION == 12)
85 Multiple version guards in _dispatcher.cpp enumerate minor versions; change to >= 0x030C0000 range comparisons to avoid per-release edits _dispatcher.cpp:30
86 pythoncapi_compat.h vendored at commit 0041177c4f predates Python 3.14 final — sync from upstream periodically pythoncapi_compat.h:1
87 co->co_flags direct struct write (3.10/3.11 only) — no public API alternative for this version range [no public alternative] — document as known CPython-internals dependency _dispatcher.cpp:655
88 tstate->frame direct field write (3.10 only) [no public alternative] — document as 3.10-specific limitation to be removed when 3.10 support is dropped _dispatcher.cpp:669

Complexity / Refactoring (89–92)

# Finding File:Line
89 compute_fingerprint: cyclomatic ~22; namedtuple sub-branch 5 levels deep; the TRY macro creates 3 confirmed ref-leak patterns in one function — extract namedtuple branch to fingerprint_namedtuple() and ndarray branch to fingerprint_ndarray() _typeof.cpp:262
90 call_cfunc triplicated across 3 version guards (3.10/3.11 ~113 lines, 3.12 ~67 lines, 3.13+ ~77 lines); any bug fix must be applied to all 3; extract shared call_cfunc_impl _dispatcher.cpp:585,968,1071
91 parallel_for: alloca(sizeof(size_t) * arg_len) with runtime-controlled arg_len — stack overflow is silent; replace with bounded malloc workqueue.c:366
92 dufunc__add_loop: single packed malloc holds functions[]/data[]/types[] contiguously via pointer arithmetic — extract layout into a documented struct with named offset accessors _internal.c:498

Tensions

  • Private API use in _dispatcher.cpp (Findings 37–45) vs. no public alternatives: The traceback injection (traceback_add) and profiling/monitoring hooks (call_cfunc, invoke_monitoring) use private CPython internals because no public API exists for these operations in the covered Python versions. Findings 40, 87, and 88 are correctly marked as "no public alternative" — the pragmatic response is to document the dependency per-version, confirm the code is guarded with exact version checks, and prioritize cleanup as those Python versions approach EOL (3.10 EOL: October 2026; 3.11 EOL: October 2027).

  • m_size = -1 and free-threading: All 14 modules declare Py_MOD_GIL_NOT_USED (FT migration begun) while also using m_size = -1 (single-phase init, global state). These are orthogonal axes — GIL-not-used is about GIL locking, not subinterpreter isolation — but the global state (type caches, NRT allocator, private data dict) makes true FT safety harder and subinterpreter support non-trivially impossible. The FT findings (60–65) are the live surface.

  • compute_fingerprint TRY macro (Findings 4–6) vs. performance: The TRY macro was introduced to reduce boilerplate; it creates the ref-leak pattern because it returns -1 without access to locally-acquired references. A safe alternative: use a local error_label: cleanup block instead of TRY-triggered returns, accepting the boilerplate for the safety.


Policy Decisions (POLICY) — 8 findings

# Finding
93 MOD_DEF macro hard-codes m_size=-1, m_traverse=NULL, m_clear=NULL, m_free=NULL for all 14 modules — project-wide decision: document that numba explicitly does not support subinterpreters (Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED)
94 All 10 PyTypeObject instances are static (not heap types) — per-version tp_watched/tp_versions_used tail fields must be added for each new CPython release; consider heap-type migration if subinterpreter support ever becomes a goal
95 structmember.h included via _pymodule.h — deprecated since 3.12; T_BOOL, T_OBJECT_EX, PyMemberDef available directly from Python.h; remove the include
96 NB_SUPPORTED_PYTHON_MINOR macro in _pymodule.h lists supported minor versions as an enumeration; change to range check (PY_MINOR_VERSION >= 10 && PY_MINOR_VERSION <= 14) to reduce per-release edits
97 traceback_add re-implements CPython's internal _PyTraceback_Add — ongoing per-version maintenance burden; evaluate whether _PyTraceback_Add is accessible as a public symbol (declared in cpython/traceback.h) to reduce duplication
98 _Py_HashSecret use in modulemixin.c is stable in practice (siphash key layout unchanged since Python 3.4) but is a private struct — document the dependency or access via ctypes.pythonapi if auditability is required
99 pythoncapi_compat.h should be synced from upstream on each Python minor release; track upstream changelog at https://github.com/python/pythoncapi-compat
100 C/Python parity (parity-checker agent) was unavailable due to session limit — re-run separately as: 13 .pyi stubs exist and should be cross-checked against their C implementations, particularly _dispatcher.pyi vs _dispatcher.cpp and dufunc.pyi vs _internal.c

Strengths

The extension does several things well that should be preserved:

  • NRT refcounts are atomic: std::atomic_size_t in nrt.cpp for NRT_MemInfo refcounts is correct and already FT-safe.
  • GIL acquisition in dtors: pyobject_dtor in _nrt_python.c correctly uses PyGILState_Ensure/Release before Py_DECREF — a frequent source of bugs in other extensions.
  • dufunc_call exception guard: if (PyErr_ExceptionMatches(PyExc_TypeError)) { PyErr_Clear(); … } before retrying with compiled args is a gold-standard guarded clear (as opposed to Findings 31–35).
  • typecode_using_fingerprint exception guard: PyErr_ExceptionMatches(PyExc_NotImplementedError) correctly guards the clear before fallback.
  • EnvironmentType GC slots: env_traverse, env_clear, and PyObject_GC_UnTrack in env_dealloc are all present and correct — this is the reference pattern that ClosureType and DispatcherType should follow.
  • Version-gated monitoring: The 3.13+ call_cfunc path correctly uses the public PyMonitoring_* API instead of internal struct access — a good migration example for the 3.12 path.
  • pythoncapi_compat.h: Vendoring this file correctly abstracts version-specific API differences; it already provides PyThreadState_EnterTracing/LeaveTracing and PyModule_AddObjectRef that fix several findings above.
  • global_compiler_lock in compiler_lock.py correctly serializes LLVM compilation — the most expensive shared operation.

Code Removal Opportunities

Findings that enable net line reduction once fixed:

  • PyModule_AddObjectRef migration (Findings 26–30): Replace Py_INCREF(x); if (PyModule_AddObject(m, name, x) < 0) { Py_DECREF(x); return …; } 4-line pattern with 1-line PyModule_AddObjectRef call. Applies to ~15 sites across 6 module init functions. Est. 30–45 lines removed.
  • Dead 3.10/3.11 compat code (Finding 90): When Python 3.10 and 3.11 support is dropped, the entire call_cfunc 3.10/3.11 block (lines 581–698, ~118 lines) + helper functions call_trace, call_trace_protected, initialize_trace_info, _nb_PyCode_InitAddressRange, _nb_PyLineTable_InitAddressRange can be deleted. Est. 200+ lines removed.
  • Dead deprecation guards (version-compat): Guards (PY_MINOR_VERSION == 10) and (PY_MINOR_VERSION == 11) in _helperlib.c become dead code once 3.10/3.11 are dropped. Est. 50–80 lines removed.
  • PyErr_Fetch/Restore migration to single-object API (Findings 46–47): On 3.12+ the three-object fetch/restore pair collapses to a single PyErr_GetRaisedException/PyErr_SetRaisedException call. Est. 3–5 lines saved per site, 4 sites.

Recommended Action Plan

Immediate — crash bugs (Findings 1, 11, 12)

  1. Fix Finding 1 — _typeof.cpp:114: Move malloc NULL check before memcpy:
    w->buf = (char *) malloc(newsize);
    if (!w->buf) { PyErr_NoMemory(); return -1; }
    memcpy(w->buf, w->static_buf, w->allocated);
  2. Fix Finding 2 — _typeof.cpp:118: Save pointer before realloc:
    char *tmp = (char *) realloc(w->buf, newsize);
    if (!tmp) { PyErr_NoMemory(); return -1; }
    w->buf = tmp;
  3. Fix Finding 11 — modulemixin.c:210: Add Py_INCREF before Py_DECREF, or restructure loop 2 to use pycfunction_new (matching loop 1's pattern that correctly keeps the object alive).

Immediate — UAF and silent corruption (Findings 17, 21–25, 31–32)

  1. Fix Finding 17 — _box.c:147: Check PyLong_AsVoidPtr result:
    fnptr = PyLong_AsVoidPtr(mi_rel_fn);
    if (fnptr == (void*)-1 && PyErr_Occurred()) { goto cleanup; }
  2. Fix Findings 21–23 — _dispatcher.cpp: Add Py_VISIT(self->argnames) + visit the functions vector in traverse; add Dispatcher_tp_clear; add PyObject_GC_UnTrack as first line of Dispatcher_dealloc.
  3. Fix Finding 24 — _internal.c: Add Py_TPFLAGS_HAVE_GC, dufunc_traverse, dufunc_clear, and PyObject_GC_UnTrack in dufunc_dealloc.
  4. Fix Finding 25 — _dynfunc.c: Add PyObject_GC_UnTrack as first line of closure_dealloc.
  5. Fix Findings 31–32 — _dispatcher.cpp:1361,1475: Guard PyErr_Clear with fatal-exception check:
    if (!PyErr_ExceptionMatches(PyExc_MemoryError) &&
        !PyErr_ExceptionMatches(PyExc_KeyboardInterrupt) &&
        !PyErr_ExceptionMatches(PyExc_SystemExit)) {
        PyErr_Clear();
    } else { goto CLEANUP; }

Immediate — module init reference safety (Findings 26–30)

  1. Fix Findings 26–30 — Replace all PyModule_AddObject calls with PyModule_AddObjectRef (from pythoncapi_compat.h). Template:
    if (PyModule_AddObjectRef(m, "name", obj) < 0) {
        Py_DECREF(obj); return MOD_ERROR_VAL;
    }
    Py_DECREF(obj);  // for new references only

Short-term — reference counting in _typeof.cpp (Findings 3–10)

  1. Fix Finding 3 — typecode_using_fingerprint:655: On HASHTABLE_SET failure, call string_writer_clear(key); free(key).
  2. Fix Findings 4–6 — compute_fingerprint: Replace TRY macro with explicit error labels that DECREF acquired references. Alternatively, use Py_XDECREF at the return -1 sites using a goto error: cleanup block.
  3. Fix Findings 7–10 — ndarray_key/cache_ndarray_typecode/get_cached_ndarray_typecode/cache_typecode: Check every PyLong_FromLong result; use Py_XDECREF for temporaries; check return of PyDict_GetItem/PyDict_SetItem.

Short-term — GIL / error handling / null safety (Findings 12–20, 33–35)

  1. Fix Finding 12 — _nrt_python.c:396: Check PyObject_New, PyTuple_New; reverse NRT_MemInfo_acquire on failure paths.
  2. Fix Findings 13–15 — _dispatcher.cpp: Ensure Py_INCREF(args) happens before any goto CLEANUP; check PyDict_New() result.
  3. Fix Findings 16, 18 — Dispatcher_Insert: propagate PyLong_AsLong errors; check PyString_AsString in traceback_add_loc.
  4. Fix Finding 19 — _random.c:219: Add PyGILState_Release(gilstate) before Py_FatalError in the error block.
  5. Fix Findings 33–35 — Guard PyErr_Clear() calls with PyErr_ExceptionMatches before clearing.

Short-term — deprecated API migration (Findings 41–51)

  1. Fix Findings 41–42 — PyEval_GetLocals() → PyEval_GetFrameLocals() on 3.13+:
    #if PY_VERSION_HEX >= 0x030D0000
        locals = PyEval_GetFrameLocals();
    #else
        locals = PyEval_GetLocals();
    #endif
    Note: PyEval_GetFrameLocals() returns a new reference; adjust ownership logic.
  2. Fix Findings 46–48 — Migrate PyErr_Fetch/PyErr_Restore to single-exception API on 3.12+; add #if PY_VERSION_HEX < 0x030C0000 guard around PyUnicode_READY().
  3. Fix Finding 51 — Replace tstate->tracing/cframe->use_tracing with PyThreadState_EnterTracing/PyThreadState_LeaveTracing from pythoncapi_compat.h.

Longer-term — ABI compliance (Findings 36–45)

  1. Findings 36, 43–45 — _PySet_NextEntry, _PyErr_ChainExceptions*, _PyErr_StackItem: migrate to public APIs as Python 3.10/3.11 approach EOL. These private APIs may change between minor releases.
  2. Findings 37–40 — Private internal headers: track CPython changelog for any struct layout changes in 3.12 monitoring structs; confirm exact version guards for each.
  3. Findings 49–50 — _Py_c_pow → cpow(); _Py_HashSecret → sys.hash_info access at module init time.

Longer-term — structural (Findings 89–100)

  1. Finding 90 — Extract shared call_cfunc_impl to eliminate triplication; consolidate C_TRACE macro.
  2. Finding 89 — Extract namedtuple and ndarray branches from compute_fingerprint.
  3. Findings 93–96 — Update MOD_DEF documentation; migrate to range-based version checks in NB_SUPPORTED_PYTHON_MINOR.
  4. Finding 100 — Re-run parity-checker agent to assess .pyi stub accuracy.

External Tool Findings

cppcheck found 4 format-string mismatches in debug/test functions (not crash-risk but affect diagnostic output):

File Line Issue
numba/cext/dictobject.c 970 %llu with signed long long hash → use %lld
numba/cext/dictobject.c 1099 %u with signed int → use %d
numba/cext/setobject.c 760 %llu with signed long long hash → use %lld
numba/cext/setobject.c 863 %u with signed int → use %d

Sibling Bug Summary (from git-history-analyzer)

The git-history-analyzer found these structurally adjacent bugs not caught by other agents:

Sibling Origin Finding Location
compute_fingerprint default_val leaked via TRY Finding 4 (_fields leak) _typeof.cpp:364 → Finding 5
compute_fingerprint descr leaked via TRY Finding 4 _typeof.cpp:375 → Finding 6
ndarray_key Py_DECREF(NULL) Finding 10 (cache_typecode) _typeof.cpp:743 → Finding 7
string_writer_ensure realloc-into-self Finding 1 (memcpy before NULL) _typeof.cpp:118 → Finding 2
pycc_init_ loop 2 dangling ptr Pattern: no sibling; loop 1 correctly retains modulemixin.c:210 → Finding 11
NRT_meminfo_as_pyobject correctly checked NRT_adapt_ndarray_to_python_acqref is the only NRT function with unchecked allocs No sibling

TRY-macro audit note: The TRY(func, w, arg) macro in _typeof.cpp (line 207: if (func(w, arg)) return -1;) is a systematic ref-leak amplifier — any new reference acquired before a TRY call and not stored in a GC-visible location will be leaked on macro-triggered early return. A project-wide search for TRY( preceded by PyObject_Get*/PyArray_* is recommended.


Report generated by cext-review-toolkit. Before sharing with maintainers, see WORKING_WITH_MAINTAINERS.md: a 100-finding report is a homework assignment — reach out first, ask what format they need, and never auto-file issues or PRs.

C Extension Analysis Report — NumPy

Extension: numpy._core._multiarray_umath

Scope: C:\Users\שזדס\repos\numpy (entire project)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer


Executive Summary

NumPy's C extension layer is large and well-engineered, with active free-threading hardening ongoing since 2024. The codebase shows a disciplined pattern of goto fail error handling, NULL-initialised cleanup labels, and mutex-protected lazy imports. However, 15 FIX-class issues were found spanning five categories: crash-on-OOM during module import (lapack_litemodule, multiarraymodule), GIL released when NpyIter_Reset fails (ufunc_object.c — thread-state corruption on large array operations), PyErr_Fetch/Normalize/Restore used at 14+ sites (deprecated in 3.12, removed in 3.14 — no migration in progress), GC cycle-detection miss in PyUFuncType (tp_traverse omits _loops and userloops), and mask not propagated in np.ma.accumulate (data integrity regression). The most impactful immediate fixes are the NpyIter GIL issue, the LapackError NULL crash, and the pickle path NULL dereferences.


Extension Profile

Field Value
Primary module numpy._core._multiarray_umath
Additional modules numpy._core._multiarray_tests, numpy.linalg.lapack_lite, numpy.fft._pocketfft_umath, numpy._core._simd, numpy._core._umath_tests
Source files 182 .c + 28 .cpp + 229 .h
Init style Multi-phase (main module); Single-phase (test/helper modules)
Python minimum 3.12 (per pyproject.toml:16)
Limited API No (Py_LIMITED_API not defined)
GIL declaration Py_MOD_GIL_NOT_USED (free-threading ready)
Code generation Hand-written (.c.src template files for dtype loops)
Free-threading status Active migration — 6 thread-safety fixes Nov 2025–Apr 2026

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety 🟡 1 3 dtypemeta.c:1399 — GetCastingImpl new ref discarded
Error Handling 🔴 5 6 lapack_litemodule.c:400 — NULL LapackError crash
NULL Safety 🟡 3 1 methods.c:1831 — pickle path NULL in tuple slot
GIL Discipline 🟡 1 2 ufunc_object.c:1183 — NpyIter_Reset without GIL on failure
Resource Lifecycle 🟢 0 1 mapping.c:3020 — tmp_iter NpyIter leak on internal error
Module State 🟡 0 5 alloc.c:91 — madvise_hugepage non-atomic
Type Slots 🟡 1 1 ufunc_object.c:5528 — tp_traverse misses _loops/userloops
PyErr_Clear Safety 🟡 3 4 ctors.c:1393 — OOM swallowed (has TODO comment)
ABI / Private APIs 🟡 2 8 _Py_TPFLAGS_HAVE_VECTORCALL / pythoncapi-compat submodule
Version Compat 🔴 1 7 PyErr_Fetch/Normalize/Restore — 14+ sites, removed 3.14
C/Python Parity 🟡 1 2 np.ma.accumulate drops mask silently
Complexity 🟡 — 4 PyArray_MapIterNew 562 lines, 21 gotos (score 27.7)

🟢 No FIX | 🟡 1–3 FIX | 🔴 4+ FIX


Findings by Priority

Must Fix (FIX) — 15

Crash / Null Dereference

# Finding File:Line Agents
1 PyArray_GetCastingImpl returns new ref that's never decremented — per-legacy-dtype registration leak multiarray/dtypemeta.c:1399 refcount-auditor
2 PyObject_GetAttrString(mod, "_reconstruct") result stored in pickle tuple via PyTuple_SET_ITEM without NULL check — segfault on OOM during unpickling multiarray/methods.c:1831 null-safety-scanner
3 PyLong_FromLong(version) and PyObject_GetAttrString(self, "shape") stored in state tuple slots without NULL check — segfault on OOM multiarray/methods.c:1863 null-safety-scanner
4 ADDCONST macro: PyLong_FromLong/PyUnicode_FromString result passed to PyDict_SetItemString then Py_DECREF(s) unchecked — Py_DECREF(NULL) crash on OOM; macro expanded 14× multiarray/multiarraymodule.c:5069 null-safety-scanner, error-path-analyzer
5 PyErr_NewException result not NULL-checked before PyDict_SetItemString — on OOM: LapackError = NULL, all subsequent PyErr_Format(LapackError, ...) crash; file untouched since 2022 linalg/lapack_litemodule.c:400 error-path-analyzer, git-history-analyzer
6 set_flaginfo declared static void — PyDict_New() unchecked, all PyDict_SetItemString(NULL, ...) → NULL deref crash on OOM multiarray/multiarraymodule.c:4876 error-path-analyzer

GIL / Thread Safety

# Finding File:Line Agents
7 NpyIter_Reset(iter, NULL) called while GIL released via NPY_BEGIN_THREADS_THRESHOLDED. On OOM failure, npyiter_allocate_buffers calls PyErr_NoMemory() without GIL; NPY_END_THREADS never called on error path → permanent GIL-released state → thread-state corruption umath/ufunc_object.c:1183 gil-discipline-checker

PyErr_Clear / Silent Error Swallowing

# Finding File:Line Agents
8 PyErr_Clear() after PyMemoryView_FromObject swallows MemoryError — NumPy authors left TODO: "Should probably not blanket ignore errors" multiarray/ctors.c:1393 pyerr-clear-auditor
9 PyErr_Clear() in PyArray_CanCastTypeTo and PyArray_CanCastArrayTo silently returns 0 on any error including OOM from PyArray_CheckCastSafety — comment: "should likely be changed"; signature change (uint→int return) required to propagate multiarray/convert_datatype.c:725,828 pyerr-clear-auditor
10 F2PyCapsule_FromVoidPtr and F2PyCapsule_AsVoidPtr call PyErr_Clear() after PyCapsule_New/PyCapsule_GetPointer failure — callers receive NULL with no exception set, cannot distinguish capsule failure f2py/fortranobject.c:1426,1436 pyerr-clear-auditor

Type Slots

# Finding File:Line Agents
11 PyUFunc_Type.tp_traverse omits self->userloops (user-registered loop dict) and self->_loops (dtype→ArrayMethod dispatch dict) — both owned, both Py_XDECREF'd in tp_dealloc. GC cannot detect/break cycles through ufunc dispatch. Also: no tp_clear defined for PyUFunc_Type. umath/ufunc_object.c:5528 type-slot-checker

ABI / Private APIs

# Finding File:Line Agents
12 _Py_TPFLAGS_HAVE_VECTORCALL (private) used instead of Py_TPFLAGS_HAVE_VECTORCALL (public, stable since 3.9) — one-character fix umath/ufunc_object.c:6928 stable-abi-checker
13 pythoncapi-compat git submodule not initialized — npy_pycompat.h unconditionally includes pythoncapi-compat/pythoncapi_compat.h from an empty directory; build breaks on clean checkout _core/src/common/npy_pycompat.h:5 stable-abi-checker

Python 3.14 Compatibility Blocker

# Finding File:Line Agents
14 PyErr_Fetch / PyErr_NormalizeException / PyErr_Restore used at 14+ direct call sites across 13 files + 2 wrapper functions in npy_3kcompat.h:269-317 — deprecated in Python 3.12, removed in Python 3.14. No migration commits in git history. Multiple (see §Detail below) version-compat-scanner, stable-abi-checker, git-history-analyzer

Finding 14 call sites:

  • multiarray/arrayfunction_override.c:440,447,488
  • multiarray/arrayobject.c:824
  • multiarray/arraywrap.c:252,269
  • multiarray/array_method.c:787
  • multiarray/buffer.c:976
  • multiarray/convert_datatype.c:968
  • multiarray/ctors.c:2444
  • multiarray/methods.c:623,672
  • multiarray/multiarraymodule.c:2436
  • multiarray/nditer_api.c:2280
  • multiarray/nditer_pywrap.c:1205
  • multiarray/strfuncs.c:18
  • multiarray/textreading/rows.c:391
  • umath/dispatching.cpp:1152
  • _core/include/numpy/npy_3kcompat.h:269,315 (wrappers npy_PyErr_ChainExceptions + npy_PyErr_ChainExceptionsCause)

Replacement pattern (3.12+):

// Before (deprecated):
PyObject *exc, *val, *tb;
PyErr_Fetch(&exc, &val, &tb);
PyErr_NormalizeException(&exc, &val, &tb);
/* ... */
PyErr_Restore(exc, val, tb);

// After (3.12+):
PyObject *exc = PyErr_GetRaisedException();
/* ... */
PyErr_SetRaisedException(exc);

Parity

# Finding File:Line Agents
15 np.ma._MaskedBinaryOperation.accumulate silently drops the mask — sibling reduce correctly computes logical_and.reduce(m, axis); accumulate fills masked positions with filly then returns unmasked result. Tests only cover unmasked arrays (confirmed) ma/core.py:1163 parity-checker

Should Consider (CONSIDER) — 27

Refcount

# Finding File:Line
16 PyModule_AddObject return values for PINF/NINF/PZERO/NZERO/NAN unchecked — if PyFloat_FromDouble returns NULL, crash; if AddObject fails, float leaks umath/umathmodule.c:218
17 PyModule_AddObject unchecked for 4 ufunc objects in test module umath/_rational_tests.c:1508,1525,1544,1575
18 Ufunc result not NULL-checked before dereference in test module umath/_operand_flag_tests.c:78

Error Handling

# Finding File:Line
19 PyErr_WarnEx return value unchecked on MINGW build path — if -W error converts warning to exception, init continues with active exception multiarray/multiarraymodule.c:4951
20 Multiple unchecked return values in register_fail in struct ufunc test umath/_struct_ufunc_tests.c:63
21 PyErr_Clear() in _PyArray_GetPriority swallows arbitrary exceptions from __array_priority__ properties — two sites with author-left TODO[gh-14801] multiarray/multiarraymodule.c:167,178
22 PyErr_Clear() in get_array_function swallows arbitrary exceptions from __array_function__ attribute — TODO[gh-14801] multiarray/arrayfunction_override.c:32
23 Dtype converter error cleared before fallback in NpyIter creation — on OOM, error is destroyed; if fallback also fails, caller gets error with no exception multiarray/nditer_pywrap.c:482
24 VOID_nonzero silently skips field on _unpack_field failure — on OOM, a structured-array field is not checked for nonzero, producing incorrect boolean result multiarray/arraytypes.c.src:2693

NULL Safety

# Finding File:Line
25 scalar_value() result passed to memcpy without NULL check — memcpy(dst, NULL, n) if NPY_STRING scalar has invalid buffer multiarray/scalarapi.c:251

Module State / Free-Threading

# Finding File:Line
26 madvise_hugepage is int (plain store), mutated by Python-callable _set_madvise_hugepage — needs _Atomic int under Py_MOD_GIL_NOT_USED multiarray/alloc.c:91
27 get_sfloat_dtype_initialized lazy-init flag not mutex-protected — two threads can both pass if (!initialized) under free-threading multiarray/_scaled_float_dtype.c:1096
28 module_loaded non-atomic bool in 3 modules (multiarraymodule.c:4927, umathmodule.c, _umath_linalg) multiple
29 typeDict plain global pointer written by array_set_typeDict (public API) without lock — use-after-free race if called concurrently with dtype lookup under free-threading multiarray/descriptor.c:143
30 userdescrs / NPY_NUMUSERTYPES non-atomic realloc in PyArray_RegisterDataType — concurrent calls or concurrent reads from scalartypes.c.src cause heap corruption multiarray/usertypes.c:224
31 _simd.c and 3 test modules use single-phase init + PyUnstable_Module_SetGIL — main module already uses stable Py_mod_gil slot _core/src/_simd/_simd.c:93 + 3 test files

Resource Lifecycle

# Finding File:Line
32 tmp_iter NpyIter not freed on NpyIter_CreateCompatibleStrides failure — internal error path only, but each failure leaks one NpyIter struct multiarray/mapping.c:3020

Type Slots

# Finding File:Line
33 dtypemeta_traverse has unconditional assert(0) — latent, never triggered (guarded by tp_is_gc), but will abort debug builds when Python-definable DType API is opened multiarray/dtypemeta.c:529

Compatibility / Deprecated APIs

# Finding File:Line
34 npy_PyFile_OpenFile calls PyEval_GetBuiltins() deprecated in 3.13 — result passed to PyDict_GetItemString without NULL check _core/include/numpy/npy_3kcompat.h:245
35 Dead Py_OptimizeFlag branch — removed in 3.12, unreachable since NumPy min is 3.12 multiarray/compiled_base.c:1464
36 _PyDict_GetItemStringWithError private API — guarded < 0x030D0000; drop guard once min is 3.13 f2py/fortranobject.c:311
37 14 sites use 0x30d00b3 (7-digit hex) instead of canonical 0x030d00b3 (8-digit hex) — numerically equal, style only npy_argparse.c, npy_import.c/h, static_string.c, umath_linalg.cpp
38 Py_mod_multiple_interpreters: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED guard always evaluates true since min is 3.12 — can be removed multiarray/multiarraymodule.c
39 PyDict_GetItemString (borrowed ref, deprecated 3.13) used in initumath — replace with PyDict_GetItemStringRef umath/umathmodule.c:224

ABI / Private APIs

# Finding File:Line
40 _Py_HashDouble private API used for float/complex scalar hashing via npy_pycompat.h:7 — no public equivalent; consider inlining CPython's IEEE 754 hash formula _core/src/common/npy_pycompat.h:7
41 PyUnstable_SetImmortal used at 3 sites under Py_GIL_DISABLED guard — no stable alternative; monitor CPython stabilization multiarray/array_method.c:492 + 2 sites
42 ->ob_type direct struct access instead of Py_TYPE() at 4 locations multiarray/methods.c:2034, umath/override.c:77, umath/_rational_tests.c:451,469,769

Parity

# Finding File:Line
43 numpy.linalg.matrix_power stub accepts object-dtype stacks but runtime raises NotImplementedError — stub should encode the restriction linalg/_linalg.py:742
44 np.ma._MaskedBinaryOperation has no reduceat method — np.ma.add.reduceat(...) raises AttributeError instead of NotImplementedError ma/core.py

Complexity Hotspots (informational, for targeted review)

# Function File Score Complexity Drivers
45 PyArray_MapIterNew multiarray/mapping.c:2783 27.7 562 lines, 21 gotos, 3 failure labels, conditional extra_op ownership
46 PyUFunc_GenericReduction umath/ufunc_object.c:3637 17.4 270 lines, 21 gotos, 3 arg-parse scopes sharing one fail:
47 PyArray_Where multiarray/multiarraymodule.c:3235 16.9 Nesting depth 7, goto fail inside threaded loop
48 ufunc_object.c (file-level) umath/ufunc_object.c — 163 gotos, 20 NPY_BEGIN_THREADS transitions — primary site for GIL-discipline bugs

Tensions

  • Finding 14 (PyErr_Fetch) vs. minimum Python version: With min 3.12, PyErr_GetRaisedException is available unconditionally — no version guard needed. The migration is purely mechanical and can use pythoncapi-compat for any downstream consumers still on 3.11.
  • Finding 29/30 (typeDict/userdescrs races) vs. "import-time only" assumption: Both set_typeDict and PyArray_RegisterDataType are public C APIs. The "called at import time while single-threaded" assumption is not enforced by the API surface and is violated by the module declaring Py_MOD_GIL_NOT_USED. These are latent, not theoretical.
  • Finding 9 (CanCastTypeTo signature) vs. public API stability: PyArray_CanCastTypeTo and PyArray_CanCastArrayTo have unsigned char return types (public API). Proper error propagation requires a signature change — a larger breaking change. A MemoryError during cast-safety checks causing a false "cannot cast" result may be acceptable as a documented limitation.

Policy Decisions (POLICY) — 6

# Finding
P1 LapackError global — write-once at module exec; not_supported pattern is correct opt-out
P2 PyDataMem_DefaultHandler — correctly immortalized under Py_GIL_DISABLED; PyDataMem_Handler uses ContextVar for per-thread override
P3 _PyLong_Bytes local function with _Py_-looking name — purely local, no name collision risk
P4 PyArray_Type omits Py_TPFLAGS_HAVE_GC — documented performance trade-off; object-dtype cycle leaks are known and accepted
P5 PyArrayDescr_Type.tp_hash runtime-patched before PyType_Ready — fragile ordering but correct; hash is intentionally PyArray_DescrHash (structural, not identity)
P6 PyUnstable_SetImmortal and PyUnstable_Object_IsUniquelyReferenced* — load-bearing for free-threading performance; no stable alternative exists; monitor CPython stabilization

Strengths

  • Multi-phase init is correctly implemented in the main module with proper m_traverse/m_clear/m_free for module state.
  • npy_import.c double-checked locking is correctly implemented using atomic_load/store + PyMutex — textbook-correct for lazy import caching under free-threading.
  • goto fail cleanup discipline is generally excellent — virtually all NpyIter allocations, LAPACK workspace allocations, and NPY_cast_info structs are correctly freed on error paths (only 1 resource leak found across 439 C/H files).
  • extobj.c SIGFPE handling is correct — floating-point error detection uses hardware status flags read after GIL reacquiry, plus PyContextVar for per-thread state; no signal-safety issues.
  • LAPACK workspace management in umath_linalg.cpp is clean throughout — all init_* functions NULL-initialise workspace pointers and use PyMem_RawFree(NULL) safety via goto no_memory patterns.
  • PyErr_Fetch sites are complete — every existing fetch/restore pair correctly uses a matching 3-tuple; no half-migrated sites.
  • pocketfft buffer management uses RAII (pocketfft::detail::arr<T>) — no manual buffer management.
  • Active free-threading hardening — 6 thread-safety fixes in 6 months (hash, ufunc dispatch cache, buffer protocol, auxdata freelist, _loops list→dict, array coercion critical sections).

Code Removal Opportunities

Since NumPy's minimum Python is 3.12, the following API replacements are available:

API / Pattern Replacement (available since) Estimated Savings Sites in NumPy
PyModule_AddObject + NULL check + Py_DECREF on failure PyModule_AddObjectRef (3.10+) ~3 lines per site ~8 sites in umathmodule.c, test modules
PyType_FromSpec + PyType_Ready + Py_INCREF + PyModule_AddObject chain PyModule_AddType (3.10+) 4–8 lines per type Multiple type registrations in multiarraymodule.c
PyErr_Fetch + PyErr_NormalizeException + PyErr_Restore (3-var triple) PyErr_GetRaisedException / PyErr_SetRaisedException (3.12+) ~6 lines → ~2 lines per site 14 direct sites + 2 wrapper functions in npy_3kcompat.h
PyDict_GetItemString (borrowed ref) PyDict_GetItemStringRef (3.13+ only) 2 lines (add Py_XDECREF) ~20 sites — wait for 3.13 minimum
_PyDict_GetItemStringWithError (private) PyDict_GetItemStringRef (3.13+) — f2py/fortranobject.c:311 — drop guard at 3.13 minimum
Dead Py_OptimizeFlag branch Delete ~5 lines compiled_base.c:1464

Recommended Action Plan

Immediate (FIX items — blocking or crash-risk)

  1. [Finding 7] ufunc_object.c:1183 — Add NPY_END_THREADS before goto fail in execute_ufunc_loop:

    if (NpyIter_Reset(iter, NULL) != NPY_SUCCEED) {
        NPY_END_THREADS;  // must reacquire GIL before any Python API call
        NPY_AUXDATA_FREE(auxdata);
        NpyIter_Deallocate(iter);
        return -1;
    }
  2. [Finding 5] lapack_litemodule.c:400 — Add NULL check:

    LapackError = PyErr_NewException("numpy.linalg.lapack_lite.LapackError", NULL, NULL);
    if (LapackError == NULL) { return -1; }
    if (PyDict_SetItemString(d, "LapackError", LapackError) < 0) { return -1; }
  3. [Finding 2/3] methods.c:1831-1865 — Add NULL checks on array_reduce's PyObject_GetAttrString and PyLong_FromLong results before PyTuple_SET_ITEM.

  4. [Finding 4/6] multiarraymodule.c:5069 and :4876 — Fix ADDCONST macro to check results; convert set_flaginfo to return int and add NULL check on PyDict_New().

  5. [Finding 11] ufunc_object.c:5528 — Add Py_VISIT(self->userloops) and Py_VISIT(self->_loops) to ufunc_traverse; add ufunc_clear implementing tp_clear.

  6. [Finding 12] ufunc_object.c:6928 — Replace _Py_TPFLAGS_HAVE_VECTORCALL with Py_TPFLAGS_HAVE_VECTORCALL (one character).

  7. [Finding 13] npy_pycompat.h:5 — Initialize pythoncapi-compat submodule (git submodule update --init numpy/_core/src/common/pythoncapi-compat) or vendor the single header.

  8. [Finding 1] dtypemeta.c:1399 — Store and release PyArray_GetCastingImpl result:

    PyObject *castimpl = PyArray_GetCastingImpl(dtype_class, dtype_class);
    if (castimpl == NULL) { return NULL; }
    Py_DECREF(castimpl);
  9. [Finding 15] ma/core.py:1163 — Implement mask propagation in _MaskedBinaryOperation.accumulate:

    masked_result._mask = umath.logical_or.accumulate(m, axis) if m is not nomask else nomask

    Add regression test using a masked array.

Short-term (CONSIDER items — important)

  1. [Finding 14] PyErr_Fetch migration — Open a tracking issue for migrating all 14 sites to PyErr_GetRaisedException/PyErr_SetRaisedException. Update npy_3kcompat.h:npy_PyErr_ChainExceptions* wrappers. Required before Python 3.14 support.

  2. [Findings 8/9/10] PyErr_Clear — Apply selective-clear patches at ctors.c:1393 (guard OOM), convert_datatype.c:725,828 (propagate non-TypeError), f2py/fortranobject.c:1426,1436 (remove clear, let exceptions propagate).

  3. [Findings 26/27/28] Module state atomics — Convert madvise_hugepage to _Atomic int; add PyMutex to get_sfloat_dtype_initialized lazy-init; convert module_loaded bools to _Atomic int.

  4. [Findings 29/30] typeDict / userdescrs — Add PyMutex guards to array_set_typeDict and PyArray_RegisterDataType. Mark with // Thread-safety: write-once-at-import only comment.

  5. [Findings 21/22] __array_priority__ / __array_function__ exceptions — Guard PyErr_Clear with PyErr_ExceptionMatches(PyExc_AttributeError) filter (track via gh-14801).

  6. [Finding 32] mapping.c:3020 — Add NpyIter_Deallocate(tmp_iter) before goto fail in PyArray_MapIterNew.

  7. [Findings 43/44] parity — Fix matrix_power stub; add reduceat stub/impl to _MaskedBinaryOperation.

Longer-term (POLICY / CONSIDER — architectural)

  1. [Finding 40] _Py_HashDouble — Track CPython stabilization; interim: inline the hash formula or backport via pythoncapi-compat.

  2. [Finding 31] Single-phase init modules — Migrate _simd.c and 3 test modules from single-phase + PyUnstable_Module_SetGIL to multi-phase + stable Py_mod_gil slot.

  3. [Finding 35] Dead code removal — Remove dead Py_OptimizeFlag branch in compiled_base.c:1464 (removed in 3.12, always-false).

  4. Code removal — Replace PyErr_Fetch/Normalize/Restore triples with PyErr_GetRaisedException (saves ~6 lines per site, 84+ lines total); replace PyModule_AddObject chains with PyModule_AddObjectRef.


Cross-Reference: Complexity and Findings

The complexity analysis identified mapping.c and ufunc_object.c as the highest-risk files. The actual findings confirm this correlation:

  • Finding 7 (GIL bug) is in ufunc_object.c
  • Finding 11 (tp_traverse) is in ufunc_object.c
  • Finding 32 (NpyIter leak) is in mapping.c
  • git-history-analyzer identified ufunc_generic_fastcall as having other early-return NULL paths that may skip cleanup (related to commit 070954a2)
  • git-history-analyzer identified PyArray_AsCArray's nd > 3 early exit as leaking ap (not addressed in commit dd102ade)

These complexity hotspots are the primary targets for future audit passes.


Before sharing this report with maintainers: A 48-finding report dropped without context is a 48-hour homework assignment. Reach out first — ask what format and scope they want. Security-adjacent findings (GIL/thread-safety, crash on OOM) warrant direct discussion before filing public issues. The synthesis above is an internal analysis artifact.

C Extension Analysis Report

Extension: Pillow

Scope: src/ (entire extension)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer

External Tools: cppcheck (without compile_commands.json)

Date: 2026-06-27


Executive Summary

Pillow's C extension layer (8 modules, 112 C/H files, ~45,841 lines) is large and mature but carries a meaningful backlog of correctness issues accumulated over its long lifespan. The most critical cluster is in _imagingcms.c, which has four distinct error-path bugs where failure returns -1 or Py_None without setting a Python exception, producing silent failures or misleading error messages. A double-ImagingDelete in _split can produce heap corruption on any OOM during a multi-band split, and _imagingmath.c's module-init PyErr_Clear means the _imagingmath module can silently lose all its math functions at import without any error being reported. The 14 FIX findings are all actionable without restructuring; none require the heap-type migration that the free-threading report identified as the gating prerequisite for thread-safety work. The ~30 CONSIDER findings cover genuine (but non-critical) resource leaks, missing GIL releases, GC-participation gaps, stub inaccuracies, and complexity hotspots.


Extension Profile

  • Modules: _avif, _imaging, _imagingcms, _imagingft, _imagingmath, _imagingmorph, _webp, _imagingtk (+ encode.c, decode.c, path.c, outline.c, display.c)
  • C files: 112 (hand-written), ~45,841 lines
  • Init style: Multi-phase (Py_mod_exec slot in all 8 modules)
  • Python targets: >= 3.10 (tested through 3.14)
  • Limited API: No
  • Types defined: 16 static PyTypeObject declarations across 10 files
  • Free-threading: All 8 modules declare Py_MOD_GIL_NOT_USED (see ft_review.md for thread-safety analysis — that work is tracked separately)
  • External tool baseline: 2 cppcheck findings (1 missingReturn in _avif.c:582 confirmed false positive by null-safety scanner; 1 moduloAlwaysTrueFalse in _imaging.c:1686)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 3 1 Borrowed-ref UAF in getlist/_putdata
Error Handling R 7 2 Four _imagingcms silent-failure returns
NULL Safety G 0 1 ImagingNewDirty NULL unchecked in _resize
GIL Discipline G 0 5 avifDecoderNthImage holds GIL during AV1 decode
Resource Lifecycle Y 2 3 LibTiff double-DECREF; JPEG encoder malloc leak
Module State G 0 0 m_size=0, no static PyObject* caches — clean
Type Slots G 0 3 Decoder/Encoder types missing GC participation
PyErr_Clear Safety Y 2 1 install_unary/install_binary swallows MemoryError
ABI Compliance G 0 0 Not claimed — 10 POLICY findings assess feasibility
Version Compat G 0 4 Non-canonical version guard; Py_IsNone() opportunity
C/Python Parity Y 1 4 getfont stub breaks internal PIL type-check
Complexity G 0 6 font_render_impl (423 lines, nesting 9)
History/Churn G 0 4 Recurring refcount pattern in _imagingft.c

G = No FIX findings | Y = 1–3 FIX findings | R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 14

# Finding File:Line Agents
1 getlist: PySequence_Fast_GET_ITEM on mutable list → borrowed-ref UAF; concurrent mutation frees item before PyLong_AsInt call src/_imaging.c:466 refcount-auditor
2 _putdata: same pattern — borrowed ref from mutable list passed to getink() which runs arbitrary Python code; highest-severity UAF window src/_imaging.c:1738 refcount-auditor
3 _putdata: PyErr_Clear() at 4 sites inside pixel-copy loop swallows TypeError/float-conversion errors; loop continues with value=-1.0 producing silently corrupt pixels src/_imaging.c:1696,1715,1727,1750 error-path-analyzer
4 _split: double ImagingDelete on error path — bands 0..i-1 freed by Py_DECREF(list) via _dealloc, band i freed inside PyImagingNew, then cleanup loop frees all again src/_imaging.c:2513–2524 refcount-auditor, error-path-analyzer
5 _font_getmask failed: label: Py_RETURN_NONE with active exception; callers cannot distinguish failure from a valid None result src/_imaging.c:2913 error-path-analyzer
6 pyCMSdoTransform: returns C int -1 for size-mismatch without setting a Python exception; caller converts the return to a Python int, silently returning True/-1 instead of raising src/_imagingcms.c:357–359,569 error-path-analyzer
7 _is_intent_supported: Py_RETURN_NONE when PyDict_New() fails; caller expects a dict and will crash or raise confusing AttributeError src/_imagingcms.c:932–934 error-path-analyzer
8 _is_intent_supported: Py_RETURN_NONE when loop allocation fails; same caller-visible impact as Finding 7 src/_imagingcms.c:959–963 error-path-analyzer
9 _is_intent_supported: PyDict_SetItem return value unchecked; silent dict construction failure src/_imagingcms.c:965 error-path-analyzer
10 PyImaging_LibTiffEncoderNew: get_packer already frees the encoder object on failure; the caller then calls Py_DECREF(encoder) a second time → double-free / heap corruption src/encode.c:750–754 resource-lifecycle-checker
11 PyImaging_JpegEncoderNew: 3 separate malloc/calloc failure paths return NULL without Py_DECREF(encoder) → encoder object leaked src/encode.c:1235–1270 resource-lifecycle-checker
12 install_unary / install_binary: PyErr_Clear() swallows MemoryError from PyCapsule_New or PyDict_SetItemString; setup_module always returns 0; the module appears imported but all 38 math function capsules can be silently absent → AttributeError on first use src/_imagingmath.c:260,269 pyerr-clear-auditor
13 _setfd (decoder + encoder): missing Py_XDECREF(state->fd) before overwriting state->fd; calling setfd twice leaks the first file object; setimage immediately adjacent correctly does this src/decode.c:252, src/encode.c:319 type-slot-checker
14 _imagingft.pyi getfont(): stub declares font_bytes and layout_engine as required positional parameters, but C format "etf|nsy#n" makes them optional; Pillow's own ImageFont.py:298–300 calls getfont without font_bytes → mypy/pyright reports a false error on valid internal code src/PIL/_imagingft.pyi:62–69 parity-checker

Should Consider (CONSIDER) — 31

Refcount / Memory

# Finding File:Line
15 _resize: NULL return from ImagingNewDirty passed unchecked to ImagingTransform; produces confusing ValueError on OOM instead of MemoryError src/_imaging.c:1992–1996
16 LibTiff encoder TIFF-tag loop: 8 calloc calls guarded with if (av) silently skip the corresponding TIFFSetField on OOM; caller sees a misleading RuntimeError("Error setting from dictionary") without any OOM indicator src/encode.c:879–1003
17 cms_profile_new: cmsHPROFILE handle leaked if PyObject_New fails; LittleCMS profile will not be closed src/_imagingcms.c:87–98
18 cms_transform_new: cmsHTRANSFORM handle leaked if PyObject_New fails; same pattern as Finding 17 src/_imagingcms.c:192–203
19 path_compact: realloc result assigned to temporary but discarded; on failure the original buffer is leaked and the path object is left with xy=NULL, count>0 — iterator over a compacted path will then dereference NULL src/path.c:342
20 PyPath_Flatten: PyErr_Clear() after PyObject_GetBuffer failure; intended for BufferError fallback but also clears unrelated MemoryError src/path.c:193
21 _webp.c:44–61 (HandleMuxError): sprintf into 100-byte message[] with kErrorMessages[-err]; structurally identical to the CVE-2021-34552 sprintf pattern that was fixed elsewhere in Pillow; array bounds rely on err ∈ [-4,-1] which is asserted but not statically provable src/_webp.c:44–61
22 JpegEncode.c:404–406 and _webp.c:759–767: sprintf for version integers into 20-byte static buffers; safe in practice (integer ranges are bounded) but follows the sprintf-into-fixed-buffer pattern that has been a recurring CVE source in Pillow (CVE-2024-28219) src/libImaging/JpegEncode.c:404, src/_webp.c:759
23 _imagingft.c:1440–1491 (font_setvaraxes_impl): same class of per-item Py_DECREF inside a malloc'd loop as the refcount bugs fixed in 9.5.0 and 10.2.0; OverflowError from PyFloat_AsDouble inside the loop may not terminate the loop cleanly src/_imagingft.c:1440–1491

GIL / Concurrency

# Finding File:Line
24 avifDecoderNthImage: full AV1 frame decode holds the GIL; the adjacent avifImageYUVToRGB at line 821 correctly releases it via ImagingSectionEnter/Leave — asymmetric release src/_avif.c:785
25 avifDecoderParse: container parsing holds the GIL; for large AVIF files this can be long-running src/_avif.c:675
26 WebPAnimEncoderAssemble: per-frame WebPAnimEncoderAdd (line 289) correctly releases GIL; the final assembly of all frames into the output bitstream does not — asymmetric with the decoder fix from 10.3.0 (#7782) src/_webp.c:332
27 cmsOpenProfileFromFile: LittleCMS disk I/O holds the GIL during profile load; the only potentially long-running CMS operation src/_imagingcms.c:109
28 display.c:539–540: PyThreadState_Swap / PyEval_RestoreThread pattern in Win32 message loop callback instead of the modern PyGILState_Ensure / PyGILState_Release idiom; fragile if the callback fires from a thread that never held the GIL src/display.c:539–540

Type Slots / GC

# Finding File:Line
29 ImagingDecoderType: holds user-supplied fd (file-like object, set via setfd) but declares no Py_TPFLAGS_HAVE_GC, tp_traverse, or tp_clear; reference cycles through fd are invisible to the cyclic GC src/decode.c:280–286
30 ImagingEncoderType: same issue; encoders hold fd for the full encode lifetime, making the cycle risk slightly more acute src/encode.c:349–355
31 Geometry.c ImagingGenericTransform: adjacent to the null-dereference-before-null-check fixed in #8398 for ImagingTransformAffine; verify NULL handling is complete in the generic fallback transform path src/libImaging/Geometry.c:774

Stub Parity

# Finding File:Line
32 Font.getsize, Font.getlength, Font.render: C format strings have many optional parameters (after |), but stubs mark all parameters required; external callers get false type errors when omitting optional params src/PIL/_imagingft.pyi:23–56
33 ImagingCore: __arrow_c_schema__ and __arrow_c_array__ Arrow C Data Interface protocol methods registered in C (_imaging.c:3761–3762) are absent from the stub; Arrow-aware libraries cannot verify the protocol statically src/PIL/_imaging.pyi
34 _Outline: stub only declares close(); C module registers line, curve, move, transform, close — 4 methods are undeclared src/PIL/_imaging.pyi:25–27
35 Five stub files (_avif.pyi, _webp.pyi, _imagingmath.pyi, _imagingmorph.pyi, _imagingtk.pyi) are entirely __getattr__: Any; each corresponding C module exports real typed types and/or methods that receive no stub coverage src/PIL/_avif.pyi etc.

Version Compatibility / Modernization

# Finding File:Line
36 Non-canonical version guard: #if PY_MAJOR_VERSION > 3 || PY_MINOR_VERSION > 11 — bare PY_MINOR_VERSION lacks major-version qualification and is confusing to readers; replace with #if PY_VERSION_HEX >= 0x030C0000 src/_imagingft.c:140
37 Py_FileSystemDefaultEncoding #else branch: correctly guarded to Python 3.10–3.11 only; becomes dead code when minimum advances past 3.11 — track for cleanup at that time src/_imagingft.c:161–177
38 _imagingcms.c type registration: PyType_Ready + PyModule_AddObjectRef two-step (4 lines/type); PyModule_AddType (3.10+) replaces with one call, saving 4 lines total src/_imagingcms.c:1455–1463
39 7 sites use == Py_None / != Py_None pointer comparisons; Py_IsNone() is the idiomatic form since 3.10 (Pillow's declared minimum) src/_imagingft.c:333,423; src/encode.c:238,1101; src/_avif.c:339; src/_webp.c:254; src/decode.c:171

Complexity Hotspots

# Finding File:Line Score Refactor?
40 font_render_impl: 423 lines, ~62 branches, nesting 8–9; three duplicated pixel-blending paths; 15 goto glyph_error targets make future resource changes unsafe src/_imagingft.c:842 11.5 Yes — extract blend_scanline()
41 PyImaging_LibTiffEncoderNew: 390 lines, 72 branches; calloc/SetField/free block copy-pasted 7 times with only type changes; OOM silently skips SetField without a proper error (latent, exposed by refactoring) src/encode.c:679 11.2 Yes — extract set_tiff_scalar_field / set_tiff_array_field
42 AvifEncoderNew: 17 parameters; three repeated avifImageSetMetadata* blocks src/_avif.c:214 7.8 Partially — extract set_avif_metadata()
43 bounding_box_and_anchors: 175 lines, 35 branches, 10 params; 2×2 nested anchor-character dispatch src/_imagingft.c:580 7.1 Partially — extract compute_anchor_offsets()
44 _prepare_lut_table: free_table_data sentinel controls 3 cleanup exit paths; extract buffer-detection phase to eliminate the sentinel src/_imaging.c:825 5.3 Yes
45 getink: nesting 5 inside IMAGING_TYPE_UINT8 + bands==1 path; early-return guard clauses would reduce to 3 src/_imaging.c:578 5.4 Partially

Tensions

  • _setfd refcount vs. type-slot scope: Finding 13 (_setfd missing Py_XDECREF) was discovered during the type-slot audit (dealloc review revealed the overwrite pattern) but belongs to refcount correctness. Reported here as FIX regardless of scope attribution.
  • path_compact realloc (Finding 19): null-safety and error-path agents both flagged this; merged into one CONSIDER here.
  • _imagingft.c FT_Done_MM_Var + ft_library_mutex: lock-discipline and shared-state auditors disagreed — lock-discipline said add mutex; shared-state said FT 2.6+ allocator may protect it. Resolution: report conservatively in ft_review.md (add mutex for consistency). Not duplicated here.
  • WebPAnimEncoderAssemble GIL: flagged by both GIL discipline (Group B) and git-history analysis (Group F, noting asymmetry with decoder fix). Reported once as Finding 26.

Policy Decisions (POLICY) — 10

# Finding
46 Py_mod_multiple_interpreters slot absent from all 8 module defs; defaults to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED — correct given 16 static types — but an explicit declaration makes the intent clear and is required for subinterpreter readiness signaling
47 Inconsistent dealloc style: 12 types use PyObject_Del(self) (older idiom); 4 newer types (_avif.c, _webp.c) use Py_TYPE(self)->tp_free(self) (preferred — respects subclass overrides); standardize to Py_TYPE(self)->tp_free(self)
48 LittleCMS global context (cmsHANDLE default context): cmsGetContextUserData is never called; all CMS operations use the global context; concurrent threads sharing a single LittleCMS context relies on LittleCMS's internal thread safety, which is documented but not verified in Pillow's CI
49 16 static PyTypeObject structs are the core blocker for stable ABI (Py_LIMITED_API) adoption — all 16 need PyType_FromSpec() conversion; this is also the dominant cost for the free-threading heap-type migration; the two roadmaps can be unified
50 _PyBytes_Resize private API (encode.c:144): not in the stable ABI; replace with PyBytes_FromStringAndSize(PyBytes_AsString(buf), status) + Py_DECREF of old buffer for limited-API compliance
51 Py_TYPE(item)->tp_name field access (path.c:616): PyTypeObject is opaque in limited API; replace with PyType_GetName() (3.11+) or PyObject_GetAttrString((PyObject*)Py_TYPE(item), "__name__") (3.10 fallback)
52 ~20 non-limited-API fast accessor macros across 6 files (PyTuple_GET_ITEM/SIZE ×15, PyList_GET_SIZE/SET_ITEM ×2, PyBytes_AS_STRING ×2, PyTuple_SET_ITEM ×1); all have stable-ABI equivalents
53 PyUnicode_READ_CHAR (_imagingft.c:461): not in limited API; replace by converting unconditionally to Py_UCS4* via PyUnicode_AsUCS4Copy (stable ABI since 3.3)
54 pythoncapi_compat.h already vendored but only included in 3 of 8 modules (encode.c, path.c, _imagingft.c); the other 5 modules would silently fail to compile if a developer adds any 3.13+ API to them; add #include "thirdparty/pythoncapi_compat.h" to the remaining 5 (or include it from libImaging/Imaging.h transitively)
55 Limited API feasibility: migration is feasible with ~1–2 weeks of work; dominant cost is the 16-type conversion (subsumed by free-threading migration); codec library coupling (libtiff, libfreetype, etc.) means abi3 wheels still need per-platform builds, but eliminates the CPython-version dimension — reducing ~30 wheels to ~6

Strengths

  • PyModule_AddObjectRef throughout: all module-attribute registrations use the steal-safe 3.10+ API — no PyModule_AddObject reference-steal bugs.
  • Multi-phase init everywhere: all 8 modules use Py_mod_exec correctly; setup_module functions hold the import lock, so PyModule_GetDict usage inside them is safe.
  • ImagingInstance.mutex and ImagingMemoryArena.mutex: per-image and per-arena PyMutex fields in Imaging.h correctly placed for free-threading (see ft_review.md).
  • pythoncapi_compat.h vendored and current: at commit c84545f0e1e21757d4901f75c47333d25a3fcff0, covering 3.14 APIs; PyDict_GetItemRef, PyList_GetItemRef, and Py_BEGIN_CRITICAL_SECTION backports are available.
  • No PyModule_AddObject steal bugs: confirmed absence across all modules.
  • PyErr_Fetch / PyErr_Restore / PyErr_NormalizeException (deprecated 3.12): not present.
  • _Py_* private API: none used in hand-written code.
  • _imagingcms.pyi: complete and correct — all 40 CmsProfile properties, both CmsTransform methods, all 6 module-level functions verified against C signatures.
  • avifDecoderNthImage / avifImageYUVToRGB: consistent GIL-release discipline on the color-conversion step; only the decode-trigger step (Finding 24) is missing it.
  • _avif.c:582 missingReturn (cppcheck): confirmed false positive — every code path in _encoder_add terminates in return NULL or Py_RETURN_NONE.

Code Removal Opportunities

Based on Pillow's Python >= 3.10 minimum:

  • PyModule_AddType (3.10+): replaces PyType_Ready + PyModule_AddObjectRef per exported type. Two exported types in _imagingcms.c can use this now (Finding 38) — saves ~4 lines. More types can adopt it after the heap-type migration.
  • Py_IsNone / Py_IsTrue / Py_IsFalse (3.10+): 7 == Py_None comparisons (Finding 39) can be replaced with the idiomatic macro.
  • Py_FileSystemDefaultEncoding #else block (Finding 37): ~18 lines of dead code removable after dropping Python 3.11.
  • PyImport_ImportModuleAttrString (3.14+): not yet available at minimum version — track for later when minimum advances.

Recommended Action Plan

Immediate (FIX — no restructuring required)

  1. Fix Finding 4 (_split double-ImagingDelete): on the error path after PyImagingNew fails, do not call ImagingDelete(bands[i]) — the failed PyImagingNew already freed it; only call ImagingDelete for bands 0..i-1.
  2. Fix Findings 6–9 (_imagingcms.c silent failures): all four are in pyCMSdoTransform and _is_intent_supported; add PyErr_SetString(PyExc_RuntimeError, ...) before every silent return -1 or Py_RETURN_NONE.
  3. Fix Finding 12 (_imagingmath.c module init): change install_unary/install_binary to return int and propagate errors; have setup_module check the return and propagate -1.
  4. Fix Finding 10 (LibTiff double-DECREF): in PyImaging_LibTiffEncoderNew, when get_packer returns failure, do not call Py_DECREF(encoder) — get_packer already freed it.
  5. Fix Finding 11 (JPEG encoder malloc leaks): add Py_DECREF(encoder); return NULL; at the 3 malloc/calloc failure paths in PyImaging_JpegEncoderNew.
  6. Fix Finding 13 (_setfd refcount): add Py_XDECREF(state->fd); before state->fd = fd; in both decode.c:_setfd and encode.c:_setfd.
  7. Fix Findings 1–2 (borrowed-ref UAF): in getlist and _putdata, call Py_INCREF(op) on the list item before passing it to PyLong_AsInt / PyFloat_AsDouble / getink(), and Py_DECREF(op) after.
  8. Fix Finding 3 (_putdata PyErr_Clear in pixel loop): replace the 4 PyErr_Clear() sites with goto error; to surface conversion failures instead of silently using -1.0.
  9. Fix Finding 5 (_font_getmask failed:): replace Py_RETURN_NONE with return NULL; — the exception is already set by the preceding call.
  10. Fix Finding 14 (getfont stub): add = 0 / = "" / = b"" defaults to the 4 optional parameters; = ... is also acceptable if the caller always provides them.

Short-term (CONSIDER)

  1. Finding 17–18 (LittleCMS handle leaks): use Py_XDECREF(profile) / cmsCloseProfile at the error exit in cms_profile_new; same pattern in cms_transform_new.
  2. Findings 24–27 (GIL release gaps): wrap avifDecoderNthImage, avifDecoderParse, WebPAnimEncoderAssemble, and cmsOpenProfileFromFile with ImagingSectionEnter/Leave.
  3. Findings 29–30 (decoder/encoder GC): add tp_traverse, tp_clear, Py_TPFLAGS_HAVE_GC to both ImagingDecoderType and ImagingEncoderType; change allocators to PyObject_GC_New.
  4. Findings 32–35 (stub parity): add = ... defaults to optional params; add Arrow protocol dunder declarations; expand _Outline, _avif, and _webp stubs.
  5. Findings 40–41 (complexity): extract blend_scanline() from font_render_impl and set_tiff_scalar/array_field() from PyImaging_LibTiffEncoderNew.
  6. Finding 19 (path_compact realloc): capture the realloc return value and handle failure explicitly rather than discarding it.
  7. Finding 38 (PyModule_AddType): replace the _imagingcms.c two-step type registration.
  8. Finding 39 (Py_IsNone): normalize 7 comparison sites across 5 files.

Longer-term (POLICY)

  1. Findings 46–47: add explicit Py_mod_multiple_interpreters slot; standardize all tp_dealloc to Py_TYPE(self)->tp_free(self).
  2. Finding 49 (heap-type migration): convert all 16 PyTypeObject static declarations to PyType_FromSpec; this is the gating prerequisite for both free-threading (see ft_review.md) and stable ABI compliance; estimated ~3–5 days.
  3. Findings 50–53 (stable ABI blockers): after heap-type migration, address _PyBytes_Resize, tp_name access, fast macros, and PyUnicode_READ_CHAR.
  4. Finding 54 (pythoncapi_compat.h): include from libImaging/Imaging.h transitively.
  5. Finding 55: evaluate abi3 wheel adoption after heap-type migration completes.

Before sharing this report externally: a ~50-finding report dropped on maintainers who didn't request it is a significant homework assignment. Reach out first, ask what shape they want the findings in, and prioritize the FIX-class items. Security findings (buffer overflow patterns, heap corruption in Finding 4/10) warrant responsible disclosure rather than public issues.

C Extension Analysis Report — propcache._helpers_c

Date: 2026-06-25
Scope: Entire project (C:\Users\שזדס\repos\propcache)
Agents run: type-slot-checker · gil-discipline-checker · resource-lifecycle-checker · module-state-checker · stable-abi-checker · version-compat-scanner · parity-checker · c-complexity-analyzer · git-history-analyzer · cppcheck (external)


Extension Profile

Field Value
Module _helpers_c
Source files 1 C file (src/propcache/_helpers_c.c, 12,530 lines)
Cython source src/propcache/_helpers_c.pyx (104 lines)
Cython version 3.2.6 (generated); 3.2.5 (pinned in requirements/cython.txt)
Init style Single-phase (PyInit__helpers_c)
Python targets ≥ 3.10 (classifiers include 3.14)
Limited API No
Code generation Cython
Types defined 2 (under_cached_property, cached_property)
Python fallback src/propcache/_helpers_py.py

External tools: cppcheck (3 findings, all in Cython-generated runtime boilerplate — see §External Tool Notes); clang-tidy skipped (no compile_commands.json).

Note on Cython-mode triage: The agents skipped refcount-auditor, error-path-analyzer, null-safety-scanner, and pyerr-clear-auditor by default (>95% false-positive rate on generator-emitted patterns). All four were manually verified to produce zero FIX-class loss for this extension.


Executive Summary

propcache is a very small Cython extension (2 types, 104 lines of .pyx) that implements property-caching descriptors for Python classes. Its core logic is concentrated in two nearly identical __get__ methods. The extension has three overlapping critical issues that collectively compromise its safety claim on the free-threaded Python build: a use-after-free in both __get__ methods under free-threading (crash-capable, non-deterministic), an incomplete refcount fix on the error path of PyDict_SetItem, and a false freethreading_compatible=True declaration that was left in place after the free-threading CI was removed. An existing FREE_THREADING_ANALYSIS.md in the repository documents the race conditions and recommends @cython.critical_section as the fix, but neither the fix nor the CI has been restored. Beyond the memory-safety cluster, the Cython implementation has four FIX-level API parity gaps vs. the pure-Python fallback (_helpers_py.py / functools.cached_property) that silently break framework introspection, annotation resolution, and Mapping-typed caches. All other dimensions (type slots, resource lifecycle, ABI compliance, version compatibility, complexity) are clean.


Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 1 0 Py_DECREF(val) unreachable on PyDict_SetItem error path
Error Handling G 0 0 (skipped — Cython-mode)
NULL Safety G 0 0 (skipped — Cython-mode)
GIL Discipline Y 2 1 UAF in both __get__ under free-threading (declared Py_MOD_GIL_NOT_USED)
Resource Lifecycle G 0 0 No non-PyObject resources; partial-init safe
Module State G 0 2 115+ static globals; missing Py_mod_multiple_interpreters
Type Slots G 0 0 All slots correct; GC flags consistent
PyErr_Clear Safety G 0 0 (skipped — Cython-mode)
ABI Compliance G 0 1 Not claimed; private APIs all guarded by Cython
Version Compat G 0 2 PyDict_GetItem soft-deprecated in 3.13; Cython pin mismatch
C/Python Parity R 4 4 .name/.attrname inaccessible; __module__ not forwarded; _cache dict requirement
Complexity G 0 0 Max cyclomatic complexity 4; max 15 LOC per method

G = no FIX findings · Y = 1–3 FIX findings · R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 7

# Finding File:Line Agents
1 Use-after-free in under_cached_property.__get__ under free-threading. PyDict_GetItem returns a borrowed reference. A concurrent thread can delete/replace the entry from inst._cache between the lookup and the return <object>val cast, freeing the object; Thread A then does Py_INCREF on freed memory → heap corruption crash. _helpers_c.pyx:46–51 gil-discipline, git-history
2 Use-after-free in cached_property.__get__ under free-threading. Structurally identical to Finding 1 but on inst.__dict__ (which is even more exposed to concurrent mutation via vars(inst) or inst.__dict__.pop). Both the cache-hit borrowed-ref window and the miss-path concurrent-eviction race apply. _helpers_c.pyx:95–101 gil-discipline, git-history
3 freethreading_compatible=True + Py_MOD_GIL_NOT_USED persists with no FT CI. Commit 6093ed1 removed the free-threaded CI build while leaving the module's GIL-not-used declaration in place. CPython disables the GIL for this module on 3.13t/3.14t builds, exposing Findings 1–2 at runtime with no automated test coverage. The existing FREE_THREADING_ANALYSIS.md already identified the races; neither fix nor CI was restored. _helpers_c.pyx:1; pyproject.toml git-history, gil-discipline
4 under_cached_property.name inaccessible in Cython. cdef object name has no readonly qualifier, so prop.name raises AttributeError on the C extension. The Python fallback exposes self.name as a public attribute. Code that reads .name to discover or clear the cache key (serializers, debuggers, cache-clearing utilities) works under PROPCACHE_NO_EXTENSIONS=1 and silently breaks under the C extension. Fix: cdef readonly object name. _helpers_c.pyx:32 parity
5 cached_property.attrname missing; name also inaccessible. functools.cached_property.attrname is documented public API (set in __set_name__, used in error messages). The Cython cached_property exposes neither attrname nor name (both are private cdef). Frameworks such as Pydantic and SQLAlchemy that introspect .attrname after class creation will fail only under the C extension. Fix: expose readonly object attrname (or rename name → attrname and add readonly). _helpers_c.pyx:69 parity
6 cached_property.__module__ not forwarded from func.__module__. The Cython cached_property.__init__ never copies func.__module__, so prop.__module__ returns "propcache._helpers_c" for every instance. functools.cached_property copies it in __init__. This breaks PEP 563 annotation resolution, pytest fixture discovery, and any framework that uses __module__ to resolve forward references. _helpers_c.pyx:71–73 parity
7 _cache must be exactly dict; breaks the stated Mapping contract. cdef dict cache = inst._cache performs a PyDict_Check at runtime. If inst._cache is any non-dict Mapping (e.g. TypedDict, MutableMapping subclass), Cython raises TypeError. The typed annotation for the _CacheImpl protocol explicitly uses Mapping[str, Any], and the Python fallback honours that contract. _helpers_c.pyx:45 parity

Should Consider (CONSIDER) — 10

# Finding File:Line Agents
8 _helpers_py.py check-then-act double-compute under free-threading. try: return inst._cache[self.name] → except KeyError: val = self.wrapped(inst); inst._cache[self.name] = val is a TOCTOU. Two threads can both see KeyError and both call the wrapped function. Unlike Findings 1–2 this is not a UAF (Python's dict internally manages refcounts); the worst outcome is an extra function call. Still inconsistent with the freethreading_compatible claim if that claim is restored. _helpers_py.py:56–61 git-history
9 Cosmetic inconsistency between twins: if val == NULL (pyx:47) vs. if val is NULL (pyx:97). Both compile identically; no behavioral difference. Introduced by commit 4c645c7 and never normalized. _helpers_c.pyx:47,97 git-history
10 under_cached_property.__doc__ semantics differ. Cython: @property returning self.wrapped.__doc__ (live, read-only). Python: stored once in __init__ (snapshot, writable). Tools that stamp a descriptor's __doc__ after construction (sphinx-autodoc, some test fixtures) raise AttributeError under Cython but succeed under Python. _helpers_c.pyx:38–40 vs _helpers_py.py:40 parity
11 cached_property.__doc__ same divergence. Same read-only-vs-writable, dynamic-vs-snapshot inconsistency as Finding 10, for cached_property.func.__doc__. _helpers_c.pyx:75–77 parity
12 cached_property.__get__ gives AttributeError on slotted classes; functools gives descriptive TypeError. cdef dict cache = inst.__dict__ propagates a raw AttributeError when the instance has __slots__. functools.cached_property converts this to TypeError("No '__dict__' attribute on {type(instance).__name__!r} instance…"). Code that catches TypeError from functools.cached_property misuse will fail silently under the C extension. _helpers_c.pyx:95 parity
13 under_cached_property[T] returns types.GenericAlias (Cython) vs. typing._GenericAlias (Python). Cython uses __class_getitem__ = classmethod(GenericAlias) (pyx:56); the Python under_cached_property(Generic[_T]) inherits Generic.__class_getitem__. isinstance(under_cached_property[str], types.GenericAlias) flips between implementations; typing.get_args() / typing.get_origin() may behave differently. _helpers_c.pyx:56 vs _helpers_py.py parity
14 Static global module state (115+ PyObject*) prevents subinterpreter isolation. In the standard non-Limited-API build (CYTHON_USE_MODULE_STATE=0), __pyx_mstate_global_static and __pyx_m are process-wide statics. A second subinterpreter shares the same type objects and string table; module reload reuses stale state. This is the normal Cython 3 pattern and is blocked only if subinterpreter support becomes a goal. _helpers_c.c:1404–1406 module-state
15 Py_mod_multiple_interpreters declaration missing in standard Python 3.12+ build. The slot is only emitted when CYTHON_USE_MODULE_STATE=1 (Limited API path). In the standard build, Python 3.12+ assumes the module supports subinterpreters and loads it into them without warning. Fix (if subinterpreters matter): build with Limited API, or patch the generated C to emit Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED unconditionally for Python 3.12+. _helpers_c.c:6676–6678 module-state
16 Cython pin mismatch. requirements/cython.txt pins cython==3.2.5 (exact), but the committed _helpers_c.c was generated with Cython 3.2.6 (line 1 of .c). CI regenerating the file will produce a different output, causing spurious diffs. requirements/cython.txt:1 version-compat
17 PyDict_GetItem soft-deprecated in Python 3.13. The .pyx source hand-imports PyDict_GetItem via from cpython.dict cimport (bypassing Cython's own CYTHON_AVOID_THREAD_UNSAFE_BORROWED_REFS wrapper). PyDict_GetItemRef (stable, added 3.13) returns a strong reference and eliminates the borrowed-ref window. This is also the correct FT fix path for Findings 1–2 without requiring @cython.critical_section. For a 3.10 floor, use the Cython-generated __Pyx_PyDict_GetItemRef abstraction already present in-tree, or implement a PY_VERSION_HEX guard. _helpers_c.pyx:4,46,96 version-compat, stable-abi

Tensions

  • GIL discipline vs. parity (Findings 1–2 vs. 8): The Cython __get__ methods are crash-capable under free-threading; the Python fallback is not (only double-compute). After fixing Findings 1–2 with @cython.critical_section, the double-compute in Finding 8 becomes the remaining parity gap. The two can be fixed independently: Cython gets critical_section; Python gets dict.setdefault or PyDict_SetDefault.
  • freethreading_compatible=True vs. no FT CI (Findings 1–3): The existing FREE_THREADING_ANALYSIS.md has already diagnosed Findings 1–2 and recommended @cython.critical_section. The decision gap is not technical — it is whether to restore FT CI and fix, or to explicitly remove freethreading_compatible=True and defer. Leaving the current state (FT declaration, no CI, unfixed crash) is the worst of all three options.
  • Module state (Findings 14–15): The full per-module-state implementation already exists in the generated C behind CYTHON_USE_MODULE_STATE. Activating it requires building with Limited API. This conflicts with the free-threaded builds (Finding 3), since Py_LIMITED_API and Py_GIL_DISABLED are mutually exclusive in the same wheel. GIL-enabled and free-threaded wheels would continue to need separate build paths regardless.

Policy Decisions (POLICY) — 5

# Finding
18 cached_property omits per-property locking. functools.cached_property gained a per-property threading.RLock in Python 3.12 to prevent double-compute. The Cython version intentionally omits this — propcache historically prioritizes throughput over strict compute-once semantics. This divergence should be documented explicitly (docstring or README), especially now that a freethreading_compatible=True claim is in play.
19 Private API uses in Cython preamble are all correctly guarded. _PyThreadState_UncheckedGet, _PyCFunctionFast, _PyDict_GetItem_KnownHash, _PyDict_NewPresized, _PyObject_GetDictPtr — all inside #if CYTHON_USE_TYPE_SLOTS or !LIMITED_API guards. Cython already provides the public-API fallback paths. No action needed.
20 abi3 adoption is feasible with 3 build-system changes; no .pyx source changes required. The generated C already contains complete #if CYTHON_COMPILING_IN_LIMITED_API alternate paths for every construct. Three changes needed: (1) --3limited to cythonize CLI args in packaging/pep517_backend/_cython_configuration.py; (2) -DPy_LIMITED_API=0x030A0000 injected into CFLAGS; (3) py_limited_api=True on the Extension in setup.cfg. GIL-enabled builds (cp310–cp314) would collapse to one abi3 wheel per platform. Free-threaded builds (cp313t, cp314t) remain per-version regardless (Py_GIL_DISABLED ⊥ Py_LIMITED_API).
21 Full per-module-state path already implemented in generated C. CYTHON_USE_MODULE_STATE=1 activates multi-phase init, per-module state struct, m_traverse/m_clear, and Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED. This is the Limited API build path. No .pyx changes required; the migration path is a build-system switch, contingent on the Policy 20 abi3 decision.
22 Cython boilerplate races are upstream issues. __pyx_lineno/__pyx_clineno non-atomic globals (used by __Pyx_AddTraceback) and deprecated PyErr_Fetch/PyErr_Restore in __Pyx_Raise are emitted identically into every Cython 3.2.x module. Under concurrent exceptions these can produce wrong traceback line numbers (not a safety issue). Both will resolve when Cython ships a free-threading-safe traceback mechanism and the .pyx is regenerated. Not actionable from this repo.

Strengths

  • Type slots are correct throughout. Both tp_dealloc, tp_traverse, tp_clear, tp_new, and tp_init are implemented correctly. tp_new initializes all PyObject* members to Py_None with INCREF, making partial-init safe. The Py_DECREF(tp) after tp_free in dealloc is correctly gated by #if CYTHON_USE_TYPE_SPECS. The Py_TPFLAGS_HAVE_GC + tp_traverse + tp_clear triple is consistent on both types.
  • No non-PyObject resources. Zero malloc/free, PyMem_Malloc, file descriptors, or external handles in the extension-specific type code. Lifecycle is entirely Python object lifetime, which Cython manages correctly.
  • Twin symmetry has been consistently maintained. All 10 commits that touched _helpers_c.pyx applied changes symmetrically to both under_cached_property and cached_property. No fix was ever applied to one twin but not the other.
  • Error messages match between implementations. __set__ ("cached property is read-only"), __set_name__ type-name-mismatch, and cached_property.__get__ name-not-set error messages are identical across Cython and Python implementations.
  • GC flags consistent, no cycle-breaker omissions. Both types participate in GC, have tp_traverse visiting all PyObject* members (including the heap type itself for the CYTHON_USE_TYPE_SPECS path), and tp_clear uses idempotent None-swap semantics.
  • __init__ re-entrancy safe. Both __init__ methods use INCREF(new) → DECREF(old) → assign order, making repeated obj.__init__() calls safe with no resource leak.
  • No complexity issues. Maximum cyclomatic complexity across the entire .pyx source is 4 (cached_property.__get__). Maximum nesting depth is 2. The implementation size is exactly matched to the problem scope.

Code Removal Opportunities

The minimum Python version is 3.10. Based on the deprecated-APIs catalog:

  • PyModule_AddType (3.10+): replaces the PyType_FromSpec + Py_INCREF + PyModule_AddObject chain. The generated C uses this pattern in the module init path. Estimated: 4–8 lines saved per type (2 types). However, since the generated C is produced by Cython and not hand-written, this opportunity applies only if the project switches to a custom module init; Cython 3.2.x does not emit PyModule_AddType directly. Not actionable without source changes.
  • PyDict_GetItemRef (3.13+): replaces the PyDict_GetItem + borrowed-ref pattern in both __get__ methods. This is also the FT-safe fix for Findings 1–2. For a 3.10 floor, a version guard is needed; the Cython-generated __Pyx_PyDict_GetItemRef abstraction already present in-tree provides the portable equivalent with no version guard. Estimated: eliminates 3 lines of explicit refcount management (Py_DECREF(val), Py_INCREF/<object>val pattern) per __get__ (×2 twins). Actionable now via Cython's in-tree abstraction.

Recommended Action Plan

Immediate (FIX items)

  1. Fix Findings 1–2: UAF under free-threading. Three viable approaches, in preference order:

    • (Preferred) Replace PyDict_GetItem + Py_DECREF with cache.get(self.name, _sentinel) (let Cython emit __Pyx_PyDict_GetItemRef under CYTHON_AVOID_THREAD_UNSAFE_BORROWED_REFS). Apply to both twins. This also fixes Finding 2 on the error path.
    • (Also valid) Add @cython.critical_section to both __get__ methods as documented in FREE_THREADING_ANALYSIS.md. Locks the descriptor instance (not the cache dict); correct because self is a cdef class.
    • (Partial) Switch from PyDict_GetItem to PyDict_SetDefault / dict.setdefault for atomic check-then-set; eliminates the eviction race but still allows double-compute.
  2. Fix Finding 3: restore FT CI or remove FT declaration. Do one of:

    • Apply Fix 1 above → restore cpython-freethreading in [tool.cibuildwheel] in pyproject.toml → re-add FT stress test.
    • Or: change _helpers_c.pyx line 1 to freethreading_compatible=False and regenerate the .c file, until the above fix lands. The current state (FT declaration, no CI, crash-capable race) is incorrect in both directions.
  3. Fix Finding 2 error-path: Py_DECREF(val) unreachable on PyDict_SetItem failure. Wrap in try/finally:

    val = PyObject_CallOneArg(self.wrapped, inst)
    try:
        PyDict_SetItem(cache, self.name, val)
    finally:
        Py_DECREF(val)
    return <object>val

    Apply to both twins (_helpers_c.pyx:49–50 and 99–100). Note: if Fix 1 above replaces the manual C-API calls entirely, this fix is subsumed.

  4. Fix Finding 4: expose under_cached_property.name. Change cdef object name → cdef readonly object name in _helpers_c.pyx:32. Regenerate .c.

  5. Fix Finding 5: add cached_property.attrname. Change cdef object name → cdef readonly object attrname (or name) in _helpers_c.pyx:69, update all internal references to the renamed field, and update __set_name__ accordingly. Regenerate .c.

  6. Fix Finding 6: forward cached_property.__module__. Add a cdef object _module field and a __module__ property in _helpers_c.pyx, or use object.__setattr__(self, '__module__', func.__module__) in __init__.

  7. Fix Finding 7: relax _cache dict requirement. Change cdef dict cache = inst._cache → cdef object cache = inst._cache and use PyObject_GetItem/PyObject_SetItem instead of PyDict_GetItem/PyDict_SetItem. This restores the Mapping contract stated in the type annotations. Note: if Fix 1 above switches to cache.get(...), Cython's subscript operators already use the generic protocol — this fix is subsumed.

Short-term (CONSIDER items)

  1. (Finding 8) Add dict.setdefault idiom to _helpers_py.py:under_cached_property.__get__ to reduce double-compute probability under free-threading, for parity with a critical-section-guarded Cython fix.
  2. (Finding 9) Normalize if val == NULL / if val is NULL between twins (_helpers_c.pyx:47 and 97) for readability.
  3. (Findings 10–11) Decide: should __doc__ be writable? If yes, add a __set__ to the property or store it as an instance attribute in both implementations. If no, document the read-only behavior.
  4. (Finding 12) Add a try/except in cached_property.__get__ that catches AttributeError on inst.__dict__ access and re-raises as TypeError with a descriptive message, matching functools.cached_property.
  5. (Finding 13) Align under_cached_property.__class_getitem__: either override in the Python class or document the types.GenericAlias vs. typing._GenericAlias difference.
  6. (Findings 14–15) If subinterpreter support becomes a goal, build with Limited API (Policy 20) to activate the existing per-module-state path and the Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED declaration.
  7. (Finding 16) Advance Cython pin: update requirements/cython.txt to cython==3.2.6 and regenerate _helpers_c.c (or regenerate with the current 3.2.5 pin and commit the result).
  8. (Finding 17) When raising the Python floor to 3.13, replace from cpython.dict cimport PyDict_GetItem with PyDict_GetItemRef for a strong-reference, FT-safe lookup. For now, the Cython-internal __Pyx_PyDict_GetItemRef (activated by removing the hand-cimport) is the portable path.

Longer-term (POLICY items)

  1. (Policy 18) Document that cached_property intentionally does not guarantee single-compute under concurrent access, unlike functools.cached_property since Python 3.12. Add a note to the README or docstring.
  2. (Policy 20) Evaluate abi3 wheel consolidation for GIL-enabled builds. The source already supports it; three build-system changes in packaging/pep517_backend/_cython_configuration.py, setup.cfg, and pyproject.toml are sufficient.
  3. (Policy 22) Track Cython upstream for free-threading-safe traceback handling and PyErr_GetRaisedException adoption; regenerate when available.

External Tool Notes

cppcheck found 3 findings in the generated C, all in Cython runtime boilerplate (lines 10000+), not in the extension's own logic:

Line Checker Severity Assessment
10362 uselessAssignmentPtrArg warning Inside __Pyx_AllocateExtensionTypeSharedCythonModule; generated Cython ABI module helper. Not maintainer-actionable.
11134 shiftTooManyBitsSigned error Inside __Pyx_NewCyFunction; Py_INCREF macro on CPython 3.13+ uses immortal-object refcount bit manipulation. Generated noise.
11607 shiftTooManyBitsSigned error Same pattern; generated Cython function-object creation helper. Generated noise.

None of these are actionable from the .pyx source. They would clear automatically if cppcheck adds Cython/CPython-aware suppression rules or if CPython changes the Py_INCREF macro expansion.


Deduplication Notes

The following findings were independently flagged by multiple agents and have been merged:

  • UAF in __get__ (Findings 1–2): Confirmed by gil-discipline-checker (as CONSIDER), git-history-analyzer (as FIX), and cross-referenced in the existing FREE_THREADING_ANALYSIS.md. Elevated to FIX given crash-capable severity.
  • Py_DECREF error-path leak (Finding 2/FIX): Flagged by type-slot-checker (out-of-scope note), resource-lifecycle-checker (out-of-scope note), and git-history-analyzer (primary). Attributed to git-history-analyzer as the agent that performed the most thorough analysis.
  • freethreading_compatible=True + no CI (Finding 3): gil-discipline-checker identified the premature declaration (as POLICY); git-history-analyzer identified the CI removal (as FIX). Merged and elevated to FIX because the combination is actively unsafe.
  • PyDict_GetItemRef recommendation (Finding 17 / Policy 19): Both version-compat-scanner and stable-abi-checker recommended switching; attributed to version-compat as the primary agent.

Before sharing externally: This report is an internal artifact. A 22-finding report delivered unsolicited is a significant time burden for maintainers. Reach out first, describe the categories of findings you've identified, and ask what format is useful. Security-class findings (use-after-free under free-threading, Findings 1–3) warrant responsible disclosure if the library is widely deployed. The existing FREE_THREADING_ANALYSIS.md suggests the maintainers are already engaged on this issue — a targeted follow-up ("the UAF is still open after removing FT CI") is likely more useful than a full report delivery.

C Extension Analysis Report

Extension: google._upb._message (UPB Python Backend)

Scope: python/ (10 C files, 8,912 lines — UPB backend only)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, c-complexity-analyzer, git-history-analyzer, external-tools (cppcheck)

Analysis Date: 2026-06-13


Before sharing this report externally (filing issues, opening PRs, sending to a maintainer): reach out first, ask what shape they want it in, and do not auto-file or auto-PR. Security findings need responsible disclosure, not a public issue. This is an internal artifact; whether and how to share it with maintainers is a separate human-judgment step. See WORKING_WITH_MAINTAINERS.md for guidance.


Executive Summary

The protobuf UPB Python backend (google._upb._message) is actively maintained (50 commits, 13 authors in the past 90 days) with a solid free-threading foundation: the PyUpb_WeakMap uses a per-instance pthread_mutex_t and PyUnstable_TryIncRef, resource lifecycle is clean, and GIL discipline is correct throughout the analyzed code. However, the extension has three critical type-slot bugs — METH_NOARGS functions with only one C parameter, producing undefined behavior on every invocation — and three broad PyErr_Clear calls in hot paths that silently swallow MemoryError, KeyboardInterrupt, and SystemExit. Two confirmed memory-safety issues (an unchecked malloc and an unchecked realloc with leak) add crash risk under memory pressure, and a third class of crash risk exists where intermediate allocations are passed to PyTuple_SetItem without a NULL check. The extension uses single-phase init and does not support subinterpreters. The pyext C++ backend (10 .cc files, ~12,500 lines) was not analyzed — the scanner's tree-sitter grammar covers C only; the highest-churn file (pyext/message.cc, 12 commits in 90 days) requires a dedicated review.


Extension Profile

Field Value
Module google._upb._message
Backend UPB (pure-C, 10 files, 8,912 lines)
Init style Single-phase (PyModule_Create)
Python targets ≥3.9 (inferred from upb_Pre310_PyType_GetDeallocSlot guard)
Limited API No — uses PyUnstable_* free-threading APIs
Types defined Arena, WeakMap, descriptors, containers, message, map, repeated
Unanalyzed pyext/ C++ backend (10 .cc files, ~12,500 lines)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety G 0 2 Scanner HIGH findings were false positives; see Strengths
Error Handling Y 1 13 AddObject at protobuf.c:424 leaks type ref on failure; 283 medium-confidence scan patterns
NULL Safety R 3 1 Unchecked malloc:808; realloc leak :1597; intermediate alloc → PyTuple_SetItem/strcmp
GIL Discipline G 0 0 callback_without_gil scanner hits are false positives (simple accessors); see Strengths
Resource Lifecycle G 0 0 No findings
Module State G 0 2 Single-phase init; 27 static globals incompatible with subinterpreters
Type Slots R 3 0 METH_NOARGS 1-param UB: message.c:1466, repeated.c:561, :576
PyErr_Clear Safety R 3 6 Broad clear in hot path: message.c:1634, :2098, :2103
ABI Compliance Y 1 3 Type ref leaked on init failure (protobuf.c:424); ob_type direct access; PyModule_AddObject vs AddObjectRef
Version Compat G 0 1 Dead code upb_Pre310_PyType_GetDeallocSlot (message.c:66)
C/Python Parity N/A — — UPB backend is the sole binding layer; no pure-Python fallback
Complexity G 0 4 4 functions CC≥15; FindInitializationErrors highest

G = No FIX findings · Y = 1–3 FIX findings · R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 10

# Finding File:Line Agents
1 PyUpb_Message_Clear takes 1 C parameter but is registered as METH_NOARGS, which requires PyObject*(*)(PyObject*, PyObject*) — calling convention mismatch is undefined behavior on every invocation message.c:1466 type-slot-checker
2 PyUpb_RepeatedContainer_Reverse takes 1 C parameter, registered METH_NOARGS — same UB as Finding 1 repeated.c:561 type-slot-checker
3 PyUpb_RepeatedContainer_Clear takes 1 C parameter, registered METH_NOARGS — same UB as Finding 1 repeated.c:576 type-slot-checker
4 malloc result for buf2 in PyUpb_ToString not checked for NULL; subsequent use crashes on OOM message.c:808 null-safety-scanner, cppcheck
5 realloc result stored in a temporary; on failure the original buf is leaked and the error silently ignored — cppcheck confirmed memleakOnRealloc message.c:1597 null-safety-scanner, cppcheck
6 Three allocation results passed to downstream functions without NULL checks: (a) PyUnicode_FromString result → PyTuple_SetItem in ByNameMap_Items (crash on OOM); (b) PyLong_FromLong result → PyTuple_SetItem in ByNumberMap_Items (crash on OOM); (c) PyObject_GetAttrString result (type_name_obj) → strcmp wrapper in PyUpb_IsNumpyNdarray (crash on OOM or missing attr) descriptor_containers.c:528, :774; convert.c:216 null-safety-scanner, manual analysis
7 AddObject wrapper leaks the type ref when PyModule_AddObject fails: the ternary type && PyModule_AddObject(...) == 0 ? type : NULL returns NULL without freeing type (PyModule_AddObject does not steal on failure in Python ≥3.9) protobuf.c:424 stable-abi-checker
8 Broad PyErr_Clear() inside loop body — swallows any pending exception including MemoryError, KeyboardInterrupt, and SystemExit message.c:1634 pyerr-clear-auditor
9 Broad PyErr_Clear() in PyUpb_MessageMeta_GetAttr cache-miss path — same swallow risk; this is a hot code path executed on every attribute access message.c:2098 pyerr-clear-auditor
10 Broad PyErr_Clear() in PyUpb_MessageMeta_GetAttr main path — same swallow risk message.c:2103 pyerr-clear-auditor

Fix 1–3 (type slot): Add PyObject* Py_UNUSED(args) as second parameter to PyUpb_Message_Clear, PyUpb_RepeatedContainer_Reverse, and PyUpb_RepeatedContainer_Clear. These are calling-convention violations; even though they work on common x86-64 ABIs today, they are UB and can break with optimization flags, LTO, or non-standard ABIs.

Fix 4–6 (NULL safety): For Fix 4, add if (!buf2) { PyErr_NoMemory(); return NULL; }. For Fix 5, store realloc result in a temp, check for NULL before overwriting buf, free original on failure. For Fix 6: check each allocation result before passing downstream; goto error on NULL.

Fix 7 (module init): Rewrite AddObject as:

PyTypeObject* AddObject(PyObject* m, const char* name, PyType_Spec* spec) {
  PyObject* type = PyType_FromSpec(spec);
  if (!type) return NULL;
  int rc = PyModule_AddObject(m, name, type);
  if (rc < 0) { Py_DECREF(type); return NULL; }
  return (PyTypeObject*)type;
}

Fix 8–10 (PyErr_Clear): Replace bare PyErr_Clear() with if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear(); (or the specific expected exception type). Bare clears in hot paths can silently drop MemoryError and SystemExit; in free-threaded builds this is especially dangerous because concurrent threads may be in memory pressure simultaneously.


Should Consider (CONSIDER) — 10

# Finding File:Line
11 PyModule_AddObject(m, "__internal_wktbases", state->wkt_bases) return value not checked — if it fails, state->wkt_bases is set but not pinned to m; the object has no other keeper and will be leaked when the module tears down (comment says "Reparent ownership to m" but the reparent silently failed) protobuf.c:138
12 upb_Pre310_PyType_GetDeallocSlot is dead code on Python ≥3.11 — the #if guard was for 3.10 and below; safe to remove now that 3.10 is EOL message.c:66
13 4 PyModule_AddObject call sites should use PyModule_AddObjectRef (Python 3.10+) to avoid the steal-before-check footgun; use #if PY_VERSION_HEX >= 0x030a0000 guards for 3.9 compatibility protobuf.c:138, :148, :152, :168
14 5 sites access ob_type directly (->ob_type) rather than via Py_TYPE() — Py_TYPE() is the portable accessor since Python 3.9 python/*.c (5 sites)
15 6 PyErr_Clear() calls are unguarded (MEDIUM from pyerr-clear-auditor) — document the specific exception being discarded or add PyErr_ExceptionMatches guard various
16 4 functions exceed cyclomatic complexity 15; FindInitializationErrors is the highest — consider extracting sub-steps message.c (4 functions)
17 cppcheck reported 12 missingReturn findings — functions lacking a return statement on at least one path; verify those paths are unreachable (via Py_UNREACHABLE() or assert(0)) or add explicit returns python/*.c (12 sites)
18 Single-phase init (PyModule_Create) does not support subinterpreters; PyState_FindModule calls are fragile in multi-interpreter scenarios protobuf.c
19 27 static PyObject* globals function as module-level caches that will be shared across subinterpreters — must move into module state struct if multi-phase init is ever adopted python/*.c (27 sites)
20 The pyext C++ backend (python/google/protobuf/pyext/, 10 .cc files, ~12,500 lines) was not analyzed — the scanner's tree-sitter grammar parses C only; 0 functions were extracted from C++ files. pyext/message.cc has the highest churn in the repo (12 commits, 6 authors in 90 days) pyext/

Tensions

  • PyErr_Clear swallow vs. intentional suppression (Findings 8–10, 15): The three hot-path clears (8–10) most likely intend to suppress AttributeError on failed attribute lookups, which is a normal flow in __getattr__. If confirmed, the one-line fix is if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear();. The six MEDIUM unguarded clears (Finding 15) are lower confidence and should be reviewed individually.

  • Module state migration vs. simplicity (Findings 18–19 vs. POLICY 21): Single-phase init is substantially simpler. The extension's 27 static globals would all need to move into a PyModuleDef.m_size-sized state struct. This is only worthwhile if subinterpreter support is an explicit goal; if not, Findings 18–19 can be deferred indefinitely.

  • Free-threading vs. broad PyErr_Clear (Findings 8–10): In free-threaded builds, swallowing MemoryError in a hot path is especially dangerous because concurrent threads may simultaneously be in memory pressure. The PyErr_Clear findings are categorically more severe in the python3t context than in the GIL build.


Policy Decisions (POLICY) — 3

# Finding
21 Multi-phase init migration: Moving from single-phase to multi-phase init requires restructuring all 27 static module-level globals (Finding 19) into a per-module state struct. This is a substantial refactor with no end-user-visible effect unless subinterpreter support is targeted; schedule as a dedicated milestone.
22 Py_LIMITED_API is not feasible: The extension uses PyUnstable_EnableTryIncRef, PyUnstable_TryIncRef, and PyUnstable_Object_IsUniquelyReferenced (all prefixed PyUnstable_*) which are explicitly excluded from the stable ABI. Adopting Py_LIMITED_API would require dropping free-threading support or waiting for these APIs to stabilize. Reassess after CPython PEP 703 settles.
23 Dedicated C++ analysis for pyext backend: pyext/ holds ~12,500 lines of C++ (the "fast" backend for users who compile from source). It is not reachable by the current scanner toolkit. A separate review using clang-tidy with compile_commands.json or a tree-sitter C++ grammar is needed — prioritize pyext/message.cc which is the most actively modified file.

Strengths

  • WeakMap free-threading implementation: PyUpb_WeakMap_Add calls PyUnstable_EnableTryIncRef at protobuf.c:185 before any PyUnstable_TryIncRef calls in _Get / _Next, satisfying CPython's documented requirement. The iteration lock is held for the entire WeakMap_Next loop body, preventing torn reads.

  • Resource lifecycle: The resource-lifecycle scanner reported zero findings. Arena and WeakMap objects are correctly paired with their destructors; no identified handle or file-descriptor leaks.

  • GIL discipline: The scanner's callback_without_gil findings at message.c:210 (PyUpb_Message_GetFieldDef) and message.c:225 (PyUpb_Message_GetMsg) are false positives — both are simple struct-field accessor functions that contain no Python C API calls and are always invoked from Python code with the GIL held. No confirmed GIL violations in 479 analyzed functions.

  • Refcount safety: The scanner's two HIGH borrowed_ref_across_call findings are false positives: (a) convert.c:216 — type_name_obj is a new reference from PyObject_GetAttrString, not borrowed; Py_DECREF at line 224 is correct cleanup; (b) message.c:1946 — wkt_base is borrowed from a dict that outlives both Py_BuildValue calls, and wkt_base is never used after they return. Zero confirmed refcount bugs after manual triage.

  • Type slot sentinels: The scanner flagged 6 PyType_Slot arrays as potentially missing {0, NULL} terminators — all 6 are false positives. Manual reads of PyUpb_GenericSequence_Slots (descriptor_containers.c:362–373) and PyUpb_Arena_Slots (protobuf.c:401–404) confirm the sentinels are present and correct.

  • Stolen-ref false positives: The scanner flagged descriptor_containers.c:536–537 and :781–782 as stolen_ref_not_nulled. These are false positives: the goto error at lines 527/773 is taken before PyTuple_SetItem is ever called, so the variables are never in a stolen state when DECREF'd. The scanner cannot model goto-based control flow.

  • Active maintenance: 50 commits from 13 authors in the past 90 days. No single-author dominance; bus-factor risk is low. The free-threading implementation shows deliberate design effort.


Code Removal Opportunities

From deprecated_apis.json — checked against the extension's minimum Python target (≥3.9):

API / Pattern Available Since Benefit Sites
PyModule_AddObjectRef Python 3.10 Eliminates the steal-before-check footgun; drops need for manual Py_DECREF on error 4 sites in protobuf.c (requires #if PY_VERSION_HEX >= 0x030a0000 guard for 3.9)
Py_TYPE(op) macro Python 3.9 Replaces 5 ->ob_type direct field accesses; improves portability 5 sites in python/*.c
PyModule_AddType Python 3.10 Replaces the PyType_FromSpec + Py_INCREF + PyModule_AddObject chain; ~4 lines saved per type Check protobuf.c init function for applicable chains

Not yet applicable (future Python targets):

  • PyImport_ImportModuleAttrString (3.14+): replaces PyImport_ImportModule + PyObject_GetAttrString + Py_DECREF(module) chains — applicable once Python 3.14 is the minimum.

Scanner Notes

Scanner Files Functions Findings Notes
scan_shared_state.py 41 479 0 No shared-state concerns detected
scan_refcounts.py 41 479 41 (8 HIGH, 33 MED) All HIGH findings manually verified as false positives; stolen_ref at dc.c:536/781 and borrowed_ref at convert.c:216, message.c:1946
scan_error_paths.py 41 479 283 (all MED) 85 missing_null_check, 60 exception_clobbering, 138 return_without_exception — medium confidence heuristics; manual triage needed to identify actionable subset
scan_null_checks.py 41 479 2 HIGH Findings 4–5; cross-confirmed by cppcheck
scan_gil_usage.py 41 479 2 MED False positives — callback_without_gil at message.c:210/225 are struct-field accessors
scan_resource_lifecycle.py 41 479 0 Clean
scan_module_state.py 41 479 34 1 HIGH single_phase_init; 6 MED/HIGH module_add_object_misuse (Finding 11); 27 LOW static_mutable_state (Finding 19)
scan_type_slots.py 41 479 11 HIGH 3 method_signature_mismatch = REAL (Findings 1–3); 6 missing_sentinel = FALSE POSITIVES
scan_pyerr_clear.py 41 479 9 3 HIGH broad_clear = Findings 8–10; 6 MED unguarded = Finding 15
measure_c_complexity.py 41 479 — 4 functions CC≥15 (Finding 16)
analyze_history.py 41 — — 50 commits/90 days; pyext/message.cc highest churn (12 commits, 6 authors)
run_external_tools.py (cppcheck) 41 — 13 12 missingReturn (Finding 17); 1 memleakOnRealloc = Finding 5; 1 unknownMacro FP
scan_type_slots.py — pyext 17 CC 0 extracted — tree-sitter C grammar cannot parse C++; pyext entirely unanalyzed (Finding 20)

Recommended Action Plan

Immediate — Fix UB and Memory Safety (Findings 1–7)

  1. Fix 1–3 — Add PyObject* Py_UNUSED(args) as second parameter to PyUpb_Message_Clear, PyUpb_RepeatedContainer_Reverse, and PyUpb_RepeatedContainer_Clear. These are calling-convention violations that produce UB on every invocation.
  2. Fix 4 — Add if (!buf2) { PyErr_NoMemory(); return NULL; } after the malloc at message.c:808.
  3. Fix 5 — Rewrite the realloc at message.c:1597: store into a temp, check for NULL (free original + return error), then assign to buf.
  4. Fix 6 — In ByNameMap_Items and ByNumberMap_Items, check PyUnicode_FromString / PyLong_FromLong before passing to PyTuple_SetItem. In PyUpb_IsNumpyNdarray, check type_name_obj for NULL before the strcmp wrapper call.
  5. Fix 7 — Rewrite AddObject (protobuf.c:424) to Py_DECREF(type); return NULL; on PyModule_AddObject failure.

Short-term — PyErr_Clear Hardening (Findings 8–10, 15)

  1. Fix 8–10 — In message.c:1634, :2098, :2103, replace PyErr_Clear() with if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear(); (or whatever the expected exception is). Confirm that MemoryError cannot legitimately be discarded at these sites.
  2. Consider 15 — For each of the 6 MEDIUM unguarded clears: document the expected exception in a comment or add an ExceptionMatches guard.

Short-term — API Modernization (Findings 11–14)

  1. Consider 11 — Check return value of PyModule_AddObject at protobuf.c:138 and propagate failure.
  2. Consider 12 — Remove upb_Pre310_PyType_GetDeallocSlot and its #if guard; Python ≤3.10 is EOL.
  3. Consider 13 — Replace PyModule_AddObject with PyModule_AddObjectRef at 4 sites using a version guard.
  4. Consider 14 — Replace 5 ->ob_type accesses with Py_TYPE().

Medium-term — Unanalyzed Backend (Finding 20)

  1. Consider 20 — Run a dedicated C++ static analysis pass on pyext/ — prioritize pyext/message.cc. Use clang-tidy with compile_commands.json or a tree-sitter C++ grammar. This backend is the code path for all users who install from source and is currently a complete blind spot.

Longer-term — Architecture (Findings 16–19, POLICY 21–23)

  1. Consider 16–17 — Refactor FindInitializationErrors (CC≥15) and audit the 12 missingReturn paths for reachability.
  2. POLICY 21 — Multi-phase init migration: schedule as a standalone milestone if subinterpreter support is needed.
  3. POLICY 22 — Monitor CPython PEP 703 stabilization; reassess Py_LIMITED_API feasibility once PyUnstable_* APIs graduate to stable.
  4. POLICY 23 — Plan and execute the dedicated pyext C++ review session.

C Extension Analysis Report — pyarrow

Date: 2026-07-04 Repo: apache/arrow @ local checkout C:\Users\שזדס\repos\arrow

Extension: pyarrow

Scope: python/pyarrow/src/arrow/python/.cc,.h (24 hand-written C++ glue files, ~925 functions) + python/pyarrow/.pyx,.pxd,*.pxi (~24 separately-compiled Cython modules, ~1023 def/cdef/cpdef bodies)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer

Method note: the toolkit's tree-sitter-based scanners could not parse most of this codebase — the C grammar cannot descend into C++ namespace arrow::py { ... } blocks (extracted ~5 of ~925 glue functions), and no tree-sitter/tree-sitter-c wheel was installable for the local Python/mingw toolchain, so Cython-side scans fell back to manual triage too. Every finding below comes from agents' manual Grep/Read triage, in two cases (type-slot-checker) cross-checked by compiling and running throwaway Cython reproductions to empirically confirm claims (hash-nulling on __eq__-without-__hash__, and an uncollectable GC cycle) rather than arguing from documentation alone.

A companion free-threading (PEP 703) review already exists at reports/ft_review.md (26 findings). This report does not re-litigate thread-safety/race findings from that review; where a finding here touches the same code for a different reason, it is cross-referenced rather than duplicated.

Executive Summary

pyarrow's hand-written C++ glue is unusually disciplined for a project this size — RAII lock/GIL patterns are followed almost without exception, and the RETURN_IF_PYERROR() idiom is the dominant, well-applied convention for bridging CPython exceptions into Arrow Status. The bugs found are real but concentrated in the deviations from that house style, not systemic collapse. The two most consequential findings are permanent, uncollectable memory leaks in FlightServerBase and KmsClient — every instance ever constructed leaks forever, confirmed by compiling a live GC reproduction — and an ExtensionType unhashability regression (also empirically confirmed), both from the type-slot-checker. A UDF-registration refcount imbalance was independently found by two agents and then had its severity corrected downward by the git-history-analyzer, which traced the actual shared_ptr lifetime through cpp/src/arrow/compute/function.cc and found the defect fires once at registry teardown, not per call — a good example of this review's cross-agent verification working as intended. Version compatibility, stable-ABI posture, and API parity are all clean-to-POLICY: no deprecated-API debt beyond what's already tracked, and abi3 adoption is honestly infeasible today because of NumPy's own non-limited C-API, not anything pyarrow could fix unilaterally.

Extension Profile

  • Module: pyarrow (24 hand-written C++ files + ~24 compiled Cython extension modules)
  • Init style: single-phase (all modules; no Py_mod_exec/Py_mod_multiple_interpreters found anywhere)
  • Python targets: >=3.10 (declared), tested through 3.13 in CI
  • Limited API: no (not claimed; assessed as infeasible today — see Finding 47)
  • Types defined: ~24 Cython extension modules' worth of cdef class types (hundreds), plus 1 hand-written static PyTypeObject (MonthDayNanoTupleType)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety R 7 5 common.cc:166 leaks one str per formatted traceback line
Error Handling R 5 4 numpy_init.cc sets the "numpy imported" flag before import succeeds
NULL Safety Y 3 2 Unchecked PyArray_Descr* family spanning numpy_internal.h/numpy_convert.cc/inference.cc
GIL Discipline G 0 2 to_reader() methods skip with nogil: where every sibling releases it
Resource Lifecycle Y 2 2 io.pxi download() leaks the destination file handle on seek/malloc failure
Module State G 0 4 Single-phase init blocks subinterpreter support across all ~24 modules
Type Slots Y 3 1 FlightServerBase/KmsClient leak every instance permanently (empirically verified)
PyErr_Clear Safety Y 3 3 PyObjectToPyInt can silently substitute a different, wrong integer value
ABI Compliance G 0 0 Infeasible today — NumPy + datetime C-APIs have no limited-API path (1 POLICY)
Version Compat G 0 2 Cython pinned >=3.1 with no upper bound (1 POLICY)
C/Python Parity G 0 0 .pyi stub is an intentional empty placeholder — no parity surface to violate
Complexity G - 6 array()/record_batch() duplicate ~40 lines of PyCapsule-protocol handling

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings

Findings by Priority

Must Fix (FIX) — 19

# Finding File:Line Agents
1 FlightServerBase holds an uncollectable Python↔C++ reference cycle — the C++ PyFlightServer stores a strong ref back to self (Py_INCREF in flight.cc:83-87), invisible to Cython's auto-generated tp_traverse (which only sees the finalizer attribute). Confirmed with a live compiled reproduction: gc.collect() does not free the object. The existing _FlightServerFinalizer/weakref.finalize machinery (added for a different reason — interpreter-shutdown segfaults, ARROW-16597) does not break this cycle. Every FlightServerBase instance ever constructed leaks forever, along with everything reachable from it. python/pyarrow/_flight.pyx:2934-3031, src/arrow/python/flight.cc:83-87, flight.h:175 type-slot-checker
2 KmsClient has the identical uncollectable self-cycle, and is worse off: it has no object-typed attribute at all, so Cython never even marks it GC-tracked. Confirmed via the same live-reproduction methodology. Every custom KMS-integration instance leaks permanently. python/pyarrow/_parquet_encryption.pyx:330-343, src/arrow/python/parquet_encryption.cc:28, parquet_encryption.h:72-87 type-slot-checker
3 ExtensionType (and every subclass, including UnknownExtensionType and all user-defined pa.ExtensionType subclasses) silently becomes unhashable: it overrides __eq__ without redefining __hash__, and Cython replicates CPython's type.__new__ hash-nulling behavior for cdef class in this situation — confirmed by compiling a minimal reproduction (hash() raises TypeError: unhashable type). DataType and all non-extension subclasses remain correctly hashable, so this is a targeted regression on the primary extension mechanism, not a codebase-wide pattern. python/pyarrow/types.pxi:355-368, 1874-1882, 2138 type-slot-checker
4 UDF registration leaks a reference imbalance: RegisterScalarFunction/RegisterVectorFunction/RegisterTabularFunction each construct two independent OwnedRefNoGIL wrappers over the same borrowed PyObject* function (one at the call site, uninc­remented; one inside RegisterUdf, with one Py_INCREF) — net −1 over the combined lifetime. RegisterScalarAggregateFunction/RegisterHashAggregateFunction get this right (one incref, one wrapper). Severity revised: git-history-analyzer traced the actual shared_ptr lifetime through cpp/src/arrow/compute/function.cc and found both owners share the same Kernel/registry-entry lifetime, so the defect fires once at kernel/registry teardown — not per call as first suspected. It is live specifically when a custom FunctionRegistry (a supported, documented registry= kwarg) is destroyed while the interpreter is still running; the default global registry is process-lifetime and its destructor is Py_IsInitialized()-guarded, which likely masks the bug there. Traced to a single origin commit, 1f2479908 (2024-08-07, "Sanitize Python reference handling in UDF implementation"), which fixed the aggregate family correctly and missed the scalar/vector/tabular family — a textbook twin-fix asymmetry. python/pyarrow/src/arrow/python/udf.cc:452-516 (vs. correct 518-635) refcount-auditor, gil-discipline-checker (independent re-spot), git-history-analyzer (origin + severity correction)
5 numpy_imported is set true before import_numpy() actually succeeds; a failed import leaves has_numpy() lying while PyArray_API is NULL, so any later entry into ~14 has_numpy() && PyArray_* gated call sites (helpers.cc:88, inference.cc:463/467, iterators.h:48/110, python_to_arrow.cc:206/408/487/702/844, numpy_internal.h:159/167/175) NULL-derefs the API table. git-history-analyzer confirms this was wrong from inception in the single commit (9ab9532a2, 2024-09-02) that added the file, and has never been revisited or tested against an import-failure path. python/pyarrow/src/arrow/python/numpy_init.cc:25-32 error-path-analyzer, git-history-analyzer
6 PyListConverter::AppendSequence/AppendIterable silently swallow real exceptions: AppendSequence uses an unchecked PySequence_Size (-1 + pending exception flows into Append(true,-1)/Extend(value,-1), then returns Status::OK()); AppendIterable's PyIter_Next loop has no RETURN_IF_PYERROR() after exhaustion, so a mid-iteration exception (e.g. RuntimeError: set changed size during iteration) is silently dropped and truncated data is committed. Compare the correct idiom at iterators.h:190-194/ipc.cc:48-55, which this pair deviates from. python/pyarrow/src/arrow/python/python_to_arrow.cc:889-907 error-path-analyzer, refcount-auditor
7 PySequence_GetItem result (line) is never decref'd while formatting a Python exception's traceback into a Status message — leaks one str per traceback line on every formatted exception. python/pyarrow/src/arrow/python/common.cc:166 refcount-auditor
8 A local PyObject* ref from PyMemoryView_GetContiguous shadows the member OwnedRef ref in PyBytesView::ParseBinary — the new memoryview is never released (leak), and the member meant to keep the buffer alive is never actually set (the view's data pointer only survives because of the leak). Compare the correct sibling ParseUuid, which does ref.reset(...). python/pyarrow/src/arrow/python/common.h:409-414 refcount-auditor
9 StringToTzinfo's fixed-offset path leaks the timedelta object (offset) passed into constructing datetime.timezone — one leak per fixed-offset timezone conversion, and also on the failure path. python/pyarrow/src/arrow/python/datetime.cc:441-446 refcount-auditor
10 Converting a MonthDayNanoInterval array to pandas leaks 4 PyLong objects per non-null row: PyDict_SetItemString does not steal, and the four PyLong_FromLong* results ("months"/"days"/"microseconds"/"nanoseconds") are never released. Contrast with the correct pattern at the same file's line 1696-1700. python/pyarrow/src/arrow/python/arrow_to_pandas.cc:1308-1312 refcount-auditor
11 ConvertToSequenceAndInferSize leaks the raw PyList_New() result (lst) plus every item already stored into it if a user iterator raises mid-loop — only the PyList_SetSlice failure path releases it. python/pyarrow/src/arrow/python/python_to_arrow.cc:1243-1253 refcount-auditor
12 PyExtensionType::ToString NULL-derefs Py_TYPE(instance.obj())->tp_name when GetInstance() fails (e.g. the user's __arrow_ext_deserialize__ raises) — GetInstance() returns nullptr with an exception set, and ToString doesn't check before dereferencing. python/pyarrow/src/arrow/python/extension_type.cc:76-84 error-path-analyzer
13 PyDecimal_ISNAN NULL-derefs PyObject_IsTrue(NULL) if obj.is_nan() raises (e.g. a decimal.Decimal subclass with a broken override) — reachable from PandasObjectIsNull on every from_pandas conversion. python/pyarrow/src/arrow/python/decimal.cc:219-224 error-path-analyzer
14 A family of PyArray_Descr*-producing calls goes unchecked while sibling call sites in the same files correctly guard the identical allocator with RETURN_IF_PYERROR(): GetSafeNumPyDtype's result feeds set_numpy_metadata/PyDataType_REFCHK with no NULL check (sinks at arrow_to_pandas.cc:315-316,527-528); NumPyScalarToArrowDataType's PyArray_DescrFromScalar result flows unchecked into NumPyDtypeToArrow (descr->type_num dereferenced with zero guard). python/pyarrow/src/arrow/python/numpy_internal.h:182-190, numpy_convert.cc:122-137 null-safety-scanner
15 UDF execution stores wrap_array(c_data)/wrap_scalar(c_data)'s return value directly into a call-argument tuple via PyTuple_SetItem with no NULL check, and the tuple is handed to the user's Python UDF (cb(function, ..., arg_tuple)) before the subsequent RETURN_NOT_OK(CheckPyError()) runs. If wrapping fails (OOM or an exception during Array/Scalar construction), a NULL element sits live inside a tuple actively used in a Python call — a crash risk on every UDF invocation path (scalar, hash-aggregate, plain exec), not just a rare setup path. Found independently by two agents from different angles (general error-path bridging vs. pointer-safety). python/pyarrow/src/arrow/python/udf.cc:196-197, 335-336, 412-413, 416-417 null-safety-scanner, error-path-analyzer
16 NativeFile.download()/_download_nothreads() open the destination file (open(stream_or_path, 'wb')) and then call self.seek(0) and malloc(buffer_size) before entering the try/finally that would run cleanup()/close the file. A seek() or allocation failure in that window leaks the opened file object, since the background writer thread (which owns the only cleanup() call) never starts. git-history-analyzer confirms this gap has survived three refactors unchanged since the threaded implementation was first written — a genuinely unexercised error path, not a false positive. python/pyarrow/io.pxi:642-739, 741-795 resource-lifecycle-checker, git-history-analyzer
17 internal::PyObjectToPyInt unconditionally clears any exception from PyNumber_Index(obj) (i.e. obj.__index__()) before falling back to the type's nb_int slot — the only case where clearing is correct is TypeError (no __index__ present). If __index__ is present and raises for a domain reason (ValueError, OverflowError, or a genuine MemoryError), this silently discards that specific error and falls back to __int__, which can succeed with a different, silently wrong integer value rather than failing cleanly — the most consequential instance of the "clear-and-fallback" anti-pattern found in this review. python/pyarrow/src/arrow/python/helpers.cc:171-192 pyerr-clear-auditor
18 PyBytesView::ParseString probes UTF-8 validity by attempting a decode and unconditionally clearing any resulting exception as "not valid UTF-8," including MemoryError from the temporary allocation — silently mislabeling OOM as an encoding failure and continuing as Status::OK(). python/pyarrow/src/arrow/python/common.h:369-387 pyerr-clear-auditor
19 StringToTzinfo unconditionally clears any exception from zoneinfo.ZoneInfo(tz) (not just the expected ZoneInfoNotFoundError/KeyError for an unrecognized zone) before falling back to pytz; if pytz is also absent, the user sees a generic "zoneinfo module or pytz package must be installed" message even when the real problem was a misspelled timezone name. git-history-analyzer traces this to a 6-week-old, deliberate refactor (9583eb787, 2026-05-22, "switch from pytz to zoneinfo by default") and frames the underlying fallback design as intentional/POLICY — the defect is specifically that the clear is exception-type-blind rather than checking for the expected lookup error first. python/pyarrow/src/arrow/python/datetime.cc:460-466 pyerr-clear-auditor, git-history-analyzer (context)

Should Consider (CONSIDER) — 27

# Finding File:Line
20 OwnedRef::operator=(OwnedRef&&) does obj_ = other.detach() without releasing the currently-held reference — assigning into a non-empty OwnedRef silently leaks. All current call sites assign into empty refs, so latent today, but this is an exported public header (OwnedRefNoGIL inherits the same bug). python/pyarrow/src/arrow/python/common.h:188-191
21 SafeCallIntoPython leaks the previous exception triple (PyErr_Fetch refs) when the wrapped call returns a Python-error Status while an exception was already pending on entry. python/pyarrow/src/arrow/python/common.h:152-165
22 NumPyDtypeUnifier has no ownership discipline at all: VisitDType's PyArray_DescrFromScalar result is never released; _NUMPY_UNIFY_PROMOTE_TO's PyArray_DescrFromType result is stored with no incref; Observe also stores borrowed descrs past the owning array's lifetime, risking a dangling read in Observe_DATETIME. python/pyarrow/src/arrow/python/inference.cc:88-95, 284-290, 464
23 PyStructConverter::Init leaks a PyBytes object if the paired PyUnicode_FromStringAndSize call fails (OOM-only path; two allocations share one RETURN_IF_PYERROR()). python/pyarrow/src/arrow/python/python_to_arrow.cc:1030-1038
24 PandasBlockCreator::AppendBlocks/SplitBlockCreator::Convert leak item/result on several error-return paths (decref happens after the error check, or not at all on some early returns). python/pyarrow/src/arrow/python/arrow_to_pandas.cc:2290-2302, 2487-2510
25 IsPyUuid's PyObject_IsInstance call clears any failure unconditionally, steering type inference down the "not a UUID" branch with no signal to the caller — low-severity since this only fails for pathological __instancecheck__ implementations. python/pyarrow/src/arrow/python/helpers.cc:338-357
26 PyObject_StdStringRepr's diagnostic PyUnicode_AsEncodedString failure is cleared unconditionally; the function has no propagation path anyway (returns std::string), but a MemoryError/KeyboardInterrupt hitting exactly here is silently absorbed rather than surfaced. python/pyarrow/src/arrow/python/helpers.cc:107-124
27 Two sites (arrow_to_pandas.cc binary-wrapping, numpy_to_arrow.cc UTF-32 encoding) clear a real exception and substitute a hand-built generic Status, bypassing the codebase's own exception-preserving RETURN_IF_PYERROR()/ConvertPyError() idiom used almost everywhere else — still returns an error, but discards the original exception type/message (could mask MemoryError). python/pyarrow/src/arrow/python/arrow_to_pandas.cc:1206, numpy_to_arrow.cc:689
28 TypeInferrer::Visit's PyArray_DescrFromScalar result (feeding VisitDType/Observe) and the _NUMPY_UNIFY_PROMOTE_TO macro's PyArray_DescrFromType result are both dereferenced with no NULL check — OOM-only trigger, same descriptor family as Finding 14. python/pyarrow/src/arrow/python/inference.cc:92-96, 284-285, 464
29 MonthDayNanoIntervalToNamedTuple passes three unchecked PyLong_FromLong* results directly into PyStructSequence_SetItem, which — like PyTuple_SET_ITEM — doesn't itself validate the item; an OOM failure would be silently stored. python/pyarrow/src/arrow/python/datetime.cc:597-608
30 Declaration.to_reader() (Acero) and Scanner.to_reader() (Dataset) call blocking C++ entry points without with nogil:, while every sibling method (to_table(), scan_batches(), etc.) releases the GIL for the identical or equivalent underlying call — verified against the actual C++ implementation (DeclarationToReader, AsyncScanner::ToRecordBatchReader → the same ScanBatches() that scan_batches() explicitly treats as GIL-release-worthy). No confirmed hard deadlock (construction dispatches and returns rather than blocking on a future), but a verifiable, load-bearing inconsistency that could bite if a future node type blocks synchronously during startup. python/pyarrow/_acero.pyx:611-626, _dataset.pyx:4032-4042
31 PyExtensionType::GetInstance() is the only method on its class that doesn't construct PyAcquireGIL itself, relying entirely on the caller already holding the GIL — no live bug found (both current callers already hold it), but it's the one self-undefending method on a class whose siblings all defend themselves. python/pyarrow/src/arrow/python/extension_type.cc:162-181
32 FlightServerBase.init() sets self.server (the shared_ptr) before calling c_server.Init(); if Init() raises, the finalizer that ensures orderly Shutdown()/Wait() (added for ARROW-16597 to avoid an interpreter-exit segfault) is never registered, so a partially-initialized server relies on the plain C++ destructor alone. Not independently verified how much of PyFlightServer::Init executes before failure (core cpp/ out of scope). python/pyarrow/_flight.pyx:2958-3031
33 NativeFile.__cinit__/__dealloc__'s own_file flag is initialized False and gates the only explicit close() call in __dealloc__, but is never set True anywhere in the codebase — dead code, not itself a leak (the wrapped Arrow C++ streams are independently shared_ptr-managed), but could mask a real gap if a future subclass relies on it. python/pyarrow/io.pxi:121-130
34 array() and record_batch() each independently implement the identical ~24-line PyCapsule import-protocol block (__arrow_c_device_array__/__arrow_c_array__ handling) — ~40 duplicated lines and two of the highest branch counts in the module; table() already avoids this by delegating to record_batch(), showing the fix pattern already exists in-tree. python/pyarrow/array.pxi:282-305, table.pxi:6005-6034
35 S3FileSystem.__init__ (24 params, cyclomatic 30) entangles a 5-way mutually-exclusive credential-mode validation with ~15 straight-line field copies mirroring the C++ S3Options struct — the validation logic is where bugs would hide; extracting it from the boilerplate would isolate that risk. python/pyarrow/_s3fs.pyx:285-427
36 TypeInferrer::GetType encodes a 16-branch type-unification priority order as implicit if/elif sequencing rather than a declarative, ordered table — correct today, but silently breaks if a future counter is inserted in the wrong branch position. python/pyarrow/src/arrow/python/inference.cc:522-620
37 NumPyConverter::ConvertData<Date32Type>/<Date64Type> duplicate an identical ~15-line cast-fallback tail, differing only in the day-unit branch (narrowing cast vs. millisecond-scaling multiply) — extracting the shared tail would make the real difference visually obvious instead of buried in two near-identical blocks. python/pyarrow/src/arrow/python/numpy_to_arrow.cc:487-562
38 NativeFile.download() hand-rolls a background writer thread, bounded queue, two closures capturing mutable shared state, a malloc'd buffer, and a busy-wait for backpressure — flagged purely as complexity (nesting 5, several interacting closures over shared mutable state are hard to verify correct by inspection), not as an asserted defect beyond Finding 16. python/pyarrow/io.pxi:642-739
39 Dataset.take/head/scanner/to_batches/to_table/count_rows repeat near-identical docstring-plus-one-line-forwarding boilerplate across ~8 call sites in the class hierarchy — a duplication cost, not a per-function complexity problem. python/pyarrow/_dataset.pyx (repeated)
40 All ~24 Cython modules use single-phase init with no Py_mod_multiple_interpreters marker; under CPython's stricter subinterpreter checks (check_multi_interpreter_extensions, the PEP 734 default posture), pyarrow is rejected outright at import time in a subinterpreter. Not actionable unless subinterpreter support becomes a real requirement — see the Migration Assessment note below. all .pyx files
41 The pandas_*/uuid_UUID C++ statics and the Cython cdef-static caches (pandas_api, _type_cache family) are shared-library-wide, not module-scoped or reachable via PyModule_GetState() — a different lens on the same globals covered thread-safety-wise in reports/ft_review.md: here the concern is cross-interpreter aliasing/lifecycle, which would corrupt state across any future subinterpreters rather than race under free-threading. python/pyarrow/src/arrow/python/helpers.cc:338-370, python/pyarrow/pandas-shim.pxi:282, types.pxi:36,3743,4119-4121
42 MonthDayNanoTupleType is a static (non-heap) PyTypeObject, a process-wide singleton that cannot be made per-interpreter without conversion to a heap type — cross-references reports/ft_review.md Finding 13 (race angle) with a distinct subinterpreter-readiness angle here. python/pyarrow/src/arrow/python/datetime.cc:61, 276-283
43 The _true Expression sentinel (Expression._scalar(True)) is duplicated verbatim across two independently-compiled modules — harmless (read-only), but illustrates that any future multi-phase migration must be repeated per compiled module, not once. python/pyarrow/_dataset.pyx:159, _dataset_parquet.pyx:65
44 6 of ~30 top-level .pyx/.pxd files have no explicit # cython: language_level = 3 directive (all others do); not a functional break today (CMake enforces Cython ≥3.1), but inconsistent and worth making explicit ahead of a future Cython 4 release where defaults could change. python/pyarrow/lib.pyx, _cuda.pyx, _orc.pyx, _parquet.pyx, _parquet_encryption.pyx, _pyarrow_cpp_tests.pyx
45 io.cc:170's PyObject_HasAttrString silently reports any exception during attribute lookup (e.g. a raising __getattr__/property) as "attribute absent" rather than propagating it; PyObject_HasAttrStringWithError (3.13+, already backported by the vendored pythoncapi_compat.h used elsewhere in the codebase, just not included in this file) is the error-propagating replacement. python/pyarrow/src/arrow/python/io.cc:170
46 If the placeholder pyarrow-stubs tree in this repo were ever wheel-packaged or published as its own PyPI -stubs distribution (it currently is not — sdist-only, per PEP 561 stub-package precedence rules), it would silently mask pyarrow's own py.typed inline annotations behind a catch-all __getattr__(name) -> Any. Currently POLICY/non-user-facing (maintainers already track removing the Any-masking __getattr__ before real stubs ship, GH-48970); worth a CI guard if packaging ever changes. python/pyarrow-stubs/pyarrow/__init__.pyi, python/pyproject.toml:85-90

Tensions

  • udf.cc severity: refcount-auditor's initial HIGH vs. git-history-analyzer's traced-through MEDIUM. Both analyses agree the bug is real (a genuine −1 refcount imbalance from a twin-fix asymmetry), but disagree on blast radius. refcount-auditor's first read suggested per-call risk; tracing the actual shared_ptr ownership through cpp/src/arrow/compute/function.cc showed the defect fires once at kernel/registry teardown, gated on a custom FunctionRegistry being destroyed mid-process. → Resolved in favor of the historian's traced analysis (Finding 4 reflects the corrected MEDIUM severity and precise trigger condition); this is presented as a positive example of this review's cross-agent verification catching an initial overstatement.
  • StringToTzinfo's PyErr_Clear: FIX (pyerr-clear-auditor) vs. CONSIDER/policy-adjacent (git-history-analyzer). The specialized agent flagged the unconditional clear as a bug; the historian, having traced it to a deliberate 6-week-old design commit, frames the fallback itself as intentional and the type-blindness of the clear as the actual, narrower defect. → Both framings are preserved in Finding 19's description rather than picking one; the actionable fix (check for the specific lookup error before clearing) is the same either way.
  • Stable-ABI feasibility vs. subinterpreter feasibility: both assessments (Findings 40 and 47) independently conclude "not feasible today," but for different root causes — subinterpreters are blocked by pyarrow's own init-style and global-state architecture (fixable, if expensive); abi3 is blocked primarily by an upstream dependency (NumPy's C-API has no limited-API path) that pyarrow cannot fix unilaterally. Worth keeping separate in any roadmap discussion — one is "our debt," the other is "someone else's prerequisite."

Policy Decisions (POLICY) — 9

# Finding
47 Stable ABI / limited API adoption is infeasible today, not from a single unfixable line but from the combination of: NumPy's own C-API (171 call sites, no limited-API-compatible header path — an upstream prerequisite, not a pyarrow bug), the CPython datetime C-API's struct-access macros (no clean limited-API equivalent short of a hot-path perf regression), a hand-written static PyStructSequence type, and ~24 modules' worth of Cython-generated static types (mechanically migratable via Cython's own cython_limited_api=True, but gated on the first two). Zero genuine private (_Py*) API usage found — a clean starting point if this is ever revisited. Recommendation: keep the current build model; revisit only if NumPy ships limited-API-compatible headers.
48 InitDatetime, NewMonthDayNanoTupleType, and import_pyarrow call Py_FatalError (process abort) on init failure instead of propagating an ImportError — import pyarrow can kill the host process rather than raising a catchable exception if datetime-capsule or struct-sequence init fails.
49 csv.cc's invalid-row handler deliberately swallows user exceptions via PyErr_WriteUnraisable and returns a default result — the handler runs on a non-Python thread, so this is a considered design choice, but it does mean a raising handler cannot abort the read.
50 flight.cc's ServeWithSignals leaves a KeyboardInterrupt raised by PyErr_CheckSignals() pending behind a Status::OK() return (acknowledged by an in-code // XXX comment) — relies on the Cython caller's except * to notice.
51 Cython is pinned >= 3.1 in pyproject.toml, both requirements files, and CMakeLists.txt's version check, with no upper bound anywhere. Cython 4 hasn't shipped yet, so this is prospective rather than active — recommend adding <4 proactively so a future major release doesn't silently build against an untested toolchain.
52 helpers.cc's pandas_*/uuid_UUID statics deliberately hold borrowed (not owned) references, with an in-code comment explaining the trade-off: OwnedRef destructors would run on a finalized interpreter otherwise. Correct and intentional for the single-interpreter case; Finding 41 covers the subinterpreter-era caveat this trade-off doesn't anticipate.
53 The default memory-pool singleton (memory.pxi/common.cc) and the signal_handlers_enabled global flag are appropriately process-global given that they wrap an inherently process-wide C++ allocator singleton and an OS-level (main-thread-only) signal-handling concept respectively — not pyarrow-specific design gaps.
54 Numerous module-level dispatch tables and caches (_array_classes, _scalar_classes, _pep3118_type_map, PRIMITIVE_TYPES, _type_aliases, DEFAULT_CALL_OPTIONS, _PARQUET_READ_OPTIONS) are populated once at import and never mutated afterward — functionally read-only constants despite being mutable container types; no action needed.
55 CategoricalWriter::AddResultMetadata does an extra, unnecessary Py_INCREF on Py_True/Py_False before PyDict_SetItemString (which increfs again) with no matching decref — harmless in practice since these are immortal singletons in 3.12+, but could be tidied for clarity.

Strengths

  • RAII discipline is close to flawless. PyAcquireGIL/PyReleaseGIL/SafeCallIntoPython are the only mechanism used to touch Python objects from contexts that might not hold the GIL, verified call-site-by-call-site across flight.cc, filesystem.cc, csv.cc, udf.cc, ipc.cc, async.h. Zero missing-release findings anywhere in scope.
  • RETURN_IF_PYERROR()/CheckPyError()/ConvertPyError() is a genuinely good, pervasive idiom for bridging CPython exceptions into Arrow Status while preserving the original exception object for later restoration — the deviations from it (Findings 6, 12, 13, 27) are the exception, not the rule.
  • The buffer-protocol implementations (PyBuffer, Buffer.__getbuffer__, Tensor.__getbuffer__) are textbook-correct, including the interpreter-finalization guard (Py_IsInitialized() checks) added after a 2023 shutdown-segfault bug report.
  • Only 3 __dealloc__ methods exist in the entire ~1023-function Cython surface, and all three are correct — nearly every native resource is owned by shared_ptr/unique_ptr members with implicit C++ destruction, which is safe by construction and explains why resource-lifecycle findings were so sparse (2 FIX, both in one file).
  • Version-compat posture is clean: zero dead PY_VERSION_HEX guards, zero unguarded post-floor API usage, zero unmigrated deprecated-API sites beyond what's already tracked, and correct, already-adopted use of pythoncapi_compat.h's 3.13+ *Ref APIs (PyDict_GetItemRef, PyDict_GetItemStringRef) in 4 files — the right template for any future modernization.
  • Complexity is genuinely low on average (avg cyclomatic 3.9, avg function length 20 lines across the C++ glue; only 1 Cython function exceeds 200 lines in a ~1023-function corpus), and several apparent "hotspots" by raw line count turned out to be docstring-heavy trivial forwarders on manual inspection — the estimator's false positives were caught before being reported.

Code Removal Opportunities

Cross-referenced against <plugin_root>/data/deprecated_apis.json:

  • PyModule_AddType (3.10+): Not applicable — grepped PyType_FromSpec/PyType_Ready/PyModule_AddObject across all hand-written glue files; zero hits. Type registration for all ~24 Cython modules is handled by Cython's own codegen, not hand-rolled boilerplate.
  • PyImport_ImportModuleAttrString (3.14+): applicable in principle to ~13 call sites of the ImportModule()+ImportFromModule() pattern in helpers.cc and its callers (arrow_to_pandas.cc, datetime.cc, decimal.cc, inference.cc), collapsing import-then-getattr into one call. Not actionable now — the declared floor is 3.10 and the vendored pythoncapi_compat.h predates this function's addition. Estimated savings if/when adopted: ~65-100 lines across 13 sites.
  • PyDict_GetItemRef (3.13+, backported): already adopted at numpy_to_arrow.cc:832 and python_to_arrow.cc:1151 via the vendored compat header — no remaining sites of the older PyDict_GetItemWithError pattern to migrate.
  • PyObject_HasAttrStringWithError (3.13+, backported): 1 unmigrated site, io.cc:170 (Finding 45) — zero line-count change, fixes a real exception-swallowing bug rather than saving lines.

Migration Assessment (Subinterpreter Readiness)

Not requested as a standalone command, but surfaced strongly enough by module-state-checker to summarize here: subinterpreter support is not feasible without a multi-quarter architectural project. All ~24 modules would need conversion to multi-phase init; every cdef module-level global (Cython has no automatic mechanism to route these into PyModule_GetState()-reachable state) would need manual extraction into per-module structs, repeated independently across all 24 compiled modules; and the shared-library-wide C++ statics (pandas_*, datetime_api, MonthDayNanoTupleType, NumPy's own API table) would need to move to per-PyInterpreterState storage, touching every call site in the conversion machinery. Recommendation: keep the current single-phase design unless subinterpreter support becomes an actual product requirement; if free-threading work (already tracked in reports/ft_review.md) or this subinterpreter question both become live, revisit together since they touch the same global-state inventory.

Recommended Action Plan

Immediate (FIX items)

  1. Fix the two permanent leaks (Findings 1, 2): change PyFlightServer::server_ and PyKmsClient::handler_ to hold weakrefs, resolved via PyWeakref_GetRef at each use — mirroring the pattern PyExtensionType::type_instance_ already uses correctly for exactly this reason.
  2. Fix ExtensionType unhashability (Finding 3): add a __hash__ alongside the existing __eq__ override, computed from the same fields (type(self), extension_name, storage_type).
  3. Fix the UDF registration refcount imbalance (Finding 4): reuse the single OwnedRefNoGIL already constructed for kernel_init as the source for udf_data in RegisterUdf, matching the aggregate family's pattern exactly.
  4. Fix numpy_init.cc's ordering bug (Finding 5): only set numpy_imported = true after import_numpy() returns success.
  5. Add the missing exception checks in AppendSequence/AppendIterable (Finding 6): RETURN_IF_PYERROR() after both the size lookup and the iteration loop.
  6. Fix the two confirmed leaks (Findings 7, 8): decref line in common.cc:166; fix the shadowed-variable bug in ParseBinary to assign through this->ref.
  7. Fix the remaining refcount leaks (Findings 9, 10, 11): wrap offset in an OwnedRef in StringToTzinfo; wrap the four PyLong fields in OwnedRef in the MonthDayNanoInterval writer; hold lst in an OwnedRef in ConvertToSequenceAndInferSize.
  8. Add NULL checks (Findings 12, 13, 14, 15): guard GetInstance()'s result in ToString; guard is_nan's result in PyDecimal_ISNAN; add RETURN_IF_PYERROR() after the PyArray_Descr* family's allocations; check wrap_array/wrap_scalar before storing into the UDF call tuple.
  9. Fix the io.pxi download() error-path leak (Finding 16): move open() inside the try/finally (or restructure so cleanup() is reachable before the writer thread starts).
  10. Fix the three PyErr_Clear findings (Findings 17, 18, 19): check the specific expected exception type (TypeError/UnicodeDecodeError/the zoneinfo lookup error) before clearing in each of the three sites, propagating anything else via ConvertPyError().

Short-term (CONSIDER items)

  1. Fix OwnedRef::operator=(OwnedRef&&) to release-before-detach (Finding 20) — cheap, prevents a latent public-header footgun.
  2. Fix the SafeCallIntoPython exception-triple leak (Finding 21) and give NumPyDtypeUnifier proper ownership discipline (Finding 22).
  3. Fix the remaining OOM-only leaks (Findings 23, 24) and the two PyErr_Clear call sites that bypass the codebase's own exception-preserving idiom (Finding 27) — low urgency, mechanical fixes.
  4. Add with nogil: to to_reader() in both Acero and Dataset for consistency with sibling methods (Finding 30); add self-GIL-acquisition to GetInstance() for consistency with its siblings (Finding 31).
  5. Reorder FlightServerBase.init() to register the finalizer before or immediately after Init() regardless of outcome (Finding 32); remove the dead own_file flag or wire it up (Finding 33).
  6. Deduplicate the array()/record_batch() PyCapsule-import block (Finding 34) using table()'s existing delegation pattern as the template; extract S3FileSystem's credential validation (Finding 35); make TypeInferrer::GetType's priority order an explicit table (Finding 36); deduplicate the ConvertData<Date32/64Type> cast tail (Finding 37).
  7. Add the 6 missing language_level directives (Finding 44) and switch io.cc:170 to PyObject_HasAttrStringWithError (Finding 45).

Longer-term (POLICY)

  1. Add an upper version bound (<4) to the Cython pin in all four locations before Cython 4 ships (Finding 51).
  2. Revisit stable-ABI adoption only if NumPy ships a limited-API-compatible C-API surface (Finding 47); revisit subinterpreter support only if it becomes an actual product requirement, ideally bundled with any future free-threading architectural work (Finding 40 / Migration Assessment).
  3. Consider whether Py_FatalError in the three init paths (Finding 48) should be softened to a catchable ImportError — a process abort on import pyarrow failure is a harsh user experience for what may be a recoverable environment issue.

Related reports: reports/ft_review.md (free-threading / PEP 703 analysis, 26 findings — cross-referenced at Findings 41, 42 above).

C Extension Analysis Report — torch/csrc/dynamo/

Extension: torch._C._dynamo

Scope: torch/csrc/dynamo/ (18 files — 13 C/C++, 5 headers, ~18,000 lines)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner,

gil-discipline-checker, resource-lifecycle-checker, module-state-checker,

type-slot-checker, pyerr-clear-auditor, stable-abi-checker,

version-compat-scanner, parity-checker, c-complexity-analyzer,

git-history-analyzer


Executive Summary

The dynamo C extension is a large, complex piece of machinery that hooks CPython's frame evaluation loop to implement JIT compilation. It is structurally sound but carries 9 FIX-class bugs concentrated in three areas: reference counting in pybind11 interop, null-safety around the PyUnicode_AsUTF8 API, and module-level GC visibility of hook callbacks. An additional FIX was found in the .pyi stubs (a method advertised but not implemented in C). The extension has excellent GIL discipline on the standard build, correct type-slot definitions, thorough version guards for CPython internals, and a comprehensive pattern of treating guard-evaluation failures as recompilation triggers (not propagated exceptions) — a design choice that is consistent and correctly applied throughout. Stable ABI migration is architecturally impossible given the depth of CPython internal access required.


Extension Profile

Property Value
Modules torch._C._dynamo (4 submodules: eval_frame, guards, autograd_compiler, init)
C/C++ files 13 source + 5 headers
Total lines ~18,000
Init style Single-phase (PyModule_Create) in all 4 modules
Python targets >=3.10
Limited API No
Code generation Mixed: eval_frame.c + extra_state.cpp hand-written C/C++; guards.cpp + python_compiled_autograd.cpp pybind11 C++
Types defined 3 (THPPyInterpreterFrame, TensorGuards, GlobalStateGuard)
Free-threading 4 modules declare Py_MOD_GIL_NOT_USED
External tools cppcheck: 11 findings, all false positives (Py_RETURN_* macros + C++ headers parsed as C)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 2 0 THPPyInterpreterFrame leaked on callback exception; dangling py::handle
Error Handling Y 1 2 cache_entry.cpp wrong NULL check after PyUnicode_AsUTF8 (cross-validated)
NULL Safety Y 2 2 get_exception_message std::string(nullptr) UB; DeviceGuard unguarded GetItem
GIL Discipline G 0 2 No standard-build violations; 2 FT-gap atomicity concerns
Resource Lifecycle Y 2 1 module PyObject leaked; python_verbose_logger no INCREF
Module State R 1 5 guard_error_hook GC-invisible (no m_traverse); 8+ globals outside ModuleState
Type Slots G 0 1 All 3 types correct; frame escape lifetime CONSIDER
PyErr_Clear Safety Y 1 2 SetGetItemGuardAccessor NULL deref before clear
ABI Compliance G 0 0 17 private API uses, all NECESSARY and versioned; migration infeasible
Version Compat G 0 5 PyErr_Fetch deprecated; PyDict_GetItemString replaceable
C/Python Parity Y 1 3 type_of_guarded_value stub advertises method with no C binding
Complexity Y 1 3 TensorGuards_check_verbose UAF; dynamo__custom_eval_frame 351 LOC

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings


Findings by Priority

Must Fix (FIX) — 9

# Finding File:Line Agents
1 dynamo_call_callback leaks THPPyInterpreterFrame when callback(...) raises — py::error_already_set propagates past Py_DECREF(frame) eval_frame_cpp.cpp:264-277 refcount-auditor
2 recursive_callback is py::handle; assigned from py::object temporary returned by get_fail_callback — temporary destructs immediately, leaving a dangling pointer installed in TLS via eval_frame_callback_set eval_frame_cpp.cpp:406, 450-457 refcount-auditor
3 CacheEntry::CacheEntry checks if (trace_annotation) (a py::object, always truthy) instead of if (trace_annotation_str) (the const char* result of PyUnicode_AsUTF8) — std::string(nullptr) is UB on failure cache_entry.cpp:13-18 null-safety-scanner, error-path-analyzer
4 torch_c_dynamo_eval_frame_init returns NULL on PyModule_AddType failure without Py_DECREF(module) — module object leaked eval_frame.c:861 resource-lifecycle-checker
5 set_verbose_logger stores logger PyObject pointer without Py_INCREF — use-after-free when Python GC collects the logger python_compiled_autograd.cpp:56, 649-653 resource-lifecycle-checker, module-state-checker
6 eval_frame.c _module has m_size=sizeof(ModuleState) and m_clear but no m_traverse — guard_error_hook and guard_complete_hook are file-scope PyObject* globals invisible to the GC; reference cycles involving hooks cannot be broken eval_frame.c:21-22, 826-832 module-state-checker
7 SetGetItemGuardAccessor::check_nopybind and check_verbose_nopybind call PyList_GetItem(lst, _index) when lst may be NULL (from a failed PySequence_List) — crashes before PyErr_Clear runs guards.cpp:5710-5715, 5724-5730 pyerr-clear-auditor
8 TensorGuards_check_verbose calls Py_DECREF(type_str) at line 547 then passes PyUnicode_AsUTF8(type_str) result into Py_BuildValue at line 549 — the C string pointer is now dangling guards.cpp:547-549 c-complexity-analyzer
9 guards.pyi declares def type_of_guarded_value(self) -> str: ... but no C binding exists for this name — AttributeError at runtime for any caller using the stub contract torch/_C/_dynamo/guards.pyi:192 parity-checker

Finding Detail: FIX items

Finding 1 — THPPyInterpreterFrame leak on callback exception

dynamo_call_callback allocates a THPPyInterpreterFrame via tp_alloc (refcount=1), then calls callback(frame, cache_entry_obj, frame_state). If the callback raises, pybind11 propagates py::error_already_set past Py_DECREF(frame) at line 276, permanently leaking the wrapper.

Fix: wrap frame in py::reinterpret_steal<py::object> so RAII handles the decref on all paths.

py::object frame_obj = py::reinterpret_steal<py::object>((PyObject*)frame);
py::object result = callback(frame_obj, cache_entry_obj, py::handle(frame_state));
return result;

Finding 2 — dangling py::handle / use-after-free

Inside eval_custom lambda (line ~450):

recursive_callback =
    convert_frame_get_fail_callback.value()(recursive_callback);

convert_frame_get_fail_callback.value()(...) returns py::object (new ref). Assigning to py::handle recursive_callback copies only the raw pointer; the py::object temporary destructs immediately, Py_DECREFing the closure to 0. The freed closure pointer is then stored in TLS via eval_frame_callback_set at line 457 and invoked on recursive frames — use-after-free.

Fix: hold the result in a py::object variable with sufficient lifetime alongside recursive_callback.

Finding 3 — wrong NULL check

const char* trace_annotation_str = PyUnicode_AsUTF8(trace_annotation.ptr());
if (trace_annotation) {   // BUG: checks py::object (always truthy after .attr())
    this->trace_annotation = std::string(trace_annotation_str);  // std::string(nullptr) = UB
} else {
    this->trace_annotation = "Unknown";  // dead code
}

Fix: if (trace_annotation_str) — check the const char*, not the wrapper.

Finding 6 — missing m_traverse for hook callbacks

static PyModuleDef _module = {
    PyModuleDef_HEAD_INIT, "eval_frame",
    .m_size = sizeof(ModuleState),
    .m_clear = clear_state,  // present
    // m_traverse MISSING — guard_error_hook and guard_complete_hook are file-scope
    // PyObject* globals not in ModuleState, invisible to the cyclic GC
};

Fix: move guard_error_hook and guard_complete_hook into ModuleState; add m_traverse that calls Py_VISIT on them.

Finding 7 — NULL list passed to PyList_GetItem

PyObject* lst = PySequence_List(obj);
PyObject* x = PyList_GetItem(lst, _index);  // segfault if lst == NULL

Fix: check lst != nullptr before calling PyList_GetItem.

Finding 8 — use-after-free in TensorGuards_check_verbose

PyObject* type_str = PyObject_Str(type_obj);
// ... ~10 lines later:
Py_DECREF(type_str);  // type_str freed here
// ... then:
Py_BuildValue("s", PyUnicode_AsUTF8(type_str));  // ptr into freed object

Fix: move Py_DECREF(type_str) after Py_BuildValue completes, or use py::object RAII.


Should Consider (CONSIDER) — 27

Null Safety / Error Handling:

# Finding File:Line
10 get_exception_message() passes PyObject_Str(pvalue) result directly to std::string(...) without NULL check — UB on exception-with-broken-str guards.cpp:1249-1250
11 DeviceGuard::check_nopybind uses PyDict_GetItem(_utils_device_dict, current_device_str) result in py_equals() without NULL check — only site in guards.cpp that does this guards.cpp:2208
12 set_eval_frame discards return value of increment_working_threads / decrement_working_threads (each returns Py_RETURN_NONE = INCREF'd Py_None) — reference leak on Python 3.10-3.11 eval_frame.c:622, 624
13 PyThread_tss_create failure handled with CHECK() / abort() instead of PyErr_SetString + return NULL — inconsistent with the _PyEval_RequestCodeExtraIndex check on line 839 eval_frame.c:847

Resource Lifecycle:

# Finding File:Line
14 eval_frame_callback_key (Py_tss_t) created by PyThread_tss_create at init but never deleted via PyThread_tss_delete — on POSIX, consumes a pthread_key_t per import eval_frame.c:846
15 THPPyInterpreterFrame stores frame->frame (raw _PyInterpreterFrame*) and frame->locals (borrowed dict ptr) — both become dangling if Python code retains the wrapper beyond the callback duration; no enforcement of lifetime contract eval_frame_cpp.cpp:264-276; eval_frame.h:26-30

Module State:

# Finding File:Line
16 8+ mutable file-scope globals not in ModuleState (guard_error_hook, guard_complete_hook, current_isolate_recompiles_id, is_skip_guard_eval_unsafe, fullgraph_compiled_frame_count, fullgraph_error_on_nested_compile, previous_eval_frame, breakpoint_code_objects) — all persist incorrectly across importlib.reload eval_frame.c:21-22, 31, 218-221, 757-765
17 python_compiled_autograd.cpp module has m_size=-1 (no per-module state), no m_clear, no m_traverse — the_autograd_compiler (INCREF'd) and python_verbose_logger (leaked) are file-scope globals with no GC visibility python_compiled_autograd.cpp:54, 56
18 All 4 modules use single-phase init without declaring Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED — incompatibility with subinterpreters is undeclared eval_frame.c:826, guards.cpp, init.cpp, python_compiled_autograd.cpp
19 torch_c_dynamo_guards_init returns nullptr on expose_inductor_symbol failure without Py_DECREF(m) — module object leaked on init failure guards.cpp:7397-7419
20 active_rstate (RuntimeState*) is a non-atomic file-scope global set in RuntimeStateGuard constructor — concurrent backward passes under free-threading would race on this pointer python_compiled_autograd.cpp:96-100

GIL / Free-Threading Gaps (not GIL violations on standard build; gaps under Py_MOD_GIL_NOT_USED):

# Finding File:Line
21 4 non-atomic globals read/written on hot eval path: current_isolate_recompiles_id (int64_t), is_skip_guard_eval_unsafe (bool), fullgraph_compiled_frame_count (int), fullgraph_error_on_nested_compile (bool) — torn reads/writes under FT eval_frame.c:31, 757, 760, 765; eval_frame_cpp.cpp:433-434, 544
22 convert_frame_get_fail_callback (function-static std::optional<py::object>) lazy-initializes with check-then-set — no std::once_flag; concurrent FT entry could double-initialize and double-register atexit eval_frame_cpp.cpp:425-457

PyErr_Clear:

# Finding File:Line
23 THP_PyFrame_MakeAndSetFrameObject pre-3.14 path: if THP_PyFrame_New_NoTrack returns NULL (OOM), the 3 saved exception objects are Py_XDECREF'd without calling PyErr_Restore — pre-existing exception is silently dropped cpython_defs.c:175-182
24 DebugContextGuard::~DebugContextGuard discards any exception raised by ctx.__exit__() without checking whether there was a pre-existing exception first — KeyboardInterrupt from a debugger q command silently swallowed on the clean-state path eval_frame_cpp.cpp:37-51

Version Compatibility:

# Finding File:Line
25 get_exception_message uses PyErr_Fetch — deprecated since Python 3.12; PyErr_GetRaisedException is the replacement (guarded #if IS_PYTHON_3_12_PLUS) guards.cpp:1245-1257
26 THP_PyFrame_MakeAndSetFrameObject pre-3.14 #else branch covers Python 3.11, 3.12, and 3.13 — but PyErr_GetRaisedException was added in 3.12, so the 3.12/3.13 sub-range still uses the deprecated PyErr_Fetch/PyErr_Restore cpython_defs.c:174-184
27 3 PyDict_GetItemString calls in TensorGuards_init return NULL for both "not found" and "exception during hash" with no PyErr_Occurred() distinction; PyDict_GetItemStringRef is available via the already-bundled pythoncapi_compat.h guards.cpp:389, 395, 502
28 TensorGuardsType and GlobalStateGuardType registered with PyModule_AddObject (reference-stealing); PyModule_AddObjectRef (3.10+) is simpler and less error-prone guards.cpp:7375-7388
29 Python 3.15+ stubs in eval_frame.c return NULL without setting a Python exception — violates CPython calling convention (callers propagate the absent exception, corrupting error state) eval_frame.c:529-564
30 Py_TPFLAGS_IMMUTABLETYPE missing from TensorGuardsType and GlobalStateGuardType tp_flags — available since Python 3.10 (the minimum); also flagged in FT analysis as needed for free-threading correctness guards.cpp:7346, 7358

Parity:

# Finding File:Line
31 _set_lru_cache is registered in C (init.cpp:510) but absent from eval_frame.pyi torch/_C/_dynamo/eval_frame.pyi
32 9 GuardAccessor subclasses exposed in C (GenericGetAttrGuardAccessor, GlobalsGuardAccessor, GlobalWeakRefGuardAccessor, FrameLocalsGuardAccessor, GetItemGuardAccessor, ListGetItemGuardAccessor, WeakRefCallGuardAccessor, CallFunctionNoArgsGuardAccessor, TupleIteratorGetItemAccessor) absent from guards.pyi guards.cpp:7629-7735; torch/_C/_dynamo/guards.pyi
33 GuardManager.is_recursive_dict_tag_matching_disabled registered at guards.cpp:7753 but absent from guards.pyi guards.cpp:7753-7755

Complexity:

# Finding File:Line
34 DIMENSION_DYNAMIC_MARKING_GUARD::check_nopybind and check_verbose_nopybind are 173 LOC (McCabe ~18) with the three-phase attribute loop duplicated between the two methods; verbose path has an asymmetric Py_DECREF(actual) ordering that warrants auditing guards.cpp:2810-2982
35 dynamo__custom_eval_frame is 351 LOC (McCabe ~22) with three capture-heavy lambdas (eval_default, eval_custom, fail) that obscure the control-flow graph; eval_custom lambda (50 LOC) should be a named static function eval_frame_cpp.cpp:343-694
36 tensors_definitely_do_not_overlap<Meta> reaches 5-level nesting in its 2D-stride special case; extracting the 2D branch into tensors_definitely_do_not_overlap_2d would reduce max nesting to 3 guards.cpp:1351-1437

Tensions

FT status vs. standard-build races (Findings 21-22): The extension declares Py_MOD_GIL_NOT_USED on all 4 modules, which means free-threaded Python ships without the GIL protecting the shared globals in Findings 21-22. The ft-review-toolkit (see agent_space/ft-analysis.md) found 7 RACE findings including the same 4 globals — those are LIVE DATA RACES on the free-threaded build. Findings 21-22 are listed here as CONSIDER because they do not crash on the GIL build, but they should be treated as FIX in the context of the FT migration.

Exception-swallowing policy in guard paths: ~70 PyErr_Clear calls in guards.cpp guard methods intentionally treat any exception as "guard mismatch." The pattern is consistent and documented (guards.cpp:1690-1695). It is correct for the guard-check use case but unusual — any reader unfamiliar with the design may mistake these for bugs. No action needed.

cpython_defs.c ABI coupling: Functions are copy-pasted from CPython internals. The IS_PYTHON_3_16_PLUS hard error forces review on CPython 3.16. Between 3.11 and 3.15, silent drift is possible. This is an acknowledged design constraint with no better alternative.


Policy Decisions (POLICY) — 4

# Finding
37 Stable ABI migration: not feasible. 17 private/unstable API uses catalogued, all NECESSARY (frame eval hook, _PyInterpreterFrame struct fields, _PyFunction_CopyWithNewCode, _PyStackRef, opcode tables). TorchDynamo cannot exist without CPython internals access. Continue maintaining version guards; keep IS_PYTHON_3_16_PLUS hard-error in cpython_defs.c.
38 Subinterpreter compatibility: not supported and should be declared. All 4 modules use single-phase init, share file-scope PyObject* globals, use static type objects, and bind to _PyInterpreterState_SetEvalFrameFunc which is inherently per-interpreter. Add Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED to all 4 module definitions to make this explicit.
39 GlobalStateGuard under-captures inductor global state (guards.cpp:777 TODO by jansel). Compiled cache entries may be reused across state changes that inductor observes but Dynamo does not guard. Tracking issue should be confirmed open and prioritized.
40 LRU cache design (extra_state.cpp:20): The use_lru = true flag is acknowledged as a short-term fix for issue #166926. The long-term cache eviction strategy for ExtraState remains unresolved.

SAFE Patterns (confirmed correct)

  • eval_frame_callback_key (Py_tss_t): Per-thread storage; the TSS key itself is process-global but the value is per-thread. Accesses from multiple threads are safe by construction. The init at line 849-850 (Py_INCREF(Py_None); eval_frame_callback_set(Py_None)) correctly establishes TLS ownership.
  • set_eval_frame TLS ownership transfer: Old callback reference is correctly transferred to the caller without double-decref. The same-callback case correctly adds an extra INCREF for the caller. Confirmed safe.
  • set_guard_complete_hook ownership: Saves old global ref, updates global with Py_XNewRef, returns old ref — correct swap-and-return pattern.
  • dict_version_state and dict_to_guard_managers (guards.cpp:883, 1688): Protected by c10::Synchronized<> — confirmed safe under FT.
  • RootGuardManager::_lock GIL/mutex ordering (guards.cpp:4165-4229): Py_UNBLOCK_THREADS → std::lock_guard<std::mutex> → Py_BLOCK_THREADS is correct. No Python API called in GIL-released region.
  • compiled_autograd GIL acquisition (python_compiled_autograd.cpp:1197): Called from C++ autograd thread without GIL; correctly uses pybind11::gil_scoped_acquire before any Python object access.
  • Eval-frame hook GIL contract: dynamo__custom_eval_frame is registered via _PyInterpreterState_SetEvalFrameFunc. CPython always calls it with the GIL held (PEP 523 contract). No GIL acquisition needed inside the hook.
  • All 3 PyTypeObject definitions: THPPyInterpreterFrame has only borrowed-reference members (no tp_dealloc needed); TensorGuards correctly deletes its C++ heap allocation; GlobalStateGuard holds no PyObject* members. All slot configurations are correct.
  • Guard-path PyErr_Clear calls (~70 sites in guards.cpp): Intentional design — any exception during guard evaluation = guard mismatch → recompile. Consistent and documented.
  • ExtraState / CacheEntry lifecycle: destroy_extra_state is correctly registered as the code-object destructor; CacheEntry PyObject fields are pybind11 RAII-managed.
  • Private CPython API version guards: All 17 private API uses have correct IS_PYTHON_3_1X_PLUS guards or are aliased to PyUnstable_* once CPython formalizes them.

Strengths

The extension has excellent version-guard discipline — every _PyInterpreterFrame field access, opcode table inclusion, and frame initialization function is correctly branched for 3.10 through 3.14. The cpython_defs.c hard-error on 3.16 prevents silent drift. The guard-path exception semantics are consistent and well-documented. GIL discipline on the standard build is correct throughout, with the RootGuardManager mutex pattern being particularly careful. The pybind11 stubs for eval_frame and compiled_autograd modules match their C implementations exactly.


Code Removal Opportunities

  • PyModule_AddObjectRef (3.10+): Replace Py_INCREF + PyModule_AddObject + error-path Py_DECREF pattern in guards.cpp:7375-7388 (2 sites). Est. 4-6 lines saved, removes a common error pattern.
  • PyDict_GetItemStringRef (3.10+ via bundled pythoncapi_compat.h): Replace 3 PyDict_GetItemString calls in guards.cpp:389,395,502. Eliminates the need for the PyErr_Occurred() ambiguity.
  • PyModule_AddType (3.10+): eval_frame.c already uses this; guards.cpp does not — would replace the PyType_Ready + PyModule_AddObjectRef pattern.

Recommended Action Plan

Immediate (FIX items — 9 total)

  1. Fix Finding 1: Wrap frame in py::reinterpret_steal<py::object> in dynamo_call_callback (eval_frame_cpp.cpp:264) so THPPyInterpreterFrame is released on exception paths.
  2. Fix Finding 2: Change recursive_callback from py::handle to py::object (or introduce a py::object recursive_callback_owned) in the eval_custom lambda (eval_frame_cpp.cpp:450-457).
  3. Fix Finding 3: Check trace_annotation_str (not trace_annotation) after PyUnicode_AsUTF8 in CacheEntry::CacheEntry (cache_entry.cpp:14).
  4. Fix Finding 4: Add Py_DECREF(module); return NULL; on PyModule_AddType failure (eval_frame.c:861-862).
  5. Fix Finding 5: Add Py_INCREF(logger) (and Py_XDECREF on the old value) in set_verbose_logger (python_compiled_autograd.cpp:651-653).
  6. Fix Finding 6: Move guard_error_hook and guard_complete_hook into ModuleState; add m_traverse calling Py_VISIT on them (eval_frame.c:826-832).
  7. Fix Finding 7: Check lst != nullptr before calling PyList_GetItem in SetGetItemGuardAccessor::check_nopybind and check_verbose_nopybind (guards.cpp:5710, 5724).
  8. Fix Finding 8: Move Py_DECREF(type_str) to after Py_BuildValue in TensorGuards_check_verbose (guards.cpp:547-549).
  9. Fix Finding 9: Remove def type_of_guarded_value(self) -> str: ... from guards.pyi (the live binding is get_type_of_guarded_value).

Short-term (CONSIDER items)

  1. Add NULL check to get_exception_message — guard PyObject_Str and PyUnicode_AsUTF8 results before std::string(...) (guards.cpp:1249-1250, Finding 10).
  2. Add NULL check to DeviceGuard::check_nopybind — check PyDict_GetItem result before py_equals (guards.cpp:2208, Finding 11).
  3. Fix PyThread_tss_create error handling — replace CHECK(result == 0) with PyErr_SetString + return NULL (eval_frame.c:847, Finding 13).
  4. Add Py_TPFLAGS_IMMUTABLETYPE to TensorGuardsType and GlobalStateGuardType tp_flags — available on all supported versions, needed for FT correctness (guards.cpp:7346, 7358, Finding 30).
  5. Add Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED to all 4 module defs to make subinterpreter incompatibility explicit (Finding 18).
  6. Fix guards.cpp module init leak — add Py_DECREF(m) before return nullptr on expose_inductor_symbol failure paths (guards.cpp:7397-7419, Finding 19).
  7. Fix 3.15+ stubs — add PyErr_SetString(PyExc_RuntimeError, "...") in each stub function that returns NULL (eval_frame.c:529-564, Finding 29).
  8. Add missing stub entries: _set_lru_cache in eval_frame.pyi, 9 GuardAccessor subclasses and is_recursive_dict_tag_matching_disabled in guards.pyi (Findings 31-33).

Longer-term (complexity, POLICY, FT readiness)

  1. Free-threading atomics (linked to FT analysis): convert the 4 non-atomic globals (Finding 21) to _Atomic / std::atomic; protect convert_frame_get_fail_callback with std::once_flag (Finding 22). These are LIVE RACES on the free-threaded build — see ft-analysis.md for details.
  2. Expand ModuleState — move at least guard_error_hook, guard_complete_hook, and the 4 boolean/int flags into ModuleState; add m_traverse (extends Finding 6 fix, Finding 16).
  3. Migrate python_compiled_autograd.cpp — add m_size, m_traverse, m_clear for the_autograd_compiler and python_verbose_logger (Finding 17).
  4. Replace deprecated APIs — PyErr_Fetch → PyErr_GetRaisedException (guarded, Findings 25-26); PyDict_GetItemString → PyDict_GetItemStringRef via bundled compat (Finding 27); PyModule_AddObject → PyModule_AddObjectRef (Finding 28).
  5. Refactor dynamo__custom_eval_frame — extract eval_custom lambda (50 LOC) to a named static function to make the control flow graph legible (Finding 35).
  6. Deduplicate DIMENSION_DYNAMIC_MARKING_GUARD — extract shared attribute-check logic from check_nopybind + check_verbose_nopybind into a helper; audit Py_DECREF(actual) ordering in verbose path (Finding 34).
  7. Confirm issue #166926 status for the LRU eviction design (Finding 40 / extra_state.cpp:20).

Cross-References

  • Free-threading safety analysis (7 RACE + 1 UNSAFE + 4 PROTECT + 4 MIGRATE findings, including the CRITICAL ExtraState concurrent std::list mutation): agent_space/ft-analysis.md.
  • CONSIDER items 21, 22, 30 in this report overlap with PROTECT/MIGRATE items in ft-analysis.md. They are listed here as C extension correctness concerns; the FT report covers them as live-race severity under Py_MOD_GIL_NOT_USED.

Generated 2026-06-15 by cext-review-toolkit:explore Agents: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer

C Extension Analysis Report
Extension: pywin32
Scope: Entire project (win32/, com/, isapi/, pythonwin/) — vendored Scintilla and
MAPIStubLibrary excluded
Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner,
gil-discipline-checker, resource-lifecycle-checker, module-state-checker,
type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner,
git-history-analyzer (parity-checker and c-complexity-analyzer blocked by tool
permissions — not assessed)
Branch: nogil+swig4
---
Executive Summary
pywin32 is a large, mature Win32/COM binding library whose C extension layer contains
a mix of long-standing correctness bugs and new technical debt introduced by the
active free-threading migration. The highest-priority issue is Register.cpp:232 — a
Py_DECREF on a borrowed reference from PyDict_GetItem that is a crash-severity
refcount corruption bug on every Python build. A cluster of similarly severe
all-builds correctness bugs follow: an always-FALSE COM gateway (PyIPropertyStorage
ReadMultiple), a Win32 API function that returns success with an active exception
set, and _Py_NewReference-pattern type constructors with Py_TPFLAGS_BASETYPE but no
tp_new, causing allocator mismatch for Python subclasses. The free-threading
migration (Py_MOD_GIL_NOT_USED declared for all ~30 modules) is structurally correct
but the underlying C code retains numerous unprotected shared dicts, GIL-reliant
lazy-init singletons, and callback-thread races that will trigger data corruption
under free-threaded Python 3.13t+. The version compatibility posture is disciplined —
no unguarded post-3.9 APIs — but 31 deprecated PyErr_Fetch/Restore call sites and
~50 files of deprecated T_* struct members need migration before Python deprecation
removal. Recommended first action: fix finding 1 (Register.cpp:232) and finding 2
(PyIPropertyStorage.cpp:451-458) immediately; they are one-line and four-line fixes
respectively and affect every pywin32 user.
---
Extension Profile
┌────────────────┬───────────────────────────────────────────────────────────────┐
│ Field │ Value │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Modules │ ~30 C extension modules │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Source files │ 214 C++ files + 11 SWIG .i files │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Init style │ Single-phase (PyModule_Create) via PYWIN_MODULE_INIT_PREPARE │
│ │ macro │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Python targets │ ≥3.9 (CI: 3.9–3.15 + 3.13t/3.14t/3.15t free-threaded) │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Limited API │ No │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Types defined │ ~21 static PyTypeObjects (win32/), ~50+ in com/ │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Code │ Mixed: hand-written C++ + SWIG 1.1-generated wrappers │
│ generation │ │
├────────────────┼───────────────────────────────────────────────────────────────┤
│ Free-threading │ Py_MOD_GIL_NOT_USED declared; C-level audit in progress │
└────────────────┴───────────────────────────────────────────────────────────────┘
---
Key Metrics
┌───────────────┬────────┬─────┬──────────┬──────────────────────────────────────┐
│ Dimension │ Status │ FIX │ CONSIDER │ Top Finding │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Refcount │ 🔴 │ 9 │ 4 │ Py_DECREF on borrowed PyDict_GetItem │
│ Safety │ │ │ │ ref — Register.cpp:232 │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Error │ 🔴 │ 7 │ 5 │ Always-FALSE ReadMultiple gateway — │
│ Handling │ │ │ │ PyIPropertyStorage.cpp:451 │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ NULL Safety │ 🟡 │ 5 │ 5 │ Unchecked PyTuple_New in COM typelib │
│ │ │ │ │ conversion (OOM crash) │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ GIL │ │ │ │ PyWin_SetAPIError inside │
│ Discipline │ 🔴 │ 1 │ 2 │ Py_BEGIN_ALLOW_THREADS — │
│ │ │ │ │ win32trace.cpp │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Resource │ 🟡 │ 3 │ 2 │ GlobalAlloc leak on SetClipboardData │
│ Lifecycle │ │ │ │ failure │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Module State │ 🟡 │ 5 │ 5 │ COM singletons (g_obEmpty etc.) │
│ │ │ │ │ overwritten on reimport │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Type Slots │ 🔴 │ 4 │ 6 │ PyHANDLE hash/equality inconsistency │
│ │ │ │ │ breaks dict/set │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ PyErr_Clear │ 🟡 │ 4 │ 7 │ COM gateway QueryInterface swallows │
│ Safety │ │ │ │ MemoryError │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ ABI / Private │ 🟡 │ 0 │ 5 │ _PyLong_Sign deprecated in 3.14 (not │
│ APIs │ │ │ │ removed) │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Version │ 🟡 │ 0 │ 3 │ 31 PyErr_Fetch/Restore deprecated │
│ Compat │ │ │ │ sites across 8 files │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ C/Python │ ⬜ │ — │ — │ Not assessed — agent blocked │
│ Parity │ │ │ │ │
├───────────────┼────────┼─────┼──────────┼──────────────────────────────────────┤
│ Complexity │ ⬜ │ — │ — │ Not assessed — agent blocked │
└───────────────┴────────┴─────┴──────────┴──────────────────────────────────────┘
🔴 = 4+ FIX findings | 🟡 = 1–3 FIX findings | ⬜ = not assessed
---
Findings by Priority
Must Fix (FIX) — 35
Findings are globally numbered. All-builds / non-OOM bugs first; OOM-triggered;
resource leaks; module state; adjacent-pattern confirmed bugs follow.
Tier A — All-builds correctness; no OOM required
#: 1
Finding: Py_DECREF(v) on borrowed reference returned by PyDict_GetItem; v is live in
the dict and this decrements its refcount, corrupting the COM type registry
File:Line: Register.cpp:232
Agents: refcount-auditor, git-history
────────────────────────────────────────
#: 2
Finding: Missing braces in PyObject_AsPreallocatedPROPVARIANTs; return FALSE is the
for-loop body, not the error branch — ReadMultiple COM gateway always fails for
non-empty arrays
File:Line: PyIPropertyStorage.cpp:451-458
Agents: error-path-analyzer
────────────────────────────────────────
#: 3
Finding: PyErr_Fetch transfers ownership of 3 objects; after Py_XDECREF(exc_info),
exc_typ, exc_val, exc_tb are never released — 3 leaks + exception permanently lost
per logging call
File:Line: win32gui.i:30-59
Agents: refcount-auditor
────────────────────────────────────────
#: 4
Finding: pPOT->Init(...) called unconditionally after new (PyPerfMonManager) that can
return NULL — NULL pointer dereference crash
File:Line: MappingManager.cpp:136-141
Agents: error-path-analyzer
────────────────────────────────────────
#: 5
Finding: PyWin_SetAPIError (sets Python exception state) called inside an active
Py_BEGIN_ALLOW_THREADS block; corrupts interpreter exception state, identified by
an
inline comment confirming the author knew
File:Line: win32trace.cpp:277-295
Agents: gil-discipline
────────────────────────────────────────
#: 6
Finding: ReturnAPIError("LoadCursor") result is discarded (no return); falls through
to PyWinLong_FromHANDLE(NULL) returning 0 with exception already set — SystemError
triggered on caller's next C API call
File:Line: win32apimodule.cpp:1149-1155
Agents: error-path-analyzer
────────────────────────────────────────
#: 7
Finding: Same discarded-error fall-through for CommandLineToArgvW; returns empty list
with active exception
File:Line: win32apimodule.cpp:1167-1187
Agents: error-path-analyzer
────────────────────────────────────────
#: 8
Finding: PyHANDLE::hash() returns _Py_HashPointer(this) (object address) but
richcompare() uses m_handle value — two distinct PyHANDLE objects wrapping the same
HANDLE compare == but hash differently; silently wrong dict/set behavior
File:Line: PyHANDLE.cpp (hashFunc + richcompare)
Agents: type-slot
────────────────────────────────────────
#: 9
Finding: Same hash/equality inconsistency on PyOVERLAPPED — hashFunc uses pointer,
richcompareFunc uses memcmp of struct contents
File:Line: PyOVERLAPPED.cpp
Agents: type-slot
────────────────────────────────────────
#: 10
Finding: Py_TPFLAGS_BASETYPE set (Python-subclassable) but tp_new = 0 and tp_dealloc
calls delete instead of tp_free — Python subclasses are allocated via tp_alloc
(Python heap) and freed via delete (MSVC heap); undefined behavior / crash under
debug allocators or ASAN
File:Line: PyHANDLE.cpp
Agents: type-slot
────────────────────────────────────────
#: 11
Finding: Same Py_TPFLAGS_BASETYPE + delete-in-dealloc mismatch; tp_new calls new
PyDEVMODEW rather than type->tp_alloc
File:Line: PyDEVMODE.cpp
Agents: type-slot
────────────────────────────────────────
#: 12
Finding: PyObject_RichCompare return value leaked on every loop iteration; NULL
return
from failed comparison not checked — one leak per iteration + silent wrong result
on
failure
File:Line: PythonEng.cpp:147-153
Agents: refcount-auditor
────────────────────────────────────────
#: 13
Finding: valueObject (refcount 1) leaked on PyDict_SetItem failure path
File:Line: Register.cpp:97-106
Agents: refcount-auditor
────────────────────────────────────────
#: 14
Finding: PyErr_Clear() in QueryInterface unconditionally swallows any Python
exception
including MemoryError; commented // ### what to do with exceptions?
File:Line: PyGatewayBase.cpp:165,170
Agents: pyerr-clear
────────────────────────────────────────
#: 15
Finding: PyErr_Clear() called before returning E_OUTOFMEMORY in getids_setup /
invoke_setup — suppresses the very MemoryError that E_OUTOFMEMORY is intended to
signal
File:Line: PyGatewayBase.cpp:259-282
Agents: pyerr-clear
────────────────────────────────────────
#: 16
Finding: PyErr_Clear() called before return E_OUTOFMEMORY in universal gateway vtable
dispatch — allocation failure exception silently discarded
File:Line: univgw.cpp:87
Agents: pyerr-clear
────────────────────────────────────────
#: 17
Finding: VLogF_Logger in ErrorUtils.cpp: when the logger call fails, PyErr_Print() is
called before PyErr_Restore() — PyErr_Print clobbers sys.last_* while the original
exception is conceptually in flight; unsafe under free-threaded Python 3.13t+ where
sys.last_* mutation is a shared-state race
File:Line: ErrorUtils.cpp:389-409
Agents: pyerr-clear
Tier B — OOM-triggered NULL dereferences (crash on allocation failure)
#: 18
Finding: PyTuple_New(len) not checked for NULL; immediate crash in PyTuple_SetItem on
OOM
File:Line: PyFUNCDESC.cpp:15-16
Agents: null-safety
────────────────────────────────────────
#: 19
Finding: PyTuple_New unchecked before immediate PyTuple_SetItem — 3 distinct sites
File:Line: PyIType.cpp:38, 132, 964
Agents: null-safety
────────────────────────────────────────
#: 20
Finding: Same pattern — 2 sites in PyObject_FromARRAYDESC and
PyObject_FromELEMDESCArray
File:Line: PyITypeObjects.cpp:83, 271
Agents: null-safety
────────────────────────────────────────
#: 21
Finding: PyTuple_New(1) unchecked; crashes at PyTuple_SET_ITEM and leaks
previously-built args tuple — in Windows Service startup path
File:Line: PythonService.cpp:1169-1170
Agents: null-safety
────────────────────────────────────────
#: 22
Finding: PyTuple_New(nbr_langs) unchecked; crash at PyTuple_SetItem in
GetFileVersionInfo translation branch
File:Line: win32apimodule.cpp:5492-5498
Agents: null-safety
Tier C — Reachable resource leaks (no OOM required)
#: 23
Finding: HANDLE + PyObject leaked on empty-file error path: if
(!m_obj->mapping_size.QuadPart) returns NULL without Py_DECREF(m_obj), skipping
mmapfile_object_dealloc which would call CloseHandle
File:Line: mmapfilemodule.cpp:569
Agents: resource-lifecycle
────────────────────────────────────────
#: 24
Finding: GlobalAlloc block leaked when SetClipboardData fails — Windows docs require
caller to free on failure; author left // XXX - should we GlobalFree the mem?
comment
File:Line: win32clipboardmodule.cpp:868
Agents: resource-lifecycle
────────────────────────────────────────
#: 25
Finding: Same GlobalAlloc leak in py_set_clipboard_text variant
File:Line: win32clipboardmodule.cpp:945-946
Agents: resource-lifecycle
Tier D — Module state correctness (incorrect behavior on reimport /
multi-interpreter)
#: 26
Finding: g_obEmpty, g_obMissing, g_obArgNotFound, g_obNothing, PyCom_InternalError
created unconditionally every PyInit_pythoncom call with no NULL guard — previous
value leaked on reimport; cross-interpreter aliasing of pythoncom.Empty singleton
File:Line: PythonCOM.cpp:2146-2178
Agents: module-state
────────────────────────────────────────
#: 27
Finding: PyModule_AddObject(module, "com_record", &PyRecord::Type) with no prior
Py_INCREF — steals reference from statically-allocated type; attempted dealloc of
static storage on module unload
File:Line: PythonCOM.cpp:2192
Agents: module-state, refcount
────────────────────────────────────────
#: 28
Finding: g_obRegisteredVTables = PyDict_New() in initunivgw() with no NULL guard —
previous dict overwritten on reimport (all registered VTables silently lost);
deferred crash at PyDict_GetItem if PyDict_New returns NULL
File:Line: univgw.cpp:741
Agents: module-state
────────────────────────────────────────
#: 29
Finding: timer_id_callback_map = PyDict_New() unconditionally overwrites on reimport
—
active timer registrations silently lost; Win32 timers continue to fire,
PyDict_GetItem returns NULL, timers orphaned
File:Line: timermodule.cpp:14
Agents: module-state
────────────────────────────────────────
#: 30
Finding: servicemanager_startup_error = PyErr_NewException(...) unconditionally on
every PyInit_servicemanager — memory leak on reimport;
PyErr_ExceptionMatches(servicemanager_startup_error) fails if callers cached the
old
class
File:Line: PythonService.cpp:553
Agents: module-state
Tier E — Adjacent pattern bugs confirmed by history analysis
#: 31
Finding: PyDict_GetItem(g_obPyCom_MapServerIIDToGateway, keyObject) result used then
Py_DECREF'd — borrowed reference; same bug class as finding 1, different function
in
different file
File:Line: PyComHelpers.cpp:274,381
Agents: git-history
────────────────────────────────────────
#: 32
Finding: pStat->pwcsName = NULL without CoTaskMemFree — STATSTG.pwcsName is allocated
by the COM storage layer; zeroing the pointer leaks COM task-allocator memory;
self-documented with // XXX - need to fix this
File:Line: PyComHelpers.cpp:465
Agents: git-history
────────────────────────────────────────
#: 33
Finding: /// XXX - this leaks this variant :-( — developer-acknowledged unfixed
VARIANT leak in type-object conversion loop, no subsequent commit has addressed it
File:Line: PyITypeObjects.cpp:208
Agents: git-history
────────────────────────────────────────
#: 34
Finding: PyTuple_New(cNames) (line 257) immediately precedes the confirmed
PyErr_Clear-masks-OOM region (lines 259-282); if the tuple allocation fails, the
clear at 259 silently drops the MemoryError that PyTuple_New set — fixing finding
15
must also guard line 257
File:Line: PyGatewayBase.cpp:257
Agents: git-history
────────────────────────────────────────
#: 35
Finding: 2021 NULL-guard fix (f4798e0a) covered PyCom_RegisterClientType but left
PyCom_RegisterGatewayObject without a g_obPyCom_MapServerIIDToGateway == NULL guard
— same crash class, incomplete fix
File:Line: Register.cpp (PyCom_RegisterGatewayObject)
Agents: git-history
---
Should Consider (CONSIDER) — 28
#: 36
Finding: PyOVERLAPPED holds obState (arbitrary user-attached object) and obhEvent —
both live PyObject* members — but has no tp_traverse/tp_clear and
Py_TPFLAGS_HAVE_GC
not set; reference cycles through obState invisible to GC; common in asyncio/IOCP
code
File:Line: PyOVERLAPPED.cpp
────────────────────────────────────────
#: 37
Finding: PyFUNCDESC, PyTYPEATTR, PyVARDESC each hold PyObject* members exposed as
assignable T_OBJECT members; no GC flags — user-created cycles not collected
File:Line: PyFUNCDESC.cpp, PyTYPEATTR.cpp, PyVARDESC.cpp
────────────────────────────────────────
#: 38
Finding: 5 ISAPI filter sub-objects (PyURL_MAP, PyPREPROC_HEADERS, etc.) hold strong
PyHFC *m_parent reference; no tp_traverse/tp_clear; parent→child→parent cycles leak
in long-running IIS processes
File:Line: isapi/PyFilterObjects.cpp
────────────────────────────────────────
#: 39
Finding: PyType_Ready return values not checked for 9 ISAPI types in
InitExtensionTypes / InitFilterTypes; partial-init types cause corrupted objects on
later instantiation
File:Line: PyExtensionObjects.cpp:1028, PyFilterObjects.cpp:1107-1113
────────────────────────────────────────
#: 40
Finding: All COM interface types define tp_richcompare but tp_hash = 0 — implicit
PyObject_HashNotImplemented; COM interface objects cannot be used as dict keys or
set members
File:Line: MiscTypes.cpp type_template
────────────────────────────────────────
#: 41
Finding: PyRecord::tp_hash = 0 with tp_richcompare defined — com_record objects
unhashable; value-type records are commonly used as dict keys
File:Line: PyRecord.cpp:398,408
────────────────────────────────────────
#: 42
Finding: univgw.cpp:741 g_obRegisteredVTables = PyDict_New() not checked for NULL —
deferred crash at first PyDict_GetItem (also listed as FIX #28; the crash path is
the FIX part, the unchecked allocation is this CONSIDER)
File:Line: univgw.cpp:741
────────────────────────────────────────
#: 43
Finding: oleargs.cpp:714 PyDict_New() not checked; passed to
PyCom_CalculatePyObjectDimension; NULL dereference inside helper at first
PyDict_GetItem call
File:Line: oleargs.cpp:714
────────────────────────────────────────
#: 44
Finding: dllmain.cpp:66-69 PyList_New(0) not checked; PyList_Append on NULL crashes
File:Line: dllmain.cpp:66
────────────────────────────────────────
#: 45
Finding: adsilib.i, mapilib.i SWIG typemaps call PyList_New(0) then immediately
PyList_Append without NULL check; pattern multiplied across all generated function
wrappers
File:Line: adsilib.i:79,124, mapilib.i:393-461
────────────────────────────────────────
#: 46
Finding: CEnterLeavePython::acquire() calls PyGILState_Ensure without
Py_IsInitialized() guard; used in CTRL_CLOSE_EVENT and service-stop callbacks that
fire during/after interpreter shutdown — crash or deadlock; PythonEng.cpp:62
already
has this guard as the correct model
File:Line: PyWinTypes.h:~702
────────────────────────────────────────
#: 47
Finding: win32trace.cpp:615-631 module init: hMutex leaked if subsequent CreateEvent
fails; hEvent leaked if second CreateEvent fails — bounded (one-time init) but
still
incorrect
File:Line: win32trace.cpp:615-631
────────────────────────────────────────
#: 48
Finding: PyGEnumVariant::Next error path calls PyErr_Clear() unconditionally — if
PyObject_Length raised MemoryError, clear drops it; E_OUTOFMEMORY HRESULT not
returned in that case
File:Line: PyGEnumVariant.cpp:47
────────────────────────────────────────
#: 49
Finding: PyDlgProcHDLG calls PyDict_GetItem + unconditional PyErr_Clear() — should
use
PyDict_GetItemWithError; OOM from dict hash error silently dropped in dialog
message
loop
File:Line: win32gui.i:702
────────────────────────────────────────
#: 50
Finding: win32evtlog.i EvtRender: PyWinObject_FromEVT_VARIANT failure cleared and
replaced with Py_None — swallows MemoryError, caller receives incomplete list
without indication
File:Line: win32evtlog.i:1000-1004
────────────────────────────────────────
#: 51
Finding: odbc.cpp:773-774 ibindFloat returns NULL instead of 0 — this is an
int-returning function; incorrect sentinel causes false-positive error detection on
any subsequent check
File:Line: odbc.cpp:773-774
────────────────────────────────────────
#: 52
Finding: odbc.cpp:1617-1620 PyModule_AddObject failure path leaks obtypes;
PYWIN_MODULE_INIT_RETURN_ERROR does not release it
File:Line: odbc.cpp:1617-1620
────────────────────────────────────────
#: 53
Finding: odbc.cpp:1174-1244 inputvars reference leaked on error paths in bulk-insert
loop
File:Line: odbc.cpp:1174-1244
────────────────────────────────────────
#: 54
Finding: PyWin_InterpreterState captures the first-importing interpreter and is never
updated; PyWinThreadState_Ensure always binds COM callbacks to the first
interpreter's thread state — silent cross-interpreter contamination in
multi-interpreter scenarios
File:Line: PyWinTypesmodule.cpp:139,143-165
────────────────────────────────────────
#: 55
Finding: PyWinExc_ApiError / PyWinExc_COMError created once with NULL guard but
shared
across all interpreters — second interpreter's pywintypes.error is an object owned
by the first interpreter's heap
File:Line: PyWinTypesmodule.cpp:797-860
────────────────────────────────────────
#: 56
Finding: timermodule.cpp:14 / win32assoc.cpp:99-190 ASSERT_GIL_HELD comments +
comment
"GIL prevents races" — these are concurrency correctness assumptions that are false
under Py_GIL_DISABLED (active on this branch)
File:Line: Multiple
────────────────────────────────────────
#: 57
Finding: dllmain.cpp:145,159 — two XXX markers: "regularly hangs" and "Needs more
thought about threading implications"; COM DllMain + Python lock acquisition is a
classic loader-lock deadlock; under free-threading the mask is removed
File:Line: dllmain.cpp:145,159
────────────────────────────────────────
#: 58
Finding: win32notify.cpp:52,70 — HACK HACK - FIX ME FIX ME markers;
PyBytes_FromString("") substituted where actual notification payload expected;
callbacks receive wrong data
File:Line: win32notify.cpp:52,70
────────────────────────────────────────
#: 59
Finding: _PyUnicode_AsString used via PYWIN_ATTR_CONVERT macro (51 callsites) — it's
now defined as PyUnicode_AsUTF8 compat shim in Python 3.14 headers, but relying on
the private name is fragile; replace with PyUnicode_AsUTF8 directly
File:Line: PyWinTypes.h:74
────────────────────────────────────────
#: 60
Finding: _PyLong_Sign marked Py_DEPRECATED(3.14) in Python 3.14 headers;
_PyLong_NumBits return type changed size_t→int64_t (functionally compatible but
signals future removal); used unguarded in COM variant hot path
File:Line: oleargs.cpp:141-142
────────────────────────────────────────
#: 61
Finding: ob_type = &SomeType direct field write pattern (~350 sites) — deprecated in
3.9; Py_SET_TYPE is the correct form and is available at pywin32's 3.9 minimum
File:Line: All C++ type constructors
────────────────────────────────────────
#: 62
Finding: structmember.h T_* macros deprecated in 3.12; ~50 files affected;
PyWinTypes.h:18 includes structmember.h globally
File:Line: PyWinTypes.h:18 + ~50 files
────────────────────────────────────────
#: 63
Finding: AXDebug.cpp frame struct access guard at >= 0x030b0000 (3.11) is too
conservative — PyFrame_GetCode / PyFrame_GetBack have been public since 3.9; the
#else struct-access branch can be dropped entirely
File:Line: AXDebug.cpp:59-61, 189-230
---
Tensions
- Free-threading enablement vs. safety: PYWIN_MODULE_INIT_PREPARE declares
Py_MOD_GIL_NOT_USED for all ~30 modules under Py_GIL_DISABLED, but the C code still
contains ~10 unprotected shared-state patterns (console handlers, RAS callback map,
timer dict, ISAPI g_callbackMap, pythonwin CAssocManager, COM helper singletons). The
declaration is correct as an aspirational gate; the protection work is Phase 1–3 of
the migration plan.
- Single-phase init vs. subinterpreter correctness: The PYWIN_MODULE_INIT_PREPARE
macro intentionally uses single-phase init (no Py_mod_multiple_interpreters). This is
a valid architectural decision for a COM/Win32 binding library where IID maps are
process-wide. However, findings 26–30 (missing NULL idempotency guards on globals)
are not a consequence of this decision — they are mechanical correctness bugs within
the single-phase model that leak memory on any reimport.
- _Py_NewReference architectural approach: The C++ constructor pattern (placement new
+ _Py_NewReference) is used in ~48 files across 119 sites. It is presently
functional — all private symbols verified present in Python 3.14 headers — but is a
long-term fragility as CPython allocator internals evolve. Replacing it requires
converting PyIBase and all ~50 COM subclasses to use tp_alloc/PyObject_Init. This is
a multi-sprint effort and should be tracked as a separate initiative rather than
treated as FIX-priority today.
---
Policy Decisions (POLICY) — 12
#: P1
Finding: 31 PyErr_Fetch/PyErr_Restore/PyErr_NormalizeException call sites across 8
files (ErrorUtils.cpp ×6, pythonpsheet.cpp, win32uimodule.cpp, win32virt.cpp,
PyHANDLE.cpp, PythonService.cpp, win32consolemodule.cpp) — deprecated in Python
3.12; migrate to PyErr_GetRaisedException/PyErr_SetRaisedException (requires #if
PY_VERSION_HEX >= 0x030c0000 guard or pythoncapi-compat)
────────────────────────────────────────
#: P2
Finding: consoleControlHandlers PyList shared between Win32 ctrl-handler OS thread
and
Python mutator threads — comment "thread-safety provided by GIL" is invalid under
Py_GIL_DISABLED; protect with PyMutex + Py_BEGIN_CRITICAL_SECTION
────────────────────────────────────────
#: P3
Finding: obHandleMap in win32rasmodule.cpp shared between RAS OS callback and
PyWinDial — comment "As we hold the thread lock, assume noone else can mod this
dict" invalid under free-threading
────────────────────────────────────────
#: P4
Finding: g_callbackMap in isapi/PyExtensionObjects.cpp protected only by GIL;
concurrent IIS requests race on this dict under free-threading
────────────────────────────────────────
#: P5
Finding: PythonService.cpp:553 static have_init flag with // XXX - this assumes GIL
held comment — races under free-threading; use std::call_once or
InitOnceExecuteOnce
────────────────────────────────────────
#: P6
Finding: PyComHelpers.cpp:37 Decimal_class lazy-init singleton; oleargs.cpp:15
PyVariant_Type; PyTime.cpp:28-46 GetTZUTC statics — all have "relies on GIL"
comments; protect with PyMutex / _Py_once_flag
────────────────────────────────────────
#: P7
Finding: win32credmodule.cpp:436-788 — 9 Cred* functions missing
Py_BEGIN_ALLOW_THREADS; incomplete migration from commit b5f7e74f; full list:
CredMarshalCredential, CredUnmarshalCredential, CredEnumerate, CredGetTargetInfo,
CredWriteDomainCredentials, CredReadDomainCredentials, CredDelete, CredRead,
CredRename
────────────────────────────────────────
#: P8
Finding: win32security.i:1446-1521 — LookupPrivilegeName, LookupPrivilegeDisplayName,
LookupPrivilegeValue missing Py_BEGIN_ALLOW_THREADS
────────────────────────────────────────
#: P9
Finding: WTSWaitSystemEvent (blocks indefinitely), ReadConsoleInput, ReadConsole,
RegConnectRegistry (network), WTSSendMessage(Wait=TRUE) — all missing
Py_BEGIN_ALLOW_THREADS; freeze all Python threads during wait
────────────────────────────────────────
#: P10
Finding: _Py_NewReference (119 sites), ob_type = field writes (~350 sites) —
architectural migration to PyObject_Init + Py_SET_TYPE; track as multi-sprint
initiative separate from individual bug fixes
────────────────────────────────────────
#: P11
Finding: pygcapi-compat not used; vendoring pythoncapi_compat.h would eliminate all
manual #if PY_VERSION_HEX guards for PyWeakref_GetRef, PyErr_GetRaisedException,
etc. and reduce future maintenance cost
────────────────────────────────────────
#: P12
Finding: g_obPyCom_MapRecordGUIDToRecordClass stores user-defined Python subclasses
in
a process-wide dict — unlike the IID maps (intentionally process-wide), this dict
is
interpreter-specific and should be separated once multi-phase init is adopted
---
Strengths
- CI matrix is excellent: 3.9–3.15 × x86/x64/ARM64 × GIL/free-threaded covers the
widest supported range in pywin32's history. The free-threaded variants are already
in CI.
- Version guard discipline: Zero unguarded post-3.9 API calls found. Every
PyWeakref_GetRef, PyUnstable_Module_SetGIL, and pre-3.11 frame access is correctly
conditionalized.
- PY_INTERFACE_PRECALL/POSTCALL macro pair: The GIL release/acquire pattern around
COM calls is consistent and used correctly in most COM dispatch paths.
- PYWIN_MODULE_INIT_PREPARE macro: Centralizes the PyUnstable_Module_SetGIL call
under #ifdef Py_GIL_DISABLED in one place — if the API is promoted to a stable name,
it's a one-file change.
- CEnterLeavePython RAII: Correct pattern for non-Python-thread callbacks; the
existing model is sound for GIL builds.
- COM interface reference management: PyCom_PyObjectFromIUnknown correctly uses
POFIU_RELEASE_ON_FAILURE to handle COM interface ownership on error paths — clean in
all surveyed paths.
- Recidivism awareness: The codebase has .git history showing 20+ years of the same
bug class in Register.cpp being fixed. The authors have been diligent about fixing
reported issues; the remaining bugs are gaps in coverage, not willful neglect.
---
Code Removal Opportunities
With minimum Python ≥3.9:
- AXDebug.cpp pre-3.11 branch (#if PY_VERSION_HEX < 0x030b0000): PyFrame_GetCode and
PyFrame_GetBack are available since 3.9. Delete the entire #else struct-access block
(~15 lines) and the #include "compile.h" / #include "frameobject.h" guards. Net: ~20
lines removed.
- PyErr_Fetch/Restore migration (once pythoncapi-compat is vendored): Each of the 31
call sites collapses from a 3-variable save/normalize/restore pattern to a 1-variable
save/restore. Net: ~3 lines saved per site → ~60 lines removed.
- structmember.h removal (once min Python ≥3.12): Remove the #include
"structmember.h" from PyWinTypes.h:18; rename T_INT → Py_T_INT etc. in ~50 files.
Net: clean removal of the deprecated header and ~41 macro rename sites.
- _Py_NewReference → PyObject_Init (long-term): After the COM type hierarchy is
refactored to use tp_alloc, the _Py_NewReference + ob_type = constructor pattern in
48 files can be replaced with PyObject_Init(this, &type). Net: 119 + 350 = ~469 line
edits, eliminates the private API entirely.
---
Recommended Action Plan
Immediate (FIX — all-builds correctness; safe to submit independently)
1. Fix finding 1 (Register.cpp:232): Change Py_DECREF(v) to nothing — v is borrowed;
the dict owns it. Confirm no Py_INCREF of v precedes this line.
2. Fix finding 2 (PyIPropertyStorage.cpp:451-458): Add braces to the if
(!PyObject_AsPROPVARIANT(...)) block so return FALSE is inside it, not the loop body.
3. Fix finding 5 (win32trace.cpp:277-295): Propagate the raw Win32 error code out of
the mutex helpers and call PyWin_SetAPIError only after Py_END_ALLOW_THREADS.
4. Fix findings 6–7 (win32apimodule.cpp): Add return before each ReturnAPIError call
in LoadCursor and CommandLineToArgvW.
5. Fix finding 3 (win32gui.i:30-59): After calling Python logger, restore then clear
all 3 Py_XDECREF-freed exception objects; or migrate to
PyErr_GetRaisedException/PyErr_SetRaisedException pattern.
6. Fix finding 4 (MappingManager.cpp:136-141): Add goto done after PyErr_SetString
for the NULL allocation case.
7. Fix findings 8–9 (PyHANDLE.cpp, PyOVERLAPPED.cpp): Make hashFunc derive the hash
from m_handle / struct contents to agree with richcompare.
8. Fix findings 10–11 (PyHANDLE.cpp, PyDEVMODE.cpp): Add tp_new that allocates via
type->tp_alloc(type, 0) and call Py_TYPE(ob)->tp_free(ob) in tp_dealloc.
9. Fix findings 14–16 (PyGatewayBase.cpp, univgw.cpp): Replace unconditional
PyErr_Clear() with PyErr_ExceptionMatches(PyExc_MemoryError) check → return
E_OUTOFMEMORY before clearing. Also add NULL check for PyTuple_New(cNames) at line
257 (finding 34).
10. Fix findings 23–25 (mmapfilemodule.cpp, win32clipboardmodule.cpp ×2): Add
Py_DECREF(m_obj) and GlobalFree(handle) / GlobalFree(hMem) on failure paths.
11. Fix findings 26–30 (module globals): Add if (x == NULL) { x = ...; } idempotency
guards; add Py_INCREF(&PyRecord::Type) before PyModule_AddObject.
12. Fix findings 31–33 (PyComHelpers.cpp:274,381,465): Remove Py_DECREF on borrowed
dict-get results; replace pwcsName = NULL with CoTaskMemFree(pStat->pwcsName);
pStat->pwcsName = NULL.
Short-term (FIX Tier B + C + selected CONSIDER)
13. Findings 18–22 (unchecked PyTuple_New): Add NULL checks + return NULL / DECREF
cleanup after every PyTuple_New call.
14. Finding 12 (PythonEng.cpp): Replace PyObject_RichCompare + pointer-compare with
PyObject_RichCompareBool.
15. Finding 13 (Register.cpp:97-106): Add Py_DECREF(valueObject); return E_FAIL; on
PyDict_SetItem failure.
16. Finding 35 (incomplete PyCom_RegisterGatewayObject guard): Add if
(g_obPyCom_MapServerIIDToGateway == NULL) return E_FAIL;.
17. Findings 39–41 (ISAPI PyType_Ready, GC traverse/clear for PyOVERLAPPED, COM
hash): Low individual cost, high correctness value for IOCP-heavy code.
Medium-term (POLICY — free-threading correctness)
18. P2–P4 (console handlers, RAS map, ISAPI g_callbackMap): Add PyMutex +
Py_BEGIN_CRITICAL_SECTION to protect each shared dict accessed from OS callback
threads.
19. P5–P6 (lazy-init singletons with "GIL prevents races" comments): Replace with
PyMutex-guarded or _Py_once_flag initialization.
20. P7–P8 (missing Py_BEGIN_ALLOW_THREADS in win32credmodule.cpp, win32security.i):
Apply pattern from existing CredWrite fix to the 9 remaining Cred* functions and 3
LookupPrivilege* functions.
Longer-term (POLICY — migration)
21. P1 (31 PyErr_Fetch/Restore sites): Vendor pythoncapi_compat.h; migrate all 8
files to PyErr_GetRaisedException/PyErr_SetRaisedException.
22. P10 (_Py_NewReference + ob_type = writes): Plan multi-sprint refactor of PyIBase
and COM type hierarchy to tp_alloc/PyObject_Init.
23. P9 (blocking Win32 calls without GIL release): Systematically audit remaining
non-Py_BEGIN_ALLOW_THREADS Win32 waits.
24. Remove the #if PY_VERSION_HEX < 0x030b0000 block in AXDebug.cpp (20-line cleanup,
zero risk at 3.9 min).

C Extension Analysis Report

Extension: refinery native Cython modules (16 .pyx files, ~6,329 lines, 3 directories)

Scope: refinery/lib/fast/, refinery/lib/seven/, refinery/lib/unrar/

Agents Run: none — manual review (see Methodology note)

Methodology note

Phase 0 discovery (discover_extension.py) reported "extensions": [] and "total_c_files": 0 for this repo. That is a false negative, not the ground truth: setup.py:88-89 builds its extension list as Extension(key, [value]) for key, value in EXTENSION_DICT.items(), and the discovery script's regex only matches Extension("literal_name", [...]) — a quoted string literal as the first argument. Because refinery constructs its extensions from a dict via a list comprehension, the regex finds nothing, even though 16 real .pyx modules exist (confirmed directly, and already fully catalogued in this repo's prior ft_review.md). The script's separate code_generation detector (which globs for *.pyx directly, independent of the broken Extension() regex) correctly reported "code_generation": "cython" despite the empty extension list — that inconsistency is itself the tell.

Phase 2's five C-grammar agents (refcount-auditor, error-path-analyzer, null-safety-scanner, pyerr-clear-auditor, gil-discipline-checker's scan_gil_usage.py, module-state-checker, type-slot-checker, stable-abi-checker, version-compat-scanner) all import tree_sitter_c and operate on .c/.h source. Since total_c_files == 0 here (no hand-written or Cython-generated C is checked into this repo — the .pyx files are compiled at install time, not committed as generated C), every one of those agents is structurally inapplicable, independent of any tooling failure. This is a correct skip, not a workaround.

This toolkit also ships five Cython-native scanners (scan_cython_cdef_int_except.py, scan_cython_cinit_candidates.py, scan_cython_buffer_protocol.py, scan_cython_nogil_pyobject.py, scan_cython_pycapsule.py) that parse .pyx directly via tree-sitter-cython, which is the genuinely applicable part of this toolkit for refinery. Attempted to install it:

pip install tree-sitter && pip install git+https://github.com/devdanzin/tree-sitter-cython.git@fix/grammar-gaps

tree-sitter/tree-sitter-c were already present under python3t.exe (CPython 3.15.0b2 free-threaded), but tree-sitter-cython's C binding failed to build against this interpreter: the build combines -DPy_LIMITED_API=0x03080000 with -DPy_GIL_DISABLED=1, and under that combination object.h leaves PyModuleDef undefined, so binding.c fails with C2079: 'module' uses undefined struct 'PyModuleDef'. This is the same class of build hostility this environment showed earlier for pydantic-core's Rust/maturin tooling (see memory pydantic-core-rust-review) — a limited-API + free-threaded-beta combination that the installed MSVC toolchain and this Python build don't reconcile. Retrying with a different wheel or interpreter would not change this; the fix needs to come from the tree-sitter-cython packaging side.

Given 16 files is well within hand-reading range, the five Cython-native scanners' detection logic (read directly from source, not run) was applied manually as a checklist against all 16 .pyx files, using Grep/Read rather than the AST tooling. This is slower than running a working scanner but not less thorough at this file count.

Safety constraint observed: per this repo's CLAUDE.md, many test samples here are malware. This review is static and read-only — no sample was executed, no refinery code was imported or run, only the toolkit's own Python scripts (none of which touch refinery code or samples) were executed.

Relationship to the prior FT review

This repo already has reports/ft_review.md (free-threading thread-safety review: 0 RACE/UNSAFE/PROTECT, 1 MIGRATE on the missing freethreading_compatible declaration). That review already covers GIL discipline (nogil-block correctness) and module-level shared-state safety in detail — this report does not re-derive those findings; see ft_review.md directly. This report covers the orthogonal Cython-correctness axis: exception-clause correctness on cdef functions, __cinit__/__init__/__dealloc__ reinit-leak shape, buffer-protocol pairing, and PyCapsule usage — none of which the FT review checked, and none of which is implied by "no shared mutable state."

Extension Profile

  • Module: refinery native acceleration modules (16 .pyx files across lib/fast/, lib/seven/, lib/unrar/)
  • Init style: standard def/cpdef module functions; no custom PyInit_* (Cython-generated)
  • Python targets: not pinned (pyproject.toml has no requires-python)
  • Limited API: no
  • Cython version: unpinned in pyproject.toml (requires = ["cython", ...], no floor)
  • Types defined: 8 cdef class types (DecoderLevels, HuffmanDecoder, HuffmanDecoder7b, BitDecoder, BitInput, DecodeTable, BlockHeader, BlockTables)
  • Type stubs: one .pyi file repo-wide, refinery/__init__.pyi — a package-root re-export list for refinery.units.*, unrelated to these 16 extension modules. Parity-checker's .pyi-vs-.pyx method-set comparison is moot here (no stub exists for the extension API surface).
  • Git history: shallow clone, single visible commit (707cd50) — git-history-analyzer's churn/regression-date method is moot; no temporal signal available.

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety N/A - - No C source (total_c_files=0); skip is correct, not a gap
Error Handling N/A - - Same
NULL Safety N/A - - Same
GIL Discipline G 0 0 Already reviewed in ft_review.md; cross-referenced, not re-derived
Resource Lifecycle G 0 0 No raw C-pointer fields in any cdef class; no __dealloc__ exists anywhere because none is needed
Module State G 0 0 Already reviewed in ft_review.md (3 read-only import-time tables, no mutable module state)
Cython cdef-int except-clause Y 0 1 5 functions in unpack50.pyx lack an explicit except/noexcept on int/bint returns — exceptions already propagate under Cython >=3.0's default; this is a minor sentinel-check/clarity nit, not a swallow risk
__cinit__/__init__/__dealloc__ shape G 0 0 No __cinit__ and no __dealloc__ exist anywhere in the codebase — no raw-pointer-owning class exists for the leak shape to apply to
Buffer Protocol G 0 0 No __getbuffer__/__releasebuffer__/PyObject_GetBuffer anywhere; all buffer access goes through Cython-managed typed memoryviews
PyCapsule G 0 0 No PyCapsule_New/PyCapsule_GetPointer usage anywhere
ABI Compliance N/A - - No limited API claimed; not assessed
Version Compat N/A - - C-grammar-only scanner; not applicable to .pyx
Parity G 0 0 Only .pyi in the repo is the package-root re-export list, unrelated to these modules
Complexity N/A - - C-grammar-only scanner (measure_c_complexity.py); not run against .pyx

G = No FIX findings | Y = 1-3 FIX/CONSIDER findings | N/A = scanner's input type doesn't exist in this repo

Findings by Priority

Must Fix (FIX) — 0

None.

Should Consider (CONSIDER) — 1

# Finding File:Line
1 Five cdef functions in unpack50.pyx — _decode_number_fast (177), _slot_to_length (233), _read_block_header (289), _read_tables (335), _read_filter_data (416) — return int/bint with no explicit except/except */except -1/noexcept clause. Verified via context7 against Cython's own migration docs that this is not a swallow risk: Cython >=3.0 changed the default so that cdef functions propagate Python exceptions unless explicitly marked noexcept (the pre-3.0 silent-swallow behavior is now opt-in via the legacy_implicit_noexcept directive, confirmed absent from this repo — grepped setup.py, pyproject.toml, and every .pyx header). So today, if any of these five functions ever did raise, the exception would propagate normally to the caller, not vanish. The only real effect of the missing clause is that Cython inserts an extra PyErr_Occurred() check after every call site (the mechanism it uses to disambiguate "returned -1" from "raised and returned -1") — a minor, not security-relevant, runtime cost. Adding except -1 (or except * for the two bint returns) would be a clarity/micro-perf nit, not a correctness fix, and the manual trace also found no exception is currently reachable from any of the five bodies anyway. refinery/lib/unrar/unpack50.pyx:177,233,289,335,416

Tensions

None — no agent disagreements; the C-grammar agents simply don't apply (no C source), so there is nothing to reconcile against the Cython-native checks.

Policy Decisions (POLICY) — 0

None.

Strengths

  • No __dealloc__ anywhere, and that's correct, not an oversight. Confirmed via grep across all 16 files: zero __dealloc__, zero __cinit__, and zero raw-pointer (cdef T*) fields inside any cdef class body (the only raw pointers found anywhere are function-local stack variables inside the lib/fast/ realloc helpers, already covered as safe in ft_review.md). Every stateful cdef class (HuffmanDecoder, HuffmanDecoder7b, BitDecoder, DecoderLevels, BitInput, DecodeTable, BlockHeader, BlockTables) holds only Python-managed fields — array.array, bytearray, typed memoryviews, fixed-size C arrays embedded by value in the struct, or references to other cdef class instances. Cython's automatic refcounting/cleanup on tp_dealloc already handles all of these correctly with zero hand-written code, which is the cinit/dealloc-leak scanner's "Shape A/B" risk having no surface to land on.
  • No buffer-protocol or PyCapsule usage at all. All raw-buffer access goes through typed memoryviews (const uint8_t[::1]) and bytearray/array.array, which Cython itself manages acquire/release for — there is no hand-rolled PyObject_GetBuffer/PyBuffer_Release pairing to get wrong, and no PyCapsule_New lifetime to audit.
  • The noexcept clauses that do exist are used correctly. lib/fast/lzfse.pyx's six noexcept nogil helper functions (_decode_v2_freq_table, _decode_lzfse_payload, _decode_lzfse_v2_block, _decode_lzfse_v1_block, _lzvn_decode, _decode_lzvn_block) signal failure via an int sentinel (written < 0) checked explicitly by their caller (lzfse_decompress, lines 942/943), rather than via a Python exception — manually traced and confirmed none of the six touches a Python object or can raise, so the noexcept annotation is accurate, not a footgun. lib/fast/xpress.pyx's and lib/fast/aplib.pyx's allocation helpers correctly pair nogil with explicit except -1 and with gil: raise MemoryError — already covered in ft_review.md, repeated here only to note it as evidence the project's authors are already applying this discipline correctly in most places; Finding 1's gap in unpack50.pyx is the exception, not the norm.
  • Top-level def functions correctly avoid the nogil/exception interaction entirely. Every function with a body full of bare raise statements (e.g. lib/fast/a3x.pyx's many raise EOFError/raise ValueError sites) is a plain def with no nogil anywhere in its body, so the GIL is held throughout and there is no swallowing risk — confirmed by checking each raise site's enclosing function signature.

Recommended Action Plan

Short-term (CONSIDER)

  1. Finding 1 — optionally add an explicit except clause to the five unpack50.pyx functions listed above (except -1 for the three int-returning functions, except -1/except * for the two bint-returning ones). Exceptions already propagate correctly today under Cython >=3.0's default, so this is purely a documentation-of-intent and minor-performance change (skips a redundant PyErr_Occurred() check per call), not a bug fix. Low priority.

Longer-term (POLICY)

None — the toolkit-mismatch issues (broken Extension() detection regex, tree-sitter-cython build incompatibility with this machine's free-threaded beta interpreter) are environment/tooling observations, not decisions about this codebase.

C Extension Analysis Report — scipy

Extension: scipy

Scope: entire scipy/ tree — first-party hand-written C/C++/Cython extension glue, ~20 extension module groups, 61 Cython .pyx files. Vendored bodies excluded (SuperLU/ARPACK/PROPACK library cores, biasedurn C++ body, fast_matrix_market/ header library, subprojects/) — see preflight/project_profile.md.

Agents Run: refcount-auditor · error-path-analyzer · null-safety-scanner · gil-discipline-checker · resource-lifecycle-checker · module-state-checker · type-slot-checker · pyerr-clear-auditor · stable-abi-checker · version-compat-scanner · c-complexity-analyzer · git-history-analyzer (12 of 13 — parity-checker skipped, no dual C/Python implementation pattern in scipy)

Related prior review: reports/ft_v1/final_report.md — a free-threading/thread-safety review of the same codebase (13 numbered findings, 4 RACE). This report is complementary: it covers correctness bugs unrelated to concurrency (refcounting, error paths, NULL safety, type-slot partial-init, module state, ABI/version compat), cross-referencing ft_v1 throughout rather than re-deriving its findings.


Executive Summary

scipy's hand-written C/C++/Cython extension surface is solid in its core numerical kernels (the translated-Fortran LSODA/VODE/FITPACK/MINPACK bodies are untouched by any finding across 12 agents) but carries 17 confirmed FIX-severity bugs concentrated in the thinner glue layer around it — Python-callback bridges, module-init boilerplate, and type constructors. None require an architectural rewrite; every fix is a small, local, idiomatic change. The most severe: a traced, reproducible deadlock in SuperLU's malloc hook reachable on the default splu/spilu factorization path (gil-discipline-checker Finding 1); a heap-corrupting double-free in scipy.linalg.interpolative's rank-estimation code (resource-lifecycle-checker Finding 1); two crash-on-trivial-use bugs in SuperLU()'s bare constructor and _Qhull.__new__ bypass (type-slot-checker Findings 1–2); and two Fortran-bridge modules (PROPACK, _dop) that silently swallow exceptions raised inside user callbacks — a bug their own sibling modules (_odepackmodule.c, _dzvodemodule.c) already fix correctly, making this a backport, not a new investigation (git-history-analyzer §2). A cross-cutting methodological finding matters as much as any single bug: tree-sitter-cpp does not build in this environment, so every AST-based scanner in this toolkit silently degraded to C-only coverage, missing all 63 in-scope C++ files; roughly half of this run's FIX findings were only found by the agents' manual grep/read fallback, meaning the "0 findings" scanner output for C++ carries no evidentiary weight and future runs need the tooling gap fixed before trusting coverage claims at face value.

A second headline result: code complexity does not predict where these bugs live. c-complexity-analyzer cross-referenced all 17 FIX findings against complexity scores and found 14 of them sit in functions scoring at or near the scanner's complexity floor (simple, short, easy to read) — they are pattern-consistency omissions (a check present in a sibling module but missing here), not reasoning failures induced by tangled code. The one genuine complex/buggy function, direct_direct_ (DIRECT optimizer, cyclomatic complexity 89), is also the one place git-history-analyzer found the maintainers have already made four separate fix attempts in the last seven months without landing on the exact flagged lines — the code is under active, informed attention and the fix is imminent by momentum, not something this report is surfacing cold.

Stable-ABI/limited-API adoption is not feasible project-wide (stable-abi-checker): three independent, non-overlapping blockers (pybind11's lack of abi3 support for 5 module groups, hand-rolled static PyTypeObjects with GC/dict participation backing public API, and NumPy's own lack of limited-API certification) rule it out regardless of any code-level fix. Version compatibility is otherwise clean: zero unguarded new-API usage, zero removed-API usage, a >=3.12 floor that is honestly enforced (version-compat-scanner).

  • Readiness: Y (Yellow) — no CRITICAL/security-class findings, but 17 FIX-severity bugs including one traced deadlock and one heap-corrupting double-free
  • FIX findings: 17
  • CONSIDER findings: 24
  • POLICY findings: 11

Extension Profile

  • Module groups: ~20 independent extension entry points (_fpumode, _uarray, _test_ccallback, _dop/_odepack/_vode, _dierckx/_fitpack, _ctest/_rank_filter_1d, _direct/_lbfgsb/_minpack/_slsqplib/_zeros, _sigtools/_spline, _superlu, _arpacklib, _propack, _sparsetools, _distance_wrap, special) + 61 Cython .pyx modules
  • Size: 264 .c, 134 .h, ~52–63 C++ (.cpp/.cc/.cxx/.hpp), 61 .pyx, 31 .pxd, 6 .pxi; 1,407 functions AST-scanned (C/H only) + manual C++/Cython coverage
  • Init style: 27 of 29 hand-written C/C++ entry points use modern multi-phase init (Py_mod_exec slots); 2 holdouts on legacy single-phase (quadpack, _uarray)
  • Python targets: >=3.12 (declared and effective — identical, no drift either direction)
  • Limited API: No — not claimed, and not feasible project-wide (see Finding 45)
  • Code generation: Mixed — hand-written C, hand-written C++, hand-written Cython all present; no generated code checked into the repo
  • Types defined: 6 hand-written PyTypeObjects (2 in C, 4 in C++) + 28 Cython cdef class declarations across 11 .pyx files

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 3 1 201-site ufunc reference leak via PyModule_AddObjectRef (special/_special_ufuncs.cpp/_gufuncs.cpp)
Error Handling R 5 2 PROPACK/_dop silently discard exceptions raised inside user callbacks
NULL Safety Y 2 1 Unchecked PyTuple_New/PyFloat_FromDouble crash six Fortran-bridge/DIRECT callback thunks on OOM
GIL Discipline Y 2 0 Traced deadlock: leaked PyGILState-equivalent hold on SuperLU's default factorization path
Resource Lifecycle Y 3 2 Partial PyMem_Realloc failure causes heap-corrupting double-free in scipy.linalg.interpolative
Module State G 0 6 10 Fortran-bridge modules falsely declare Py_MOD_PER_INTERPRETER_GIL_SUPPORTED
Type Slots Y 2 5 Bare SuperLU()/_Qhull.__new__() bypass crash — recurring pattern across 4 types, 3 languages
PyErr_Clear Safety G 0 2 Clean — 3 in-scope PyErr_Clear() sites total, none FIX-severity
ABI Compliance G 0 0 Not feasible project-wide (POLICY verdict, 3 independent blockers)
Version Compat G 0 5 Zero unguarded new-API usage; PyModule_AddObject soft-deprecation is the largest cluster (15 sites)
C/Python Parity — — — Not applicable — skipped, no dual-implementation pattern
Complexity G 0 0† Complexity does not predict defect location (14/17 FIX bugs in near-floor-complexity functions)

G = No FIX findings · Y = 1–3 FIX findings · R = 4+ FIX findings

† c-complexity-analyzer's three complexity-compounds-fix-risk observations (direct_direct_, Function::call, for_each_backend_in_domain) are deliberately folded into Findings 10, 19, and ft_v1's Finding 3 rather than counted as separate findings here — see Tensions.


Findings by Priority

Global non-restarting numbering: FIX (1–17), CONSIDER (18–41), POLICY (42–52).

Must Fix (FIX) — 17

# Finding File:Line Agent(s)
1 superlu_python_module_malloc leaks a PyGILState-equivalent hold (NPY_ALLOW_C_API) on the get_tls_global()==NULL early-return path — traced to a reproducible deadlock via SuperLU's SLU_BEGIN_THREADS/longjmp interaction, reachable on the default (non-Fact='SamePattern') splu/spilu path under OOM scipy/sparse/linalg/_dsolve/_superlu_utils.c:96-101 gil-discipline-checker
2 _Qhull.__new__(_Qhull) bypass leaves self._lock NULL (no __cinit__, only __init__); __dealloc__ unconditionally calls PyThread_acquire_lock(NULL, 0) — crashes automatically the moment any bypass-constructed instance is garbage-collected, no further call needed scipy/spatial/_qhull.pyx:254,267,359-364,371-392 type-slot-checker
3 Bare SuperLU() (the type's own public, zero-arg constructor — no bypass trick needed) creates a zeroed instance; .nnz getter unconditionally NULL-dereferences L.Store/U.Store scipy/sparse/linalg/_dsolve/_superluobject.c:147-150,228-276 type-slot-checker
4 Partial PyMem_Realloc failure (one of two parallel buffers succeeds, the other fails) leaves a stale pointer that is then double-freed (heap corruption) while the successfully-reallocated buffer leaks scipy/linalg/_decomp_interpolative.pyx:339-355 (idd_findrank), :1296-1312 (idz_findrank) resource-lifecycle-checker
5 cread_fieldnames leaks an unchecked-NULL calloc buffer on any exception (e.g. non-UTF-8 field name) while parsing MATLAB struct field names — reachable from scipy.io.loadmat() on malformed/adversarial files, no memory pressure required scipy/io/matlab/_mio5_utils.pyx:905-941 resource-lifecycle-checker
6 PROPACK's 8 SVD wrapper functions silently discard exceptions raised inside the user-supplied aprod (matvec) callback — no PyErr_Occurred() check after the Fortran-derived core returns; sibling modules (_odepackmodule.c, _dzvodemodule.c) already implement this check correctly scipy/sparse/linalg/_propack/_propackmodule.c (8 wrapper fns) error-path-analyzer
7 _dop (dopri5/dopri853) wrappers silently discard exceptions raised inside fcn/solout Python callbacks — same gap as #6, same fix pattern already proven in sibling modules scipy/integrate/_dopmodule.c:25-319 error-path-analyzer
8 Unchecked PyTuple_New/PyTuple_Pack container feeds PyTuple_SET_ITEM/PyTuple_SetItem and a later unconditional Py_DECREF — crashes on OOM in 4 _dzvodemodule.c callback thunks (12 sites) and 2 _dopmodule.c wrap functions (4 sites); the third sibling, _odepackmodule.c, gets this right scipy/integrate/_dzvodemodule.c (4 fns), scipy/integrate/_dopmodule.c (2 fns) null-safety-scanner
9 Unchecked PyFloat_FromDouble scalar stored via PyTuple_SetItem/PyList_SetItem leaves a live NULL slot in an otherwise-valid container, crashing on the next read (immediate in practice, not a remote future access) _dopmodule.c:49-50,94-95, _odepackmodule.c (8 sites), DIRsubrout.c:1087, DIRect.c:807 null-safety-scanner
10 DIRECT optimizer's Python-callback bridge: unchecked Py_BuildValue result fed directly to Py_DECREF (not Py_XDECREF) — crashes on OOM instead of propagating MemoryError scipy/optimize/_direct/DIRect.c:777-779, DIRsubrout.c:1089-1096 error-path-analyzer
11 Unchecked PyList_New result (x_seq) passed into the DIRECT C core, later unconditionally dereferenced via PyList_SetItem inside the callback bridge scipy/optimize/_directmodule.c:43 error-path-analyzer
12 Systematic reference leak: every one of ~201 ufunc registrations via PyModule_AddObjectRef (a non-stealing API) is missing the required Py_DECREF — leaks one PyUFuncObject per registration per sub-interpreter import/reload; both modules declare multi-interpreter support, so this compounds under sub-interpreter or repeated-import workloads scipy/special/_special_ufuncs.cpp (191 sites), scipy/special/_gufuncs.cpp (10 sites) refcount-auditor
13 superlu_python_module_free leaks the raw SuperLU buffer and crashes (PyDict_DelItem(dict, NULL) → hash-of-NULL, plus a subsequent Py_DECREF(NULL)) if PyLong_FromVoidPtr fails under OOM — on the hot deallocation path of every SuperLU factorization scipy/sparse/linalg/_dsolve/_superlu_utils.c:128-157 resource-lifecycle-checker
14 LU_to_csc's unit-diagonal-in-L bounds check is the only one of four structurally identical nnz-overflow guards that return -1 without calling PyErr_SetString first — surfaces as an opaque SystemError instead of the intended RuntimeError when accessing .L/.U on an inconsistent factorization scipy/sparse/linalg/_dsolve/_superluobject.c:648-649 (root cause), surfaced via SuperLU_getter:193 error-path-analyzer
15 Py_BuildValue("NNO", …, o) uses the borrowing 'O' format code (not stealing 'N') for a freshly-created, solely-owned dict — leaks one dict per call; every other Py_BuildValue site in the same file correctly uses 'N' scipy/interpolate/src/_fitpackmodule.c:2258 (fitpack_parcur, backs scipy.interpolate.splprep) refcount-auditor
16 Double-DECREF of arVi when PyArray_SetBaseObject fails: NumPy's documented contract is that this call steals the reference unconditionally including on failure, but the local arVi variable isn't NULLed on the failure branch before the shared fail: cleanup label re-DECREFs it scipy/signal/_lfilter.cc:260-277 (interacting with fail: at 312-319), in scipy_signal__sigtools_linear_filter (backs scipy.signal.lfilter) refcount-auditor
17 UNURAN_THUNK macro (expanded into 6 thunk functions) leaks a PyGILState_Ensure() hold on the PyErr_Occurred() early-return path, before the matching PyGILState_Release(). Currently latent — UNU.RAN sampling never releases the GIL today, so the skipped release is presently a no-op — but becomes a live GIL-leak/deadlock the moment any future with nogil: optimization is added to the sampling loop scipy/stats/_unuran/unuran_callback.h:5-8 (def), call sites 115,120,125,130,135,140 gil-discipline-checker

Should Consider (CONSIDER) — 24

# Finding File:Line Agent(s)
18 get_tls_global's unchecked PyDict_SetItemString + unconditional Py_DECREF(obj) can, under OOM, return a use-after-free pointer to its 3 callers scipy/sparse/linalg/_dsolve/_superlu_utils.c:36-50 refcount-auditor
19 uarray._Function's tp_new/tp_init split leaves py_ref members null after a __new__ bypass; Py_INCREF (not Py_XINCREF) on those null members in 3 getters crashes, and the same root cause reaches tp_call's canonicalize_args (PyTuple_GET_SIZE(nullptr)) — 4 crash sites from one uninitialized-construction gap scipy/_lib/_uarray/_uarray_dispatch.cxx:1094-1178,1254-1256,1452-1465,1652 refcount-auditor, type-slot-checker
20 droprule_cvt's goto fail (unreachable in practice — str.split()/bytes.split() always return a sequence) would return a converter-failure sentinel with no exception set if ever reached scipy/sparse/linalg/_dsolve/_superluobject.c:993-1002 error-path-analyzer
21 call_thunk's unified cleanup path calls PyArray_ResolveWritebackIfCopy without checking its return — a failure there after an already-successful compute could return a non-NULL result with a pending exception (SystemError) scipy/sparse/sparsetools/sparsetools.cxx:439-462 error-path-analyzer
22 PyDict_GetItem (missing-key/error-ambiguous) used for kwarg-default pruning — correctly NULL-guarded against a crash, but could silently swallow a pathological __hash__/__eq__ exception (practically unreachable given CPython's calling convention) scipy/_lib/_uarray/_uarray_dispatch.cxx:1181-1194 null-safety-scanner
23 SmallDynamicArray move constructor references a non-existent union member (storage_.size) — confirmed build-breaking by test-compilation, but the move constructor is never odr-used by current callers, so this is currently inert dead code scipy/_lib/_uarray/small_dynamic_array.h:119 resource-lifecycle-checker
24 Unchecked PyCapsule_New leaks a preceding raw buffer on its own OOM failure — one instance in test-only code (POLICY-tier, see #48), one in the public LowLevelCallable capsule-construction path (CONSIDER-tier) scipy/_lib/src/_test_ccallback.c:354-369, scipy/_lib/_ccallback_c.pyx:78-86 resource-lifecycle-checker
25 uarray._BackendState transitively owns arbitrary backend PyObject* references (via STL-container-embedded py_refs) but has no Py_TPFLAGS_HAVE_GC/tp_traverse/tp_clear — a reference cycle through a captured get_state() snapshot would be permanently uncollectable, invisible to cycle GC scipy/_lib/_uarray/_uarray_dispatch.cxx:308-328,1480-1519 type-slot-checker
26 coo_entries/ordered_pairs (ckdtree result helpers): .buf is NULL after a __new__ bypass (allocated in __init__, not __cinit__); .ndarray()/.dict() dereference the null vector<coo_entry>* unconditionally scipy/spatial/_ckdtree.pyx:159-221,245-284 type-slot-checker
27 MessageStream.close()/__dealloc__: on fdopen() failure in the tempfile fallback path, a bookkeeping flag (self._removed) isn't updated after a successful inline cleanup, causing __dealloc__ to attempt removing the same file twice — raises a second, ignored exception (printed to stderr) that obscures the real OSError scipy/_lib/messagestream.pyx:28-43,81-92 type-slot-checker
28 quadpack is the last Fortran-bridge module on legacy single-phase init (m_size=-1) — structurally cannot declare Py_mod_multiple_interpreters at all (which, per CPython's safe-subinterpreter policy, means it's correctly refused entry rather than silently misbehaving — "safer than its modernized siblings," not strictly worse) scipy/integrate/__quadpack.h:953-995 module-state-checker
29 10 modernized Fortran-bridge modules declare Py_mod_multiple_interpreters = PER_INTERPRETER_GIL_SUPPORTED while their *_error exception objects live in unisolated file-scope C globals (m_size=0, no real per-module state) — a false declaration that CPython's isolating-extensions model acts on; reachable specifically under own-GIL subinterpreter use (interpreters.create()) _dopmodule.c, _dzvodemodule.c, _odepackmodule.c, _fitpackmodule.c, _matfuncsmodule.c, _lbfgsbmodule.c, _minpackmodule.c, _slsqpmodule.c, _arpackmodule.c, _propackmodule.c module-state-checker
30 _uarray_dispatch.cxx: legacy single-phase init (like quadpack) backing public API (scipy.fft.set_global_backend), with a process-wide (not even thread-local) backend registry — no false declaration (fails closed, like quadpack), but the registry-partitioning gap is adjacent to the already-confirmed use-after-free race in the same structure (ft_v1 Finding 3) scipy/_lib/_uarray/_uarray_dispatch.cxx:157-161,1791-1852 module-state-checker
31 _uarray_dispatch.cxx: 5 PyModule_AddObject calls in PyInit__uarray with zero return-value checking (worse than the 9 Fortran-bridge modules, which check but don't null the static after) — an OOM here silently ships a module missing one of _Function/_SetBackendContext/_SkipBackendContext/_BackendState/BackendNotImplementedError scipy/_lib/_uarray/_uarray_dispatch.cxx:1811-1842 module-state-checker, version-compat-scanner
32 6 legacy PyTypeObject globals with non-static external linkage in the SuperLU wrapper (SuperLUType, SuperLUGlobalType) — a step worse than the 4 internally-linked _uarray equivalents for module-state hygiene; blocks any future heap-type/subinterpreter migration _superlu_utils.c:176, _superluobject.c:228, _uarray_dispatch.cxx:1480,1637,1688,1739 module-state-checker
33 SCIPY_TLS-based callback bridges and sf_error_actions are thread-local (correct for free-threading, per ft_v1) but not interpreter-local — a subinterpreter embedding pattern that reuses OS threads across interpreters could leak one interpreter's scipy.special.seterr() configuration into another's execution on the same thread scipy/special/sf_error.cc:11 (+ ft_v1-catalogued SCIPY_TLS globals) module-state-checker
34 sf_error_v clears an unguarded PyImport_ImportModule/PyObject_GetAttrString failure without a PyErr_ExceptionMatches guard — could in principle swallow MemoryError/KeyboardInterrupt rather than only the expected ImportError/AttributeError; low real risk since the module is necessarily already loaded scipy/special/sf_error.cc:88,104 pyerr-clear-auditor
35 _sigtools_convolve2d's complex-dtype-fallback PyErr_Clear() is unguarded, but every path through the enclosing branch terminates in an error return regardless — borderline ACCEPTABLE, kept as CONSIDER for auditability scipy/signal/_sigtoolsmodule.cc:704-721 pyerr-clear-auditor
36 PyModule_AddObject (soft-deprecated per CPython 3.13 What's New) used at 15 sites; PyModule_AddObjectRef (3.10+, already below the 3.12 floor) is a drop-in, no-guard-needed replacement 11 files — 9 Fortran-bridge exception globals + _batched_linalg_module.cc + BackendNotImplementedError in _uarray_dispatch.cxx version-compat-scanner
37 A third, previously-undocumented PyErr_Fetch/PyErr_Restore deprecated-API site (ft_v1 found two others) — a hand-written Cython cdef extern block, invisible to any C/C++ scanner scipy/stats/_unuran/unuran_wrapper.pyx:28-31, call sites 529/532, 576/579, 1532/1535 version-compat-scanner
38 PyErr_NormalizeException at the precise line ft_v1's Finding 2 referenced only in prose — same deprecated-API family as #37 scipy/_lib/_uarray/_uarray_dispatch.cxx:1286 version-compat-scanner
39 Dead Python-2-era vestiges: a PyCObject_Import declaration (removed from CPython since 3.2, never called) and a #define PyInt_AsLong PyLong_AsLong shim (never used) — both one-line, zero-risk deletions scipy/io/matlab/_streams.pyx:12, scipy/interpolate/src/_fitpackmodule.c:6 version-compat-scanner
40 9 sites use the discouraged (not formally deprecated) PyDict_GetItem/PyDict_GetItemString, which silently swallows any non-lookup exception from __hash__/__eq__; the strong-ref, error-checked PyDict_GetItemRef needs a 3.13 floor bump or pythoncapi-compat, but PyDict_GetItemWithError (3.0+) is a no-guard-needed intermediate improvement available today _arpackmodule.c (4), _uarray_dispatch.cxx (2), _slsqpmodule.c (2), _superlu_utils.c (1) version-compat-scanner
41 4 independently-discovered types across 3 languages (C: SuperLU; Cython: _Qhull, coo_entries/ordered_pairs; C++: uarray._Function) share one root pattern — constructible in a way (object.__new__ inheritance, Cython __new__ bypass, bare zero-arg call) that leaves a raw pointer/object member null, which a downstream slot or method then dereferences unconditionally. None uses Py_TPFLAGS_DISALLOW_INSTANTIATION (3.10+) or an equivalent guard Findings 3, 4, 17, 19, 26 (this table) type-slot-checker

Tensions

  • module-state-checker sharpens, and partially overturns, ft_v1's severity framing for the 11 *_error globals. ft_v1 rated these LOW/"hygiene, not correctness" based on single-interpreter safety alone. module-state-checker checked the actual Py_mod_multiple_interpreters slot value the 10 modernized modules declare and found they assert PER_INTERPRETER_GIL_SUPPORTED — a claim the unisolated globals don't back up. This raises the practical stakes from "someday, if subinterpreters happen" to "the declaration is actively false today" (Finding 29), while correctly keeping quadpack and _uarray's legacy single-phase modules at the lower end (they fail closed instead of lying) — the opposite of the naive "single-phase is strictly worse" framing ft_v1 implicitly used. Resolution: this report's Finding 29 supersedes ft_v1's characterization for the 10 modernized modules; ft_v1's framing stands unchanged for the 2 single-phase holdouts.
  • gil-discipline-checker overturns two of ft_v1's own SAFE-pattern claims. ft_v1's unsafe-api-detector asserted all cited PyGILState_Ensure/Release sites were correctly paired, based on literal-macro presence/purpose checking. This run's gil-discipline-checker traced macro aliases (NPY_ALLOW_C_API, invisible to a literal PyGILState_Ensure grep) and macro-expansion-hidden call sites (UNURAN_THUNK) through every exit path, finding two genuine leaks (Findings 1 and 17) that presence-checking missed. Both prior reports were doing the methodology they set out to do correctly; the newer pass simply applied a stricter bar. No action needed beyond noting this as the reason Findings 1 and 17 exist despite ft_v1's earlier "confirmed clean" note.
  • c-complexity-analyzer's negative correlation result is itself a finding, not just context. It explicitly recommends against using its own hotspot list as a bug-location proxy for this codebase and instead flags direct_direct_, Function::call, and for_each_backend_in_domain only as places where an already-known fix (Findings 10, 19, and ft_v1 Finding 3 respectively) is harder to land correctly because the surrounding function is also complex. This is folded into the Recommended Action Plan below rather than listed as independent findings, to avoid double-counting.
  • Scope question inherited from ft_v1, still open: scipy/_lib/_uarray/ is vendored-with-a-scipy-patch but resolved in-scope by ft_v1's synthesis (patched, lives directly under scipy/_lib/, backs public API). This report's _uarray-related findings (19, 25, 26, 30, 31, 32, 38) inherit that same scope resolution — flagged again here only so a reader who disagrees with ft_v1's call knows exactly which of this report's findings are affected.

Policy Decisions (POLICY) — 11

# Finding
42 9 Fortran-bridge modules leave their static PyObject *<name>_error global dangling (not NULL'd) after the correctly-handled PyModule_AddObject failure branch — individually correct steal-on-success handling, the residual gap is unreachable in practice (no code reads the global before a successful full module init)
43 Py_mod_gil slot (3.13+) vs. PyUnstable_Module_SetGIL fallback used inconsistently: 8 files carry both mechanisms (redundant, inert on official CPython builds), ~20 rely on the slot alone — a style/consistency call, not a functional gap on mainstream toolchains
44 scipy's 61 Cython-compiled modules correctly default to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED (empirically verified by compiling probe .pyx files) — a strength, not a defect: Cython's conservative default protects 6× the module count of the 10 C modules that make the false declaration in Finding 29
45 Stable-ABI/limited-API adoption is not feasible as a project-wide initiative. Three independent, non-overlapping blockers: (a) pybind11 (pinned >=2.13.2, no abi3 support) blocks 5 module groups outright; (b) 6 hand-rolled static PyTypeObjects with GC/tp_dictoffset participation — one backing public API (scipy.fft.set_global_backend) — require a heap-type rewrite with real correctness risk, not a mechanical fix; (c) NumPy's C API, depended on by ~40 files, has no official Py_LIMITED_API certification. One module (_fpumode.c) is already fully limited-API-clean, proving the codebase can write portable code — but a single module out of ~20 yields no practical wheel-reduction benefit. ~25 non-opaque PyTuple_GET_ITEM-class macro sites are mechanically fixable as general hygiene but don't unblock any module on their own (each is independently blocked by (b) or (c)).
46 Cython version is unbounded (>=3.2.0, no <4 ceiling) in pyproject.toml, meson.build, pixi.toml — a future Cython 4.0 would be picked up silently by any fresh build/CI run; directive usage itself is already 4.0-clean (no deprecated directives found)
47 pythoncapi-compat is a Medium adoption opportunity — verified (not assumed) via direct header fetch: backports PyModule_Add/PyModule_AddType/PyModule_AddObjectRef (would collapse Finding 12's fix and ~211 other call sites to one-line changes), but does not backport PyErr_GetRaisedException/SetRaisedException (zero help for Findings 37–38 and ft_v1's PyErr_Fetch findings, which don't need the header anyway since the 3.12 floor already has the native APIs unconditionally)
48 test_get_data_capsule's unchecked PyCapsule_New leak (8 bytes, OOM-only) and _test_ccallback.c's error_flag-without-exception path are both confined to scipy's internal test-support module, unreachable from any public/documented API — recorded for completeness, not actionable
49 _sigtools_remez's err < 0 dispatch and _sigtools_convolve2d's analogous switch are both non-exhaustive against their current callees' documented return codes — verified unreachable today (every actual return value is handled), but fragile against a future edit to the numerically-dense callee that introduces a new error code with no compiler warning; a one-line else/default catch-all is cheap insurance
50 Cross-cutting: 4 independently-discovered types (Finding 41) sharing the same "constructible-but-unguarded" pattern across C, C++, and Cython suggests a project-wide policy worth adopting — Py_TPFLAGS_DISALLOW_INSTANTIATION (or an equivalent Cython __cinit__ guard) as the default posture for any internal-only type not meant to be constructed directly from Python
51 Tooling gap, cross-cutting and consequential: tree-sitter-cpp does not build in this environment (confirmed independently by all 12 agents this run and by every ft_v1 agent). Every AST-based scanner in this toolkit silently falls back to .c/.h-only scanning, meaning all 63 in-scope C++ files receive zero automated coverage in any single run — this is not a scipy defect, but it means "0 findings" from any scanner-only pass on this codebase carries no evidentiary weight for its C++ half. Roughly half of this run's 17 FIX findings came from agents' manual fallback sweeps, not the scanners. Fix the toolkit's C++ build environment (vendor a prebuilt tree-sitter-cpp wheel, or patch the free-threaded-MSVC PyModuleDef compile error) before treating a future run's automated coverage claims at face value.
52 git-history-analyzer's environment note, worth codifying as toolkit guidance: git rev-list --count HEAD returning a large number is not sufficient evidence a clone is fully unshallowed — this run's repo showed 3,589 commits (a plausible-looking count) while git rev-parse --is-shallow-repository still returned true, corrupting blame/log output for any file predating the shallow boundary until git fetch --unshallow was re-run. Future runs should check --is-shallow-repository directly, not infer from commit count.

Strengths

  • The core numerical kernels are untouched by any finding. LSODA, VODE, FITPACK, MINPACK, TNC, and quadpack's translated-Fortran bodies (some of the highest raw-complexity functions in the tree, cyclomatic complexity up to 200) drew zero findings across 12 agents — the f2c-style translations are stable, heavily validated, and correctly left alone rather than "cleaned up."
  • sparsetools.cxx's call_thunk — the generic marshalling dispatcher every _sparsetools thunk routes through, ~369 lines of genuinely complex spec-string-driven dispatch — attracted four independent full reads (error-path-analyzer, null-safety-scanner, refcount-auditor, resource-lifecycle-checker) and yielded exactly one minor CONSIDER finding. This is the clearest counter-evidence in the whole run to "complexity implies bugs": disciplined complexity (one consolidated cleanup label, no duplicated error handling) shipped fewer bugs than several of this report's much simpler buggy functions.
  • _odepackmodule.c/_dzvodemodule.c correctly implement the callback-exception-check pattern that PROPACK/_dop (Findings 6–7) omit, and correctly NULL-check every PyTuple_New allocation that the sibling modules in Finding 8 don't — proof the correct pattern is well-understood and already proven in this exact codebase, making Findings 6–9 backports rather than novel designs.
  • _fpumode.c is a clean, minimal, fully modernized reference module: multi-phase init, correct Py_mod_multiple_interpreters declaration backed by real m_size=0 state, version-gated Py_mod_gil, zero limited-API violations. Worth using as the template for any future module-state or ABI migration.
  • Zero private (_Py) CPython API usage anywhere in first-party scope (verified by tree-wide regex sweep) — the only version-compat concerns are the already-catalogued deprecated-but-public PyErr_Fetch/PyErr_Restore family.
  • No buffer-protocol implementations, no custom tp_richcompare/__richcmp__ overrides anywhere in first-party scipy — two entire finding categories are structurally absent from this codebase's attack surface.
  • PyCapsule-based raw-buffer ownership transfer is a consistent, correctly-applied idiom across _matfuncsmodule.c, _slsqpmodule.c, and the DIRECT optimizer's unified cleanup:/fail: label pattern (13+ buffers funneled through one exit path, NULL-safe free() throughout).
  • The py_ref RAII wrapper in _uarray_dispatch.cxx (copy/move/steal/ref/reset) is textbook-correct and consistently applied — every finding in that file is about a call site that bypasses the wrapper's safety (raw Py_INCREF on .get()), not a flaw in the wrapper itself.

Code Removal Opportunities

Per version-compat-scanner's detailed table (see agents/version-compat-scanner.md for full derivation):

  • PyModule_Add family (via pythoncapi-compat, verified backported, or a future 3.13 floor): would collapse Finding 12's ~211 ufunc-registration sites from create+check+Py_DECREF to one call each — ≈200 lines removable, plus fixes Finding 12 as a side effect.
  • PyModule_AddType (3.9+, already below floor) replacing PyType_Ready+Py_INCREF+PyModule_AddObject for _uarray's 4 static types — ~8 lines, plus fixes the 4 unchecked-return defects in Finding 31.
  • PyModule_AddObjectRef (3.10+, already below floor) replacing PyModule_AddObject for the 11 remaining exception-object sites (Finding 36) — 0 net lines, removes the soft-deprecation flag.
  • Dead code: delete the PyCObject_Import declaration and PyInt_AsLong macro shim (Finding 39) — 2 lines, zero risk.

Total mechanically-safe removals available today with no version guard needed: ~10 lines plus 4 fixed correctness bugs; the larger ~200-line opportunity is gated on a pythoncapi-compat adoption decision (Finding 47).


Recommended Action Plan

Reference findings by their global number.

Immediate (FIX items)

  1. Findings 1–3 (SuperLU malloc-hook deadlock, _Qhull dealloc crash, bare SuperLU() NULL-deref) — highest severity, smallest fixes (one missing NPY_DISABLE_C_API call; one __cinit__ addition; one Py_TPFLAGS_DISALLOW_INSTANTIATION flag or NULL guard).
  2. Findings 4–5 (_decomp_interpolative.pyx double-free, _mio5_utils.pyx leak on untrusted .mat files) — both memory-safety issues reachable without any privileged technique; #5 in particular is reachable from ordinary use of a widely-used public API against adversarial input.
  3. Findings 6–7 (PROPACK/_dop exception-swallowing) — per git-history-analyzer §2, this is a backport of an already-proven fix, not new design work: copy the if (PyErr_Occurred()) { goto fail; } placement from _odepackmodule.c's fb747bc964 fix into the 8 PROPACK wrappers and the 2 _dop wrap functions.
  4. Findings 10–11 (DIRECT optimizer) — per git-history-analyzer §3, three separate 2025–2026 fix PRs have already worked in this exact 30-line block without landing on these lines; extend the existing PyErr_NoMemory()/PyErr_Format pattern from PR #24752 to close Findings 8–11 for this module in one pass while the code is "hot" and reviewers already have context. Per c-complexity-analyzer, also extract the callback-invocation block from direct_direct_ into a small helper while fixing it — the surrounding function's complexity (cyclomatic 89) is what makes the fix risky to land correctly in place.
  5. Findings 8–9 (unchecked tuple/scalar allocations in _dzvodemodule.c/_dopmodule.c) — mechanical, _odepackmodule.c in the same file family already shows the correct pattern.
  6. Finding 12 (ufunc leak) — per git-history-analyzer §5, this is 2 years old but actively growing (new leaky sites added as recently as 3 weeks before this review); fixing the root cause is a single mechanical change applicable to all ~201 sites at once and stops every future migration from inheriting the broken template.
  7. Findings 13–17 (remaining SuperLU/refcount/GIL bugs) — batch into one deliberate SuperLU-subsystem pass per git-history-analyzer §1 (the file is dormant, not under active churn, so natural rediscovery is unlikely without a deliberate audit).

Short-term (CONSIDER items)

  1. Finding 29 — downgrade the 10 Fortran-bridge modules' Py_mod_multiple_interpreters slot to NOT_SUPPORTED (one line per file, zero behavior change for the single-interpreter case) until real per-module state backs the claim.
  2. Finding 19 — a single guard at the top of uarray._Function's public entry points (getters + tp_call) closes all 4 crash sites from one root cause.
  3. Findings 36–40 — the PyModule_AddObject/PyErr_Fetch deprecation cluster and the dead Python-2 shims (Finding 39) are all zero-risk, no-guard-needed cleanups given the 3.12 floor; worth a single sweep PR.
  4. Finding 41 — the recurring cross-language "constructible but unguarded" pattern (Findings 3, 4, 17, 19, 26) is worth fixing as one coordinated pass rather than four independent point fixes, given it recurred independently four times in this review alone.

Longer-term (POLICY)

  1. Finding 45 — do not pursue project-wide stable-ABI/limited-API adoption; the blockers are structural (framework choice, public-API-backing static types, an unresolved upstream NumPy dependency), not a fixable backlog.
  2. Finding 47 — evaluate adopting pythoncapi-compat specifically for the PyModule_Add-family cluster (~211 sites, verified real benefit); it provides no help for the PyErr_Fetch family, so don't expect it to close those findings.
  3. Finding 50 — adopt Py_TPFLAGS_DISALLOW_INSTANTIATION (or equivalent) as a default policy for internal-only types, closing the whole bug class Finding 41 identifies rather than each instance individually.
  4. Finding 51 — fix this review toolkit's own tree-sitter-cpp build gap before the next run; roughly half of this run's real findings came from manual fallback sweeps because the scanners couldn't see 63 C++ files at all.

C Extension Analysis Report

Extension: tornado.speedups

Scope: entire project — the C extension (tornado/speedups.c) plus its pure-Python fallback (tornado/util.py) and call sites for parity analysis

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer (13/13 — full pipeline)

Executive Summary

tornado/speedups.c is a 100-line, single-function C extension (websocket_mask), and it is about as clean as a C extension gets: ten independent safety/correctness/quality agents — refcounts, error paths, NULL safety, GIL discipline, resource lifecycle, module state, type slots, exception handling, stable-ABI compliance, and complexity — all returned zero FIX findings on the file itself, several with explicit multi-agent cross-validation on the one genuinely arguable line (PyBytes_AsString(result), unchecked but provably safe, confirmed independently three separate ways). The extension's limited-API adoption, multi-phase init, and free-threading declaration are all correct and, per the auditors, earned rather than merely asserted. The two things actually worth attention are outside the C file itself: a release-blocking packaging bug in setup.py (a cog-template comma omission that will break the next PyPI upload), and a historically significant, already-fixed security issue — a ~12.5-year-latent out-of-bounds read in the mask-parsing code, reachable by an attacker via the XSRF cookie, fixed on master in May 2026 and shipped in v6.5.6+. Three low-severity, easily-fixed parity gaps exist between the C implementation and its pure-Python fallback (input-type acceptance envelope), with a single one-line change ("s#s#" → "y*y*") that closes all three at once.

Extension Profile

  • Module: tornado.speedups (1 C file, 100 lines)
  • Init style: multi-phase (PEP 489) — PyModuleDef_Init, m_size=0
  • Python targets: >= 3.10
  • Limited API: Yes — Py_LIMITED_API = 0x030a0000 (3.10 floor) on non-free-threaded builds; skipped on free-threaded builds (no abi3 there). Verified compliant, zero violations.
  • Types defined: 0

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety G 0 0 PyBytes_AsString(result) unchecked but provably safe (3-agent cross-validation)
Error Handling G 0 0 Mask-length check traced via git blame to a real, load-bearing OOB fix
NULL Safety G 0 0 Same line independently confirmed a third way; scanner blind spot noted (doesn't apply its own heuristic there)
GIL Discipline G 0 0 Clean; 3 POLICY notes (see #5-7 below)
Resource Lifecycle G 0 0 No non-PyObject resources exist at all; "s#" needs no PyBuffer_Release (context7-confirmed)
Module State G 0 0 Already ideal (multi-phase, m_size=0); subinterpreter posture is Partial (see #5)
Type Slots G – – Zero types defined — nothing in this agent's domain
PyErr_Clear Safety G 0 0 Zero PyErr_Clear() calls exist
ABI Compliance G 0 0 Exemplary limited-API adoption; CI already runs abi3audit
Version Compat Y 1 0 setup.py classifier-comma bug (packaging, not C code — see Finding 1)
C/Python Parity G 0 3 Input-type acceptance envelope differs between C and Python fallback; mask-length check is synchronized (the model case)
Complexity G – 0 0 hotspots; all 3 functions score 1.0; the one non-trivial function's branching is inherent (performance-justified)

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings


Historical Finding (not a current-code bug — already fixed, included for its significance)

A ~12.5-year-latent, attacker-reachable out-of-bounds read existed in the mask-parsing code, and was fixed in commit d3562c9a (2026-05-26), shipped in v6.5.6+.

  • Introduced: 2013-10-25 (Cython predecessor e8dc5e42) / 2013-11-05 (C rewrite 0ca7fa01) — mask[i % 4] read past a short mask when data_len reached a high enough index.
  • Worsened to unconditional: 2017-04-25 (63beb7a6, a masking-performance optimization) switched the first read to ((uint32_t*)mask)[0] — this reads all 4 bytes of mask unconditionally on every call, even when data_len == 0, turning a conditional edge case into an over-read on every call with a short mask.
  • Fixed: 2026-05-26, d3562c9a on master (cherry-picked to the 6.5 maintenance branch as 96dc88c2). The fix adds if (mask_len != 4) { PyErr_SetString(PyExc_ValueError, "mask must be 4 bytes"); return NULL; } at speedups.c:22-26, and adds the identical guard to the pure-Python fallback (util.py:411-412) in the same commit, backed by a shared parameterized test (MaskFunctionMixin.test_length_validation, websocket_test.py:797-802) that runs against both implementations.
  • Reachability: three of the four call sites always pass exactly-4-byte masks (os.urandom(4), internally-generated frame masks) and were never at risk. The fourth, RequestHandler._decode_xsrf_token (web.py:1602-1620), derives the mask directly from an attacker-controlled _xsrf cookie (mask = binascii.a2b_hex(...) of a hex fragment the client supplies) — a crafted cookie with a non-4-byte mask field triggered the over-read on any build with the compiled extension. The surrounding try/except Exception (web.py:1633) discards the result but does not prevent the read itself, which happens inside the C call before any exception fires. The pure-Python fallback was always bounds-checked (array.array indexing raises IndexError) — only compiled-extension installs were exposed.
  • Current status: fixed on master and in v6.5.6 / v6.5.7. Releases ≤ v6.5.5 remain exposed on installs where the compiled extension is present. No CVE was found associated with this fix in-repo; this is a reachability characterization from git-history analysis, not a published advisory.
  • Recommendation: users on Tornado ≤ v6.5.5 with the C extension installed should upgrade to ≥ v6.5.6.

This finding has security-relevant framing (the XSRF-cookie reachability) that goes beyond what the original commit message states. Per this toolkit's standard guidance, before sharing this report externally — with maintainers, in an issue, or otherwise — see WORKING_WITH_MAINTAINERS.md in the plugin repo: reach out first, don't auto-file, and treat security-relevant characterizations with care even when the underlying fix already shipped.


Findings by Priority

Global numbering: FIX (1), CONSIDER (2-4), POLICY (5-7).

Must Fix (FIX) — 1

# Finding File:Line Agents
1 Cog-template classifier generation omits trailing commas, silently concatenating 5 version classifiers + the Implementation :: CPython line into one invalid PyPI classifier string setup.py:121-131 (template: maint/scripts/runcog.sh) version-compat-scanner

Description: The cog template cog.outl(f"\"Programming Language :: Python :: 3.{minor}\"") emits each classifier without a trailing comma. Python's implicit adjacent-string-literal concatenation silently fuses all the generated lines and the following "Implementation :: CPython" line into a single invalid classifier string — the parsed list has 4 entries instead of the intended ~9. python_requires is unaffected (pip installability is intact today), but PyPI rejects uploads containing unknown classifiers, so this will surface as a release-blocking failure the next time a release is cut from master (introduced by commit 54fe62a5, 2025-08-20; not present on the branch6.5 used for releases through v6.5.7, so no release has hit this yet).

Suggested Fix: Add the missing comma to the cog template ("Programming Language :: Python :: 3.{minor}",) and re-run maint/scripts/runcog.sh — editing the generated setup.py lines directly will be overwritten on the next cog run.

Note: this is a build-metadata/packaging bug, not a defect in the C extension's code — see Tensions below.


Should Consider (CONSIDER) — 3

# Finding File:Line
2 C's "s#" format accepts str arguments and silently masks their UTF-8 encoding; the Python fallback rejects all str speedups.c:17 / util.py:413-414
3 Python's array.array("B", ...) accepts bytearray/memoryview/array('B')/int-iterables; C's "s#" rejects every non-bytes buffer type, including read-only memoryview speedups.c:17 / util.py:413-414
4 Identical invalid input raises different exception types because the two sides validate type and length in opposite order speedups.c:17,22-26 / util.py:411-414

Description (all three): tornado.speedups.websocket_mask (C) and tornado.util._websocket_mask_python (pure-Python fallback) are byte-identical on their advertised contract (bytes mask of length 4, bytes data — confirmed via live testing across data lengths 0-9, 1000, 1003, plus an involution check), and the security-critical mask-length validation is synchronized (see Historical Finding above). All remaining divergence is in the acceptance envelope for non-bytes inputs:

  • Finding 2: s# converts str to its UTF-8 encoding and hands that buffer to C — "ábc" (3 characters, 4 UTF-8 bytes) is accepted as a valid mask by C, while the Python side's array.array("B", "ábc") raises TypeError unconditionally for any str. Concrete crossover: websocket_mask("abcd", b"x") works on a wheel install, raises TypeError on a no-compiler source install.
  • Finding 3: this is the reverse of the usual "C is more permissive" pattern — here Python's array.array constructor happily accepts bytearray, any memoryview (even a read-only one over bytes), array('B'), and plain lists of ints (0-255), all producing byte-identical results to the bytes case. C's "s#" rejects all of them with TypeError because it refuses any type exposing a bf_releasebuffer slot. bytearray/memoryview are the natural containers for network payloads, making this the most likely real-world trip wire for third-party callers.
  • Finding 4: C checks type first (via PyArg_ParseTuple) then length; Python checks length first (len()) then type (array.array). For bytearray(b"abc") (wrong length and writable), C raises TypeError (never reaches the length check) while Python raises ValueError. For 4 lone Unicode surrogates, C raises UnicodeEncodeError (a ValueError subclass) while Python raises TypeError — landing on opposite sides of an except ValueError boundary.

Security impact: LOW for all three. Which implementation runs is decided at install time (compiler availability, env vars), not per-request by a network attacker. All four in-tree call sites pass only bytes, and the one attacker-influenced call site (_decode_xsrf_token) is wrapped in a catch-all except Exception. The value of fixing these is contract clarity and portability, not vulnerability closure.

Suggested Fix (closes all three at once): change speedups.c:17's format string from "s#s#" to "y*y*" (Py_buffer, with matching PyBuffer_Release calls on both buffers before return). y* rejects str outright (closing #2) and accepts all C-contiguous bytes-like objects including bytearray/memoryview/array('B') (closing #3 and most of #4's crossovers — bytearray(b"abc") would then be ValueError on both sides, and every str would be TypeError on both sides, since C would no longer UTF-8-encode). Update speedups.pyi's type annotation accordingly if this is adopted. Regression risk of this change is negligible: every current in-tree caller and every existing test passes bytes only; the str-acceptance quirk is undocumented, untested, and not exercised anywhere in the codebase (verified via git history — no commit or test has ever relied on it).


Tensions

  • The "Y" in Version Compat's Key Metrics row is a build-metadata bug, not a C-code defect. Every agent that reviewed speedups.c's actual source code — refcounts, error paths, NULL safety, GIL discipline, resources, module state, type slots, exception handling, ABI compliance, complexity — found it flawless. The one FIX finding (setup.py's classifier-comma bug) lives entirely in the Python packaging layer and has zero relationship to the C extension's correctness. Don't let the single "Y" row read as "the extension has a bug" — read it as "the release process has a bug."

Policy Decisions (POLICY) — 3

# Finding
5 Subinterpreter (PEP 684, own-GIL) import compatibility is Partial, not Yes/No
6 Version guards use bare Py_LIMITED_API >= instead of the defensive Py_LIMITED_API + 0 idiom
7 Optional: size-gated GIL release for large payloads, mirroring CPython hashlib's HASHLIB_GIL_MINSIZE pattern

Finding 5 (module-state-checker, corroborated independently by gil-discipline-checker): CPython's documented default for a multi-phase module that does not define the Py_mod_multiple_interpreters slot is Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED — shared-GIL subinterpreters are allowed, but own-GIL (PEP 684, 3.12+) subinterpreters raise ImportError (this enforcement was added in 3.12.0b1). speedups.c's two version-guarded slots (Py_mod_multiple_interpreters @ 0x030c0000, Py_mod_gil @ 0x030d0000, lines 76-84) compile in correctly on free-threaded builds, but on the standard build, setup.py's Py_LIMITED_API = 0x030a0000 pin means both guard conditions evaluate false and neither slot compiles in at all. Consequence: import tornado.speedups fails inside an own-GIL subinterpreter on a standard wheel install; tornado/util.py:425-430 catches the ImportError and silently falls back to _websocket_mask_python — Tornado keeps working, just loses the C fast path per subinterpreter — except that TORNADO_EXTENSION=1 turns this into a hard failure instead of a silent fallback. No action needed today (this is a natural consequence of the abi3 floor, not a bug); the cog-templated pin will reach 0x030c0000 automatically once 3.11 support is dropped (EOL Oct 2027), or maintainers could ship per-version 3.12+ wheels sooner if PEP 734 (concurrent.interpreters) demand materializes.

Finding 6 (gil-discipline-checker, cross-referenced by stable-abi-checker): the guards at speedups.c:78,81 use Py_LIMITED_API >= 0x0... rather than the defensive (Py_LIMITED_API + 0) >= 0x0... idiom CPython itself uses in some headers (which tolerates a bare #define Py_LIMITED_API with no value). This is reachable only via a non-standard, undocumented bare definition of the macro — not reachable through Tornado's own build or any standard compiler -D flag. Cosmetic; no urgency.

Finding 7 (gil-discipline-checker): websocket_mask's XOR loop holds the GIL for its entire duration, which is correct for typical WebSocket frame sizes, but data_len is unbounded on the write path (and up to 10 MiB by default on the read path). A size-gated GIL release for the large-payload tail — the pattern CPython's own hashlib uses via HASHLIB_GIL_MINSIZE — is a legitimate, optional throughput enhancement, not a correctness fix.


Strengths

  • Multi-phase init done right: PyModuleDef_Init, m_size=0, correctly-terminated slots array, both the free-threading (Py_MOD_GIL_NOT_USED) and per-interpreter-GIL (Py_MOD_PER_INTERPRETER_GIL_SUPPORTED) declarations are independently verified as earned, not just asserted — the file genuinely has no shared mutable state to make them unsafe.
  • Exemplary limited-API adoption: zero violations found in a full compliance sweep; a single cog-templated variable (min_python_minor=10) consistently drives the Py_LIMITED_API macro value, the cp310 wheel tag, and python_requires together, so the three can't drift out of sync. CI already runs abi3audit on the built wheel as a mechanical check.
  • Zero attack surface by construction in several dimensions: no types defined (nothing for type-slot-checker to find), zero PyErr_Clear() calls (nothing to swallow), no non-PyObject resources of any kind (nothing to leak).
  • Mask-length validation is the model case for dual-implementation hardening: when a real bug was found (the OOB read), the fix landed in both implementations in the same commit, with a shared parameterized test that exercises both, correctly skipped when the extension isn't built.
  • Zero complexity hotspots: all three functions score 1.0 on the composite metric; the one function with non-trivial branching (websocket_mask's 8/4/1-byte stride cascade) was judged inherent, performance-justified complexity rather than a refactor target.
  • Complete historical fix-pattern propagation: every one of the file's ~15 lifetime commits that fixed a bug class (C89 loop-variable declaration, int→Py_ssize_t for 64-bit safety, <stdint.h> inclusion) was applied consistently everywhere it applies — no half-propagated fix exists anywhere in the file's 12-year history.
  • Stable, not churning: speedups.c changed exactly once in the last 181 days / 100 commits (the mask-length fix) — this is mature, settled code, not an active-development risk area.

Code Removal Opportunities

None apply. Checked <plugin_root>/data/deprecated_apis.json's code_removal_opportunities list against this codebase:

  • PyModule_AddType — N/A, no type registrations exist to consolidate.
  • PyImport_ImportModuleAttrString — N/A, no import-then-getattr chains in the C code.
  • PyDict_GetItemRef — N/A, no dict lookups in the C code.
  • PyObject_HasAttrStringWithError — N/A, no PyObject_HasAttrString usage.

Recommended Action Plan

Immediate (FIX items)

  1. Fix Finding 1 — add the missing comma to the classifier cog template (maint/scripts/runcog.sh) and re-run it, before the next release is cut (it will otherwise break that release's PyPI upload).

Short-term (CONSIDER items)

  1. Findings 2-4 — change speedups.c:17's PyArg_ParseTuple format from "s#s#" to "y*y*" (with PyBuffer_Release on both buffers), which closes all three parity gaps in one change with negligible regression risk. Update speedups.pyi's type annotation to reflect the wider accepted-buffer contract if adopted.

Longer-term (POLICY)

  1. Finding 5 — no action needed now; the subinterpreter-support gap on standard abi3 wheels resolves itself when the Py_LIMITED_API floor advances past 3.12 (automatic once 3.11 support drops, or sooner via per-version wheels if own-GIL subinterpreter demand appears).
  2. Finding 6 — cosmetic Py_LIMITED_API + 0 guard hygiene; low priority, bundle with any unrelated touch to those lines.
  3. Finding 7 — optional size-gated GIL release for large payloads; a throughput idea, not a defect, worth considering only if profiling shows contention.

Already Resolved (see Historical Finding above)

  • The mask-length OOB read is fixed on master and in v6.5.6+. No action needed in this codebase; the only actionable item is informational — users on ≤ v6.5.5 with the compiled extension should upgrade.

C Extension Analysis Report — pyarrow

Date: 2026-07-04 Repo: apache/arrow @ local checkout C:\Users\שזדס\repos\arrow

Extension: pyarrow

Scope: python/pyarrow/src/arrow/python/.cc,.h (24 hand-written C++ glue files, ~925 functions) + python/pyarrow/.pyx,.pxd,*.pxi (~24 separately-compiled Cython modules, ~1023 def/cdef/cpdef bodies)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer

Method note: the toolkit's tree-sitter-based scanners could not parse most of this codebase — the C grammar cannot descend into C++ namespace arrow::py { ... } blocks (extracted ~5 of ~925 glue functions), and no tree-sitter/tree-sitter-c wheel was installable for the local Python/mingw toolchain, so Cython-side scans fell back to manual triage too. Every finding below comes from agents' manual Grep/Read triage, in two cases (type-slot-checker) cross-checked by compiling and running throwaway Cython reproductions to empirically confirm claims (hash-nulling on __eq__-without-__hash__, and an uncollectable GC cycle) rather than arguing from documentation alone.

A companion free-threading (PEP 703) review already exists at reports/ft_review.md (26 findings). This report does not re-litigate thread-safety/race findings from that review; where a finding here touches the same code for a different reason, it is cross-referenced rather than duplicated.

Executive Summary

pyarrow's hand-written C++ glue is unusually disciplined for a project this size — RAII lock/GIL patterns are followed almost without exception, and the RETURN_IF_PYERROR() idiom is the dominant, well-applied convention for bridging CPython exceptions into Arrow Status. The bugs found are real but concentrated in the deviations from that house style, not systemic collapse. The two most consequential findings are permanent, uncollectable memory leaks in FlightServerBase and KmsClient — every instance ever constructed leaks forever, confirmed by compiling a live GC reproduction — and an ExtensionType unhashability regression (also empirically confirmed), both from the type-slot-checker. A UDF-registration refcount imbalance was independently found by two agents and then had its severity corrected downward by the git-history-analyzer, which traced the actual shared_ptr lifetime through cpp/src/arrow/compute/function.cc and found the defect fires once at registry teardown, not per call — a good example of this review's cross-agent verification working as intended. Version compatibility, stable-ABI posture, and API parity are all clean-to-POLICY: no deprecated-API debt beyond what's already tracked, and abi3 adoption is honestly infeasible today because of NumPy's own non-limited C-API, not anything pyarrow could fix unilaterally.

Extension Profile

  • Module: pyarrow (24 hand-written C++ files + ~24 compiled Cython extension modules)
  • Init style: single-phase (all modules; no Py_mod_exec/Py_mod_multiple_interpreters found anywhere)
  • Python targets: >=3.10 (declared), tested through 3.13 in CI
  • Limited API: no (not claimed; assessed as infeasible today — see Finding 47)
  • Types defined: ~24 Cython extension modules' worth of cdef class types (hundreds), plus 1 hand-written static PyTypeObject (MonthDayNanoTupleType)

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety R 7 5 common.cc:166 leaks one str per formatted traceback line
Error Handling R 5 4 numpy_init.cc sets the "numpy imported" flag before import succeeds
NULL Safety Y 3 2 Unchecked PyArray_Descr* family spanning numpy_internal.h/numpy_convert.cc/inference.cc
GIL Discipline G 0 2 to_reader() methods skip with nogil: where every sibling releases it
Resource Lifecycle Y 2 2 io.pxi download() leaks the destination file handle on seek/malloc failure
Module State G 0 4 Single-phase init blocks subinterpreter support across all ~24 modules
Type Slots Y 3 1 FlightServerBase/KmsClient leak every instance permanently (empirically verified)
PyErr_Clear Safety Y 3 3 PyObjectToPyInt can silently substitute a different, wrong integer value
ABI Compliance G 0 0 Infeasible today — NumPy + datetime C-APIs have no limited-API path (1 POLICY)
Version Compat G 0 2 Cython pinned >=3.1 with no upper bound (1 POLICY)
C/Python Parity G 0 0 .pyi stub is an intentional empty placeholder — no parity surface to violate
Complexity G - 6 array()/record_batch() duplicate ~40 lines of PyCapsule-protocol handling

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings

Findings by Priority

Must Fix (FIX) — 19

# Finding File:Line Agents
1 FlightServerBase holds an uncollectable Python↔C++ reference cycle — the C++ PyFlightServer stores a strong ref back to self (Py_INCREF in flight.cc:83-87), invisible to Cython's auto-generated tp_traverse (which only sees the finalizer attribute). Confirmed with a live compiled reproduction: gc.collect() does not free the object. The existing _FlightServerFinalizer/weakref.finalize machinery (added for a different reason — interpreter-shutdown segfaults, ARROW-16597) does not break this cycle. Every FlightServerBase instance ever constructed leaks forever, along with everything reachable from it. python/pyarrow/_flight.pyx:2934-3031, src/arrow/python/flight.cc:83-87, flight.h:175 type-slot-checker
2 KmsClient has the identical uncollectable self-cycle, and is worse off: it has no object-typed attribute at all, so Cython never even marks it GC-tracked. Confirmed via the same live-reproduction methodology. Every custom KMS-integration instance leaks permanently. python/pyarrow/_parquet_encryption.pyx:330-343, src/arrow/python/parquet_encryption.cc:28, parquet_encryption.h:72-87 type-slot-checker
3 ExtensionType (and every subclass, including UnknownExtensionType and all user-defined pa.ExtensionType subclasses) silently becomes unhashable: it overrides __eq__ without redefining __hash__, and Cython replicates CPython's type.__new__ hash-nulling behavior for cdef class in this situation — confirmed by compiling a minimal reproduction (hash() raises TypeError: unhashable type). DataType and all non-extension subclasses remain correctly hashable, so this is a targeted regression on the primary extension mechanism, not a codebase-wide pattern. python/pyarrow/types.pxi:355-368, 1874-1882, 2138 type-slot-checker
4 UDF registration leaks a reference imbalance: RegisterScalarFunction/RegisterVectorFunction/RegisterTabularFunction each construct two independent OwnedRefNoGIL wrappers over the same borrowed PyObject* function (one at the call site, uninc­remented; one inside RegisterUdf, with one Py_INCREF) — net −1 over the combined lifetime. RegisterScalarAggregateFunction/RegisterHashAggregateFunction get this right (one incref, one wrapper). Severity revised: git-history-analyzer traced the actual shared_ptr lifetime through cpp/src/arrow/compute/function.cc and found both owners share the same Kernel/registry-entry lifetime, so the defect fires once at kernel/registry teardown — not per call as first suspected. It is live specifically when a custom FunctionRegistry (a supported, documented registry= kwarg) is destroyed while the interpreter is still running; the default global registry is process-lifetime and its destructor is Py_IsInitialized()-guarded, which likely masks the bug there. Traced to a single origin commit, 1f2479908 (2024-08-07, "Sanitize Python reference handling in UDF implementation"), which fixed the aggregate family correctly and missed the scalar/vector/tabular family — a textbook twin-fix asymmetry. python/pyarrow/src/arrow/python/udf.cc:452-516 (vs. correct 518-635) refcount-auditor, gil-discipline-checker (independent re-spot), git-history-analyzer (origin + severity correction)
5 numpy_imported is set true before import_numpy() actually succeeds; a failed import leaves has_numpy() lying while PyArray_API is NULL, so any later entry into ~14 has_numpy() && PyArray_* gated call sites (helpers.cc:88, inference.cc:463/467, iterators.h:48/110, python_to_arrow.cc:206/408/487/702/844, numpy_internal.h:159/167/175) NULL-derefs the API table. git-history-analyzer confirms this was wrong from inception in the single commit (9ab9532a2, 2024-09-02) that added the file, and has never been revisited or tested against an import-failure path. python/pyarrow/src/arrow/python/numpy_init.cc:25-32 error-path-analyzer, git-history-analyzer
6 PyListConverter::AppendSequence/AppendIterable silently swallow real exceptions: AppendSequence uses an unchecked PySequence_Size (-1 + pending exception flows into Append(true,-1)/Extend(value,-1), then returns Status::OK()); AppendIterable's PyIter_Next loop has no RETURN_IF_PYERROR() after exhaustion, so a mid-iteration exception (e.g. RuntimeError: set changed size during iteration) is silently dropped and truncated data is committed. Compare the correct idiom at iterators.h:190-194/ipc.cc:48-55, which this pair deviates from. python/pyarrow/src/arrow/python/python_to_arrow.cc:889-907 error-path-analyzer, refcount-auditor
7 PySequence_GetItem result (line) is never decref'd while formatting a Python exception's traceback into a Status message — leaks one str per traceback line on every formatted exception. python/pyarrow/src/arrow/python/common.cc:166 refcount-auditor
8 A local PyObject* ref from PyMemoryView_GetContiguous shadows the member OwnedRef ref in PyBytesView::ParseBinary — the new memoryview is never released (leak), and the member meant to keep the buffer alive is never actually set (the view's data pointer only survives because of the leak). Compare the correct sibling ParseUuid, which does ref.reset(...). python/pyarrow/src/arrow/python/common.h:409-414 refcount-auditor
9 StringToTzinfo's fixed-offset path leaks the timedelta object (offset) passed into constructing datetime.timezone — one leak per fixed-offset timezone conversion, and also on the failure path. python/pyarrow/src/arrow/python/datetime.cc:441-446 refcount-auditor
10 Converting a MonthDayNanoInterval array to pandas leaks 4 PyLong objects per non-null row: PyDict_SetItemString does not steal, and the four PyLong_FromLong* results ("months"/"days"/"microseconds"/"nanoseconds") are never released. Contrast with the correct pattern at the same file's line 1696-1700. python/pyarrow/src/arrow/python/arrow_to_pandas.cc:1308-1312 refcount-auditor
11 ConvertToSequenceAndInferSize leaks the raw PyList_New() result (lst) plus every item already stored into it if a user iterator raises mid-loop — only the PyList_SetSlice failure path releases it. python/pyarrow/src/arrow/python/python_to_arrow.cc:1243-1253 refcount-auditor
12 PyExtensionType::ToString NULL-derefs Py_TYPE(instance.obj())->tp_name when GetInstance() fails (e.g. the user's __arrow_ext_deserialize__ raises) — GetInstance() returns nullptr with an exception set, and ToString doesn't check before dereferencing. python/pyarrow/src/arrow/python/extension_type.cc:76-84 error-path-analyzer
13 PyDecimal_ISNAN NULL-derefs PyObject_IsTrue(NULL) if obj.is_nan() raises (e.g. a decimal.Decimal subclass with a broken override) — reachable from PandasObjectIsNull on every from_pandas conversion. python/pyarrow/src/arrow/python/decimal.cc:219-224 error-path-analyzer
14 A family of PyArray_Descr*-producing calls goes unchecked while sibling call sites in the same files correctly guard the identical allocator with RETURN_IF_PYERROR(): GetSafeNumPyDtype's result feeds set_numpy_metadata/PyDataType_REFCHK with no NULL check (sinks at arrow_to_pandas.cc:315-316,527-528); NumPyScalarToArrowDataType's PyArray_DescrFromScalar result flows unchecked into NumPyDtypeToArrow (descr->type_num dereferenced with zero guard). python/pyarrow/src/arrow/python/numpy_internal.h:182-190, numpy_convert.cc:122-137 null-safety-scanner
15 UDF execution stores wrap_array(c_data)/wrap_scalar(c_data)'s return value directly into a call-argument tuple via PyTuple_SetItem with no NULL check, and the tuple is handed to the user's Python UDF (cb(function, ..., arg_tuple)) before the subsequent RETURN_NOT_OK(CheckPyError()) runs. If wrapping fails (OOM or an exception during Array/Scalar construction), a NULL element sits live inside a tuple actively used in a Python call — a crash risk on every UDF invocation path (scalar, hash-aggregate, plain exec), not just a rare setup path. Found independently by two agents from different angles (general error-path bridging vs. pointer-safety). python/pyarrow/src/arrow/python/udf.cc:196-197, 335-336, 412-413, 416-417 null-safety-scanner, error-path-analyzer
16 NativeFile.download()/_download_nothreads() open the destination file (open(stream_or_path, 'wb')) and then call self.seek(0) and malloc(buffer_size) before entering the try/finally that would run cleanup()/close the file. A seek() or allocation failure in that window leaks the opened file object, since the background writer thread (which owns the only cleanup() call) never starts. git-history-analyzer confirms this gap has survived three refactors unchanged since the threaded implementation was first written — a genuinely unexercised error path, not a false positive. python/pyarrow/io.pxi:642-739, 741-795 resource-lifecycle-checker, git-history-analyzer
17 internal::PyObjectToPyInt unconditionally clears any exception from PyNumber_Index(obj) (i.e. obj.__index__()) before falling back to the type's nb_int slot — the only case where clearing is correct is TypeError (no __index__ present). If __index__ is present and raises for a domain reason (ValueError, OverflowError, or a genuine MemoryError), this silently discards that specific error and falls back to __int__, which can succeed with a different, silently wrong integer value rather than failing cleanly — the most consequential instance of the "clear-and-fallback" anti-pattern found in this review. python/pyarrow/src/arrow/python/helpers.cc:171-192 pyerr-clear-auditor
18 PyBytesView::ParseString probes UTF-8 validity by attempting a decode and unconditionally clearing any resulting exception as "not valid UTF-8," including MemoryError from the temporary allocation — silently mislabeling OOM as an encoding failure and continuing as Status::OK(). python/pyarrow/src/arrow/python/common.h:369-387 pyerr-clear-auditor
19 StringToTzinfo unconditionally clears any exception from zoneinfo.ZoneInfo(tz) (not just the expected ZoneInfoNotFoundError/KeyError for an unrecognized zone) before falling back to pytz; if pytz is also absent, the user sees a generic "zoneinfo module or pytz package must be installed" message even when the real problem was a misspelled timezone name. git-history-analyzer traces this to a 6-week-old, deliberate refactor (9583eb787, 2026-05-22, "switch from pytz to zoneinfo by default") and frames the underlying fallback design as intentional/POLICY — the defect is specifically that the clear is exception-type-blind rather than checking for the expected lookup error first. python/pyarrow/src/arrow/python/datetime.cc:460-466 pyerr-clear-auditor, git-history-analyzer (context)

Should Consider (CONSIDER) — 27

# Finding File:Line
20 OwnedRef::operator=(OwnedRef&&) does obj_ = other.detach() without releasing the currently-held reference — assigning into a non-empty OwnedRef silently leaks. All current call sites assign into empty refs, so latent today, but this is an exported public header (OwnedRefNoGIL inherits the same bug). python/pyarrow/src/arrow/python/common.h:188-191
21 SafeCallIntoPython leaks the previous exception triple (PyErr_Fetch refs) when the wrapped call returns a Python-error Status while an exception was already pending on entry. python/pyarrow/src/arrow/python/common.h:152-165
22 NumPyDtypeUnifier has no ownership discipline at all: VisitDType's PyArray_DescrFromScalar result is never released; _NUMPY_UNIFY_PROMOTE_TO's PyArray_DescrFromType result is stored with no incref; Observe also stores borrowed descrs past the owning array's lifetime, risking a dangling read in Observe_DATETIME. python/pyarrow/src/arrow/python/inference.cc:88-95, 284-290, 464
23 PyStructConverter::Init leaks a PyBytes object if the paired PyUnicode_FromStringAndSize call fails (OOM-only path; two allocations share one RETURN_IF_PYERROR()). python/pyarrow/src/arrow/python/python_to_arrow.cc:1030-1038
24 PandasBlockCreator::AppendBlocks/SplitBlockCreator::Convert leak item/result on several error-return paths (decref happens after the error check, or not at all on some early returns). python/pyarrow/src/arrow/python/arrow_to_pandas.cc:2290-2302, 2487-2510
25 IsPyUuid's PyObject_IsInstance call clears any failure unconditionally, steering type inference down the "not a UUID" branch with no signal to the caller — low-severity since this only fails for pathological __instancecheck__ implementations. python/pyarrow/src/arrow/python/helpers.cc:338-357
26 PyObject_StdStringRepr's diagnostic PyUnicode_AsEncodedString failure is cleared unconditionally; the function has no propagation path anyway (returns std::string), but a MemoryError/KeyboardInterrupt hitting exactly here is silently absorbed rather than surfaced. python/pyarrow/src/arrow/python/helpers.cc:107-124
27 Two sites (arrow_to_pandas.cc binary-wrapping, numpy_to_arrow.cc UTF-32 encoding) clear a real exception and substitute a hand-built generic Status, bypassing the codebase's own exception-preserving RETURN_IF_PYERROR()/ConvertPyError() idiom used almost everywhere else — still returns an error, but discards the original exception type/message (could mask MemoryError). python/pyarrow/src/arrow/python/arrow_to_pandas.cc:1206, numpy_to_arrow.cc:689
28 TypeInferrer::Visit's PyArray_DescrFromScalar result (feeding VisitDType/Observe) and the _NUMPY_UNIFY_PROMOTE_TO macro's PyArray_DescrFromType result are both dereferenced with no NULL check — OOM-only trigger, same descriptor family as Finding 14. python/pyarrow/src/arrow/python/inference.cc:92-96, 284-285, 464
29 MonthDayNanoIntervalToNamedTuple passes three unchecked PyLong_FromLong* results directly into PyStructSequence_SetItem, which — like PyTuple_SET_ITEM — doesn't itself validate the item; an OOM failure would be silently stored. python/pyarrow/src/arrow/python/datetime.cc:597-608
30 Declaration.to_reader() (Acero) and Scanner.to_reader() (Dataset) call blocking C++ entry points without with nogil:, while every sibling method (to_table(), scan_batches(), etc.) releases the GIL for the identical or equivalent underlying call — verified against the actual C++ implementation (DeclarationToReader, AsyncScanner::ToRecordBatchReader → the same ScanBatches() that scan_batches() explicitly treats as GIL-release-worthy). No confirmed hard deadlock (construction dispatches and returns rather than blocking on a future), but a verifiable, load-bearing inconsistency that could bite if a future node type blocks synchronously during startup. python/pyarrow/_acero.pyx:611-626, _dataset.pyx:4032-4042
31 PyExtensionType::GetInstance() is the only method on its class that doesn't construct PyAcquireGIL itself, relying entirely on the caller already holding the GIL — no live bug found (both current callers already hold it), but it's the one self-undefending method on a class whose siblings all defend themselves. python/pyarrow/src/arrow/python/extension_type.cc:162-181
32 FlightServerBase.init() sets self.server (the shared_ptr) before calling c_server.Init(); if Init() raises, the finalizer that ensures orderly Shutdown()/Wait() (added for ARROW-16597 to avoid an interpreter-exit segfault) is never registered, so a partially-initialized server relies on the plain C++ destructor alone. Not independently verified how much of PyFlightServer::Init executes before failure (core cpp/ out of scope). python/pyarrow/_flight.pyx:2958-3031
33 NativeFile.__cinit__/__dealloc__'s own_file flag is initialized False and gates the only explicit close() call in __dealloc__, but is never set True anywhere in the codebase — dead code, not itself a leak (the wrapped Arrow C++ streams are independently shared_ptr-managed), but could mask a real gap if a future subclass relies on it. python/pyarrow/io.pxi:121-130
34 array() and record_batch() each independently implement the identical ~24-line PyCapsule import-protocol block (__arrow_c_device_array__/__arrow_c_array__ handling) — ~40 duplicated lines and two of the highest branch counts in the module; table() already avoids this by delegating to record_batch(), showing the fix pattern already exists in-tree. python/pyarrow/array.pxi:282-305, table.pxi:6005-6034
35 S3FileSystem.__init__ (24 params, cyclomatic 30) entangles a 5-way mutually-exclusive credential-mode validation with ~15 straight-line field copies mirroring the C++ S3Options struct — the validation logic is where bugs would hide; extracting it from the boilerplate would isolate that risk. python/pyarrow/_s3fs.pyx:285-427
36 TypeInferrer::GetType encodes a 16-branch type-unification priority order as implicit if/elif sequencing rather than a declarative, ordered table — correct today, but silently breaks if a future counter is inserted in the wrong branch position. python/pyarrow/src/arrow/python/inference.cc:522-620
37 NumPyConverter::ConvertData<Date32Type>/<Date64Type> duplicate an identical ~15-line cast-fallback tail, differing only in the day-unit branch (narrowing cast vs. millisecond-scaling multiply) — extracting the shared tail would make the real difference visually obvious instead of buried in two near-identical blocks. python/pyarrow/src/arrow/python/numpy_to_arrow.cc:487-562
38 NativeFile.download() hand-rolls a background writer thread, bounded queue, two closures capturing mutable shared state, a malloc'd buffer, and a busy-wait for backpressure — flagged purely as complexity (nesting 5, several interacting closures over shared mutable state are hard to verify correct by inspection), not as an asserted defect beyond Finding 16. python/pyarrow/io.pxi:642-739
39 Dataset.take/head/scanner/to_batches/to_table/count_rows repeat near-identical docstring-plus-one-line-forwarding boilerplate across ~8 call sites in the class hierarchy — a duplication cost, not a per-function complexity problem. python/pyarrow/_dataset.pyx (repeated)
40 All ~24 Cython modules use single-phase init with no Py_mod_multiple_interpreters marker; under CPython's stricter subinterpreter checks (check_multi_interpreter_extensions, the PEP 734 default posture), pyarrow is rejected outright at import time in a subinterpreter. Not actionable unless subinterpreter support becomes a real requirement — see the Migration Assessment note below. all .pyx files
41 The pandas_*/uuid_UUID C++ statics and the Cython cdef-static caches (pandas_api, _type_cache family) are shared-library-wide, not module-scoped or reachable via PyModule_GetState() — a different lens on the same globals covered thread-safety-wise in reports/ft_review.md: here the concern is cross-interpreter aliasing/lifecycle, which would corrupt state across any future subinterpreters rather than race under free-threading. python/pyarrow/src/arrow/python/helpers.cc:338-370, python/pyarrow/pandas-shim.pxi:282, types.pxi:36,3743,4119-4121
42 MonthDayNanoTupleType is a static (non-heap) PyTypeObject, a process-wide singleton that cannot be made per-interpreter without conversion to a heap type — cross-references reports/ft_review.md Finding 13 (race angle) with a distinct subinterpreter-readiness angle here. python/pyarrow/src/arrow/python/datetime.cc:61, 276-283
43 The _true Expression sentinel (Expression._scalar(True)) is duplicated verbatim across two independently-compiled modules — harmless (read-only), but illustrates that any future multi-phase migration must be repeated per compiled module, not once. python/pyarrow/_dataset.pyx:159, _dataset_parquet.pyx:65
44 6 of ~30 top-level .pyx/.pxd files have no explicit # cython: language_level = 3 directive (all others do); not a functional break today (CMake enforces Cython ≥3.1), but inconsistent and worth making explicit ahead of a future Cython 4 release where defaults could change. python/pyarrow/lib.pyx, _cuda.pyx, _orc.pyx, _parquet.pyx, _parquet_encryption.pyx, _pyarrow_cpp_tests.pyx
45 io.cc:170's PyObject_HasAttrString silently reports any exception during attribute lookup (e.g. a raising __getattr__/property) as "attribute absent" rather than propagating it; PyObject_HasAttrStringWithError (3.13+, already backported by the vendored pythoncapi_compat.h used elsewhere in the codebase, just not included in this file) is the error-propagating replacement. python/pyarrow/src/arrow/python/io.cc:170
46 If the placeholder pyarrow-stubs tree in this repo were ever wheel-packaged or published as its own PyPI -stubs distribution (it currently is not — sdist-only, per PEP 561 stub-package precedence rules), it would silently mask pyarrow's own py.typed inline annotations behind a catch-all __getattr__(name) -> Any. Currently POLICY/non-user-facing (maintainers already track removing the Any-masking __getattr__ before real stubs ship, GH-48970); worth a CI guard if packaging ever changes. python/pyarrow-stubs/pyarrow/__init__.pyi, python/pyproject.toml:85-90

Tensions

  • udf.cc severity: refcount-auditor's initial HIGH vs. git-history-analyzer's traced-through MEDIUM. Both analyses agree the bug is real (a genuine −1 refcount imbalance from a twin-fix asymmetry), but disagree on blast radius. refcount-auditor's first read suggested per-call risk; tracing the actual shared_ptr ownership through cpp/src/arrow/compute/function.cc showed the defect fires once at kernel/registry teardown, gated on a custom FunctionRegistry being destroyed mid-process. → Resolved in favor of the historian's traced analysis (Finding 4 reflects the corrected MEDIUM severity and precise trigger condition); this is presented as a positive example of this review's cross-agent verification catching an initial overstatement.
  • StringToTzinfo's PyErr_Clear: FIX (pyerr-clear-auditor) vs. CONSIDER/policy-adjacent (git-history-analyzer). The specialized agent flagged the unconditional clear as a bug; the historian, having traced it to a deliberate 6-week-old design commit, frames the fallback itself as intentional and the type-blindness of the clear as the actual, narrower defect. → Both framings are preserved in Finding 19's description rather than picking one; the actionable fix (check for the specific lookup error before clearing) is the same either way.
  • Stable-ABI feasibility vs. subinterpreter feasibility: both assessments (Findings 40 and 47) independently conclude "not feasible today," but for different root causes — subinterpreters are blocked by pyarrow's own init-style and global-state architecture (fixable, if expensive); abi3 is blocked primarily by an upstream dependency (NumPy's C-API has no limited-API path) that pyarrow cannot fix unilaterally. Worth keeping separate in any roadmap discussion — one is "our debt," the other is "someone else's prerequisite."

Policy Decisions (POLICY) — 9

# Finding
47 Stable ABI / limited API adoption is infeasible today, not from a single unfixable line but from the combination of: NumPy's own C-API (171 call sites, no limited-API-compatible header path — an upstream prerequisite, not a pyarrow bug), the CPython datetime C-API's struct-access macros (no clean limited-API equivalent short of a hot-path perf regression), a hand-written static PyStructSequence type, and ~24 modules' worth of Cython-generated static types (mechanically migratable via Cython's own cython_limited_api=True, but gated on the first two). Zero genuine private (_Py*) API usage found — a clean starting point if this is ever revisited. Recommendation: keep the current build model; revisit only if NumPy ships limited-API-compatible headers.
48 InitDatetime, NewMonthDayNanoTupleType, and import_pyarrow call Py_FatalError (process abort) on init failure instead of propagating an ImportError — import pyarrow can kill the host process rather than raising a catchable exception if datetime-capsule or struct-sequence init fails.
49 csv.cc's invalid-row handler deliberately swallows user exceptions via PyErr_WriteUnraisable and returns a default result — the handler runs on a non-Python thread, so this is a considered design choice, but it does mean a raising handler cannot abort the read.
50 flight.cc's ServeWithSignals leaves a KeyboardInterrupt raised by PyErr_CheckSignals() pending behind a Status::OK() return (acknowledged by an in-code // XXX comment) — relies on the Cython caller's except * to notice.
51 Cython is pinned >= 3.1 in pyproject.toml, both requirements files, and CMakeLists.txt's version check, with no upper bound anywhere. Cython 4 hasn't shipped yet, so this is prospective rather than active — recommend adding <4 proactively so a future major release doesn't silently build against an untested toolchain.
52 helpers.cc's pandas_*/uuid_UUID statics deliberately hold borrowed (not owned) references, with an in-code comment explaining the trade-off: OwnedRef destructors would run on a finalized interpreter otherwise. Correct and intentional for the single-interpreter case; Finding 41 covers the subinterpreter-era caveat this trade-off doesn't anticipate.
53 The default memory-pool singleton (memory.pxi/common.cc) and the signal_handlers_enabled global flag are appropriately process-global given that they wrap an inherently process-wide C++ allocator singleton and an OS-level (main-thread-only) signal-handling concept respectively — not pyarrow-specific design gaps.
54 Numerous module-level dispatch tables and caches (_array_classes, _scalar_classes, _pep3118_type_map, PRIMITIVE_TYPES, _type_aliases, DEFAULT_CALL_OPTIONS, _PARQUET_READ_OPTIONS) are populated once at import and never mutated afterward — functionally read-only constants despite being mutable container types; no action needed.
55 CategoricalWriter::AddResultMetadata does an extra, unnecessary Py_INCREF on Py_True/Py_False before PyDict_SetItemString (which increfs again) with no matching decref — harmless in practice since these are immortal singletons in 3.12+, but could be tidied for clarity.

Strengths

  • RAII discipline is close to flawless. PyAcquireGIL/PyReleaseGIL/SafeCallIntoPython are the only mechanism used to touch Python objects from contexts that might not hold the GIL, verified call-site-by-call-site across flight.cc, filesystem.cc, csv.cc, udf.cc, ipc.cc, async.h. Zero missing-release findings anywhere in scope.
  • RETURN_IF_PYERROR()/CheckPyError()/ConvertPyError() is a genuinely good, pervasive idiom for bridging CPython exceptions into Arrow Status while preserving the original exception object for later restoration — the deviations from it (Findings 6, 12, 13, 27) are the exception, not the rule.
  • The buffer-protocol implementations (PyBuffer, Buffer.__getbuffer__, Tensor.__getbuffer__) are textbook-correct, including the interpreter-finalization guard (Py_IsInitialized() checks) added after a 2023 shutdown-segfault bug report.
  • Only 3 __dealloc__ methods exist in the entire ~1023-function Cython surface, and all three are correct — nearly every native resource is owned by shared_ptr/unique_ptr members with implicit C++ destruction, which is safe by construction and explains why resource-lifecycle findings were so sparse (2 FIX, both in one file).
  • Version-compat posture is clean: zero dead PY_VERSION_HEX guards, zero unguarded post-floor API usage, zero unmigrated deprecated-API sites beyond what's already tracked, and correct, already-adopted use of pythoncapi_compat.h's 3.13+ *Ref APIs (PyDict_GetItemRef, PyDict_GetItemStringRef) in 4 files — the right template for any future modernization.
  • Complexity is genuinely low on average (avg cyclomatic 3.9, avg function length 20 lines across the C++ glue; only 1 Cython function exceeds 200 lines in a ~1023-function corpus), and several apparent "hotspots" by raw line count turned out to be docstring-heavy trivial forwarders on manual inspection — the estimator's false positives were caught before being reported.

Code Removal Opportunities

Cross-referenced against <plugin_root>/data/deprecated_apis.json:

  • PyModule_AddType (3.10+): Not applicable — grepped PyType_FromSpec/PyType_Ready/PyModule_AddObject across all hand-written glue files; zero hits. Type registration for all ~24 Cython modules is handled by Cython's own codegen, not hand-rolled boilerplate.
  • PyImport_ImportModuleAttrString (3.14+): applicable in principle to ~13 call sites of the ImportModule()+ImportFromModule() pattern in helpers.cc and its callers (arrow_to_pandas.cc, datetime.cc, decimal.cc, inference.cc), collapsing import-then-getattr into one call. Not actionable now — the declared floor is 3.10 and the vendored pythoncapi_compat.h predates this function's addition. Estimated savings if/when adopted: ~65-100 lines across 13 sites.
  • PyDict_GetItemRef (3.13+, backported): already adopted at numpy_to_arrow.cc:832 and python_to_arrow.cc:1151 via the vendored compat header — no remaining sites of the older PyDict_GetItemWithError pattern to migrate.
  • PyObject_HasAttrStringWithError (3.13+, backported): 1 unmigrated site, io.cc:170 (Finding 45) — zero line-count change, fixes a real exception-swallowing bug rather than saving lines.

Migration Assessment (Subinterpreter Readiness)

Not requested as a standalone command, but surfaced strongly enough by module-state-checker to summarize here: subinterpreter support is not feasible without a multi-quarter architectural project. All ~24 modules would need conversion to multi-phase init; every cdef module-level global (Cython has no automatic mechanism to route these into PyModule_GetState()-reachable state) would need manual extraction into per-module structs, repeated independently across all 24 compiled modules; and the shared-library-wide C++ statics (pandas_*, datetime_api, MonthDayNanoTupleType, NumPy's own API table) would need to move to per-PyInterpreterState storage, touching every call site in the conversion machinery. Recommendation: keep the current single-phase design unless subinterpreter support becomes an actual product requirement; if free-threading work (already tracked in reports/ft_review.md) or this subinterpreter question both become live, revisit together since they touch the same global-state inventory.

Recommended Action Plan

Immediate (FIX items)

  1. Fix the two permanent leaks (Findings 1, 2): change PyFlightServer::server_ and PyKmsClient::handler_ to hold weakrefs, resolved via PyWeakref_GetRef at each use — mirroring the pattern PyExtensionType::type_instance_ already uses correctly for exactly this reason.
  2. Fix ExtensionType unhashability (Finding 3): add a __hash__ alongside the existing __eq__ override, computed from the same fields (type(self), extension_name, storage_type).
  3. Fix the UDF registration refcount imbalance (Finding 4): reuse the single OwnedRefNoGIL already constructed for kernel_init as the source for udf_data in RegisterUdf, matching the aggregate family's pattern exactly.
  4. Fix numpy_init.cc's ordering bug (Finding 5): only set numpy_imported = true after import_numpy() returns success.
  5. Add the missing exception checks in AppendSequence/AppendIterable (Finding 6): RETURN_IF_PYERROR() after both the size lookup and the iteration loop.
  6. Fix the two confirmed leaks (Findings 7, 8): decref line in common.cc:166; fix the shadowed-variable bug in ParseBinary to assign through this->ref.
  7. Fix the remaining refcount leaks (Findings 9, 10, 11): wrap offset in an OwnedRef in StringToTzinfo; wrap the four PyLong fields in OwnedRef in the MonthDayNanoInterval writer; hold lst in an OwnedRef in ConvertToSequenceAndInferSize.
  8. Add NULL checks (Findings 12, 13, 14, 15): guard GetInstance()'s result in ToString; guard is_nan's result in PyDecimal_ISNAN; add RETURN_IF_PYERROR() after the PyArray_Descr* family's allocations; check wrap_array/wrap_scalar before storing into the UDF call tuple.
  9. Fix the io.pxi download() error-path leak (Finding 16): move open() inside the try/finally (or restructure so cleanup() is reachable before the writer thread starts).
  10. Fix the three PyErr_Clear findings (Findings 17, 18, 19): check the specific expected exception type (TypeError/UnicodeDecodeError/the zoneinfo lookup error) before clearing in each of the three sites, propagating anything else via ConvertPyError().

Short-term (CONSIDER items)

  1. Fix OwnedRef::operator=(OwnedRef&&) to release-before-detach (Finding 20) — cheap, prevents a latent public-header footgun.
  2. Fix the SafeCallIntoPython exception-triple leak (Finding 21) and give NumPyDtypeUnifier proper ownership discipline (Finding 22).
  3. Fix the remaining OOM-only leaks (Findings 23, 24) and the two PyErr_Clear call sites that bypass the codebase's own exception-preserving idiom (Finding 27) — low urgency, mechanical fixes.
  4. Add with nogil: to to_reader() in both Acero and Dataset for consistency with sibling methods (Finding 30); add self-GIL-acquisition to GetInstance() for consistency with its siblings (Finding 31).
  5. Reorder FlightServerBase.init() to register the finalizer before or immediately after Init() regardless of outcome (Finding 32); remove the dead own_file flag or wire it up (Finding 33).
  6. Deduplicate the array()/record_batch() PyCapsule-import block (Finding 34) using table()'s existing delegation pattern as the template; extract S3FileSystem's credential validation (Finding 35); make TypeInferrer::GetType's priority order an explicit table (Finding 36); deduplicate the ConvertData<Date32/64Type> cast tail (Finding 37).
  7. Add the 6 missing language_level directives (Finding 44) and switch io.cc:170 to PyObject_HasAttrStringWithError (Finding 45).

Longer-term (POLICY)

  1. Add an upper version bound (<4) to the Cython pin in all four locations before Cython 4 ships (Finding 51).
  2. Revisit stable-ABI adoption only if NumPy ships a limited-API-compatible C-API surface (Finding 47); revisit subinterpreter support only if it becomes an actual product requirement, ideally bundled with any future free-threading architectural work (Finding 40 / Migration Assessment).
  3. Consider whether Py_FatalError in the three init paths (Finding 48) should be softened to a catchable ImportError — a process abort on import pyarrow failure is a harsh user experience for what may be a recoverable environment issue.

Related reports: reports/ft_review.md (free-threading / PEP 703 analysis, 26 findings — cross-referenced at Findings 41, 42 above).

C Extension Analysis Report

Extension: wrapt._wrappers

Scope: src/wrapt/_wrappers.c (entire C extension — this is a single-file extension)

Agents Run: refcount-auditor, error-path-analyzer, null-safety-scanner, gil-discipline-checker, resource-lifecycle-checker, module-state-checker, type-slot-checker, pyerr-clear-auditor, stable-abi-checker, version-compat-scanner, parity-checker, c-complexity-analyzer, git-history-analyzer (all 13; all/default aspect set)

Methodology: All scanner scripts were run with python3t.exe (free-threaded CPython 3.15.0b2, with tree-sitter/tree-sitter-c installed for it). External tools: cppcheck ran cleanly; clang-tidy was skipped (no compile_commands.json in this repo). Context7 (ctx7) was requested for CPython C API documentation lookups but its network calls failed (fetch failed) in this environment — agents relied on their built-in Python/C API knowledge, cross-checked directly against the actual #if PY_VERSION_HEX guards already present in the source, which multiple agents verified against upstream CPython behavior. 11 of 13 agents hit an account usage limit on the first dispatch and were re-run to completion after the reset; all 13 results below are from completed, real runs.

Executive Summary

wrapt's C extension (src/wrapt/_wrappers.c, 4928 lines, hand-written, python_requires >=3.9) is unusually well-engineered: reference counting, error handling, NULL safety, GC traverse/clear, and version-guard discipline are all confirmed correct by dedicated agents with essentially zero real bugs in those categories — 1 confirmed reference leak (module-init error path, low practical severity) is the only FIX-tier finding across 13 independent audits. The extension's real risk is concentrated in one place: it declares Py_mod_gil = Py_MOD_GIL_NOT_USED (free-threading support, already shipped in the tagged 2.2.2 release) while having zero Py_BEGIN_CRITICAL_SECTION calls anywhere in the file, and several agents independently confirmed genuine unsynchronized-mutation races reachable from pure Python (see the companion FREE_THREADING_ANALYSIS.md in this repo for the full, dedicated writeup of the headline __wrapped__ use-after-free; this report adds three more instances of the same underlying pattern on other struct fields). A separate, empirically-verified parity audit (the agent actually built the extension and diffed behavior against the pure-Python fallback) found two real C-vs-Python behavioral divergences. The remaining findings are code-quality/maintainability opportunities (13× duplicated arithmetic-operator implementations, ~750 lines) and a stable-ABI feasibility assessment (moderate difficulty, would require bumping the floor from 3.9 to 3.10).

Extension Profile

  • Module: wrapt._wrappers (1 C file, 4928 lines)
  • Init style: multi-phase (PyModuleDef_Slot, Py_mod_exec), with Py_mod_multiple_interpreters (3.12+) and Py_mod_gil (3.13+) opt-ins
  • Python targets: >=3.9 (declared and verified effective floor — only one real version-guard gap found, see Finding 7)
  • Limited API: no (not claimed; feasibility assessed in Findings 11-15, 23, 25)
  • Types defined: 6 heap types (ObjectProxy, CallableObjectProxy, PartialCallableObjectProxy, _FunctionWrapperBase, BoundFunctionWrapper, FunctionWrapper), all via PyType_FromModuleAndSpec
  • Dual implementation: yes — pure-Python fallback in src/wrapt/wrappers.py, selected via src/wrapt/__wrapt__.py (parity audited, see Findings 9-10)
  • Git history: this checkout is a shallow clone (1 commit); git-history-analyzer could not mine fix-completeness/regression data — see its section below.

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety Y 1 0 wrapt_create_type leaks the type ref on PyModule_AddObject* failure (Finding 1); 77 other scanner hits triaged as false positives
Error Handling G 0 0 All 329 raw scanner hits triaged as false positives; error-path discipline confirmed correct
NULL Safety G 0 0 All scanner hits false positives (one scope-conflation artifact); raise_uninitialized_wrapper_error guard applied consistently ~50×
GIL Discipline Y* 0 5 Zero Py_BEGIN_CRITICAL_SECTION anywhere despite Py_MOD_GIL_NOT_USED; unlocked struct-field races on 3 more field groups beyond the documented __wrapped__ bug (Findings 3-6)
Resource Lifecycle G 0 0 All 6 heap types' tp_new/tp_dealloc/tp_traverse/tp_clear/weakref wiring confirmed correct
Module State Y 1 0 Same leak as Refcount Safety (Finding 1); otherwise 0 static globals, full traverse/clear coverage, subinterpreter-safe
Type Slots G 0 1 Partial-init NULL-deref via lazy __wrapped_factory__ + inconsistent == Py_None checks (Finding 2)
PyErr_Clear Safety G 0 1 16/19 clear sites correctly guarded; 3 use a sound WriteUnraisable-then-clear idiom; one polyfill NULL-without-exception gap (Finding 8)
ABI Compliance G 0 5 Not claimed; ~24 struct-access/macro sites would need replacement for abi3 (Findings 11-15), feasibility Moderate (Finding 25)
Version Compat G 0 2 One real gap: wrapt_is_proxy unconditionally uses deprecated PyErr_Fetch/Restore (Finding 7); rest of scanner output was false positives (including a bug in the toolkit's own deprecated_apis.json)
C/Python Parity Y* 0 2 __qualname__/__annotations__ stale in Python fallback vs. live in C (Finding 9); __enter__/__aenter__ arity mismatch (Finding 10) — both empirically verified
Complexity G - 5 No per-function hotspots (max score 2.8/5.0), but 13× duplicated inplace_* operators = ~750 lines of invisible-to-scoring duplication (Finding 16)

* GIL Discipline and C/Python Parity are marked Y despite 0 FIX-tier findings from their respective agents' own rubric, because the practical severity is high (confirmed use-after-free paths, real behavioral divergence) — see Tensions below.

Findings by Priority

Findings are numbered globally: FIX (1), then CONSIDER (2-21), then POLICY (22-25).

Must Fix (FIX) — 1

# Finding File:Line Agents
1 wrapt_create_type leaks the PyType_FromModuleAndSpec reference when PyModule_AddObjectRef/PyModule_AddObject fails — the WRAPT_ADD_TYPE macro return -1;s directly, skipping Py_DECREF(type) src/wrapt/_wrappers.c:4567-4612 module-state-checker, refcount-auditor (independently confirmed), error-path-analyzer (cross-checked)

Only reachable under MemoryError/dict-resize-failure during module init — low practical severity, but a genuine, deterministic bug on inspection. Suggested fix (both agents converged on the same shape):

type = PyType_FromModuleAndSpec(module, spec, bases);
if (type == NULL)
  return -1;
if (WRAPT_ADD_TYPE_FAILS(module, name, type))   /* don't return -1 from inside the macro */
{
  Py_DECREF(type);
  return -1;
}
*out = (PyTypeObject *)type;
return 0;

Should Consider (CONSIDER) — 20

# Finding File:Line Agent
2 Partial-init NULL dereference: raise_uninitialized_wrapper_error's lazy-__wrapped_factory__ success path only sets self->wrapped, leaving sibling fields (args/enabled/binding/parent/instance/owner) NULL; several call sites test them with == Py_None/!= Py_None instead of truthiness, so a NULL slips through _wrappers.c:288-375 (root), consumers at 3448 (unconditional, no guard at all), 3693/3695, 3804/3812/3859/3862-3864 type-slot-checker
3 FunctionWrapperBase/BoundFunctionWrapper fields (instance, wrapper, enabled, binding, parent, owner) are reassigned via reentrant tp_init with no lock, and read unguarded from ~13 call sites; concrete UAF: WraptBoundFunctionWrapper_call captures self->instance into a borrowed local at line 4237 then uses it at line 4258, racing a concurrent __init__ re-invocation that could free it _wrappers.c:3545-3576 (writer), 4119-4315 (sharpest reader) gil-discipline-checker
4 PartialCallableObjectProxy.args/.kwargs — same reentrant-init race, with a stale-size TOCTOU: PyTuple_GET_SIZE(self->args) computed once then self->args re-read in a loop, both without a captured strong reference _wrappers.c:3314-3331 (writer), 3448-3479 (reader) gil-discipline-checker
5 self->dict lazy-recreation in WraptObjectProxy_get_self_dict races with Py_CLEAR(self->dict) in tp_clear/dealloc — two threads racing post-clear can each allocate a dict, silently losing one _wrappers.c:2859-2870 vs. 525-547 gil-discipline-checker
6 init_called is a plain (non-atomic) int field read/written without synchronization — low-impact (wrong error message under a race, not memory corruption) but formally a data race _wrappers.c:16, 362, 399, 433, 452 gil-discipline-checker
7 wrapt_is_proxy uses PyErr_Fetch/PyErr_Restore (deprecated since 3.12) unconditionally, unlike the three tp_dealloc functions elsewhere in the same file which correctly version-gate to PyErr_GetRaisedException/PyErr_SetRaisedException on 3.12+ — an inconsistency against the file's own established convention, not a functional break _wrappers.c:233-284 (specifically lines 243, 280) version-compat-scanner (pyerr-clear-auditor independently confirmed the function's logic is otherwise correct on every exit path)
8 PyType_GetModuleByDef polyfill (Python 3.9/3.10 only) can return NULL on tp_mro == NULL without setting an exception, violating the C API's NULL-return contract; practically unreachable since tp_mro is populated by PyType_Ready for any live instance's type _wrappers.c:112-154, specifically 116-120 pyerr-clear-auditor
9 Empirically verified: ObjectProxy.__qualname__ (and __annotations__ on Python <3.14) is a live getset in C but a one-time snapshot copied into the Python fallback's instance __dict__ at __init__ time — mutating the wrapped object's __qualname__ after wrapping is visible through the C extension but not through the Python fallback C: _wrappers.c:2643-2674; Python: wrappers.py:193-218, 295-334 parity-checker (built the extension, diffed live)
10 Empirically verified: proxy.__enter__(extra_arg)/__aenter__ silently forward extra arguments to the wrapped object in C (METH_VARARGS | METH_KEYWORDS), but raise TypeError in the Python fallback (zero-arg signature) — an asymmetry the Python source itself doesn't have between __enter__/__exit__ that the C extension doesn't preserve C: _wrappers.c:2291-2317, 2351-2377; Python: wrappers.py:637-647 parity-checker (built the extension, diffed live)
11 Direct tp_mro/tp_dict field access while walking a type's MRO — not visible under Py_LIMITED_API (would fail to compile, not just misbehave) _wrappers.c:116, 193, 204, 236 stable-abi-checker
12 PyTuple_GET_ITEM/PyTuple_GET_SIZE (non-limited macros) used at 15 sites, several on wrapt's hot call-forwarding path _wrappers.c: MRO walks (122-252) plus call-forwarding at 3345-3463, 4187-4220 stable-abi-checker
13 Direct tp_alloc/tp_free slot field access in tp_new/tp_dealloc (the classic "don't hardcode the allocator, subclasses may override it" pattern — correct for a static type but not limited-API-legal) _wrappers.c:384, 546, 3414, 3660 stable-abi-checker
14 Direct tp_name field access for error messages and repr() _wrappers.c:151, 371, 575-576, 3897 stable-abi-checker
15 Direct tp_descr_get slot field access for descriptor-protocol pass-through; the limited-API replacement (PyType_GetSlot) only supports static types since 3.10, and wrapped here is often a plain function/method (a static type) — this specific replacement would force the version floor from 3.9 to 3.10 _wrappers.c:3818, 3824, 3893, 3902 stable-abi-checker
16 13 near-identical WraptObjectProxy_inplace_* operator implementations (~750 lines, ~15% of the file) differ only by dunder-name string and two PyNumber_* function pointers — the same refcount/error-path pattern gets independently scanner-flagged 13 times because it's copy-pasted 13 times _wrappers.c:1030-2196 (13 functions) c-complexity-analyzer
17 The ~13 non-inplace binary operators (add, subtract, multiply, …) follow the same smaller reducible pattern (~300 lines) _wrappers.c:607-661 and siblings c-complexity-analyzer
18 The enabled-gate check-and-short-circuit block (~30 lines) is duplicated verbatim between WraptFunctionWrapperBase_call and WraptBoundFunctionWrapper_call _wrappers.c:3693-3724 and 4142-4173 c-complexity-analyzer
19 WraptFunctionWrapperBase_descr_get (highest cyclomatic complexity in the file, 29) contains two ~35-line near-duplicate "resolve descriptor → resolve bound_type → construct" blocks _wrappers.c:3784-3942 c-complexity-analyzer
20 WraptFunctionWrapper_init's binding-classification cascade repeats the same 5-line PyObject_IsInstance-then-branch block 4 times _wrappers.c:4402-4545, specifically 4453-4491 c-complexity-analyzer
21 26 sites use the Py_INCREF(x); return x; idiom that Py_NewRef (3.10+, backed by pythoncapi-compat) would collapse to one line — only a net win if bundled with Finding 24's pythoncapi-compat adoption, since doing it manually at a 3.9 floor means adding 26 individual version guards _wrappers.c: ~26 sites, e.g. 1066-1067, 2868-2869 version-compat-scanner

Tensions

  • Shipped free-threading support vs. confirmed unsynchronized mutation. The Py_mod_gil = Py_MOD_GIL_NOT_USED declaration is not experimental — it's in the tagged 2.2.2 release this checkout is built from. Yet gil-discipline-checker confirmed (and this report's companion FREE_THREADING_ANALYSIS.md independently confirmed via manual source reading) that zero critical sections exist anywhere in the file, and the same unlocked-mutation pattern recurs across ObjectProxy.__wrapped__ (documented separately), FunctionWrapperBase's six fields (Finding 3, with a concrete UAF), and PartialCallableObjectProxy.args/.kwargs (Finding 4). This is the single highest-priority item in this whole review, and it's why GIL Discipline is marked Y in the metrics table above despite the agent's own rubric classifying every instance as CONSIDER rather than FIX — the CONSIDER classification reflects "requires a specific concurrent-mutation trigger," not "low severity if triggered" (the triggers described are ordinary API usage: reassigning .__wrapped__ or re-invoking __init__, not obscure edge cases).
  • Stable-ABI feasibility (Moderate) vs. the declared 3.9 floor. stable-abi-checker's feasibility assessment (Finding 25) recommends a 3.10 floor if abi3 is pursued, driven specifically by Finding 15 (tp_descr_get replacement needs PyType_GetSlot-on-static-types, 3.10+). version-compat-scanner independently confirmed the current effective floor is genuinely 3.9 with no gaps. Adopting abi3 as currently scoped means trading 3.9 support for wheel-count reduction — a maintainer call, not a code-correctness issue.
  • "No complexity hotspots" vs. "13 identical scanner hits." c-complexity-analyzer's per-function scoring found nothing above 2.8/5.0 (no hotspots by its own threshold), while refcount-auditor and error-path-analyzer's raw (pre-triage) scanner output shows the identical finding pattern independently flagged across all 13 inplace_* functions. These aren't in real disagreement — it's a scoring blind spot: per-function complexity metrics have no cross-function duplication signal, so ~750 lines of copy-pasted logic (Finding 16) is invisible to the score even though it's the single highest-leverage refactor in the file (a real bug in this pattern today would need finding and fixing in 13 places).

Policy Decisions (POLICY) — 4

# Finding
22 The lazy __wrapped_factory__/__wrapped_get__ initialization path (_wrappers.c:288-375) has a code comment explicitly stating __wrapped_get__ "should use a lock to protect against multiple threads trying to initialize the wrapped object at the same time" — but this responsibility is entirely pushed to Python-level subclass code with zero C-side enforcement, while the module simultaneously advertises full free-threading support.
23 The wrapt_slots[] module-slot table conditionally includes Py_mod_multiple_interpreters (3.12+) and Py_mod_gil/Py_MOD_GIL_NOT_USED (3.13+) gated on build-time PY_VERSION_HEX — correct for today's per-minor-version wheels, but stable-abi-checker flagged (medium-high confidence, not empirically verified this session) that this likely blocks a single .abi3.so binary from cleanly spanning both a low floor (3.9) and these newer opt-ins simultaneously; worth confirming against CPython's actual unknown-module-slot behavior before any abi3 migration work starts.
24 pythoncapi-compat is not currently used. Of the extension's 4 hand-rolled version-guard blocks, only 2 (PyObject_GetOptionalAttrString, WRAPT_ADD_TYPE's PyModule_AddObjectRef branch — together ~30 of ~113 guard lines) are actually covered by the library; the other 2 (PyType_GetModuleByDef polyfill, the PyErr_Fetch/GetRaisedException dance including Finding 7's fix) are explicitly listed as unsupported by pythoncapi-compat itself. Net benefit of adoption: modest, not a clear win either way.
25 Overall stable-ABI feasibility: Moderate. Every violation found (Findings 11-15) has a direct, viable limited-API replacement — none hit a dead end. The complicating factors are that most of the struct-access sites sit on wrapt's genuinely hot call-wrapping path (this library's entire value proposition is low per-call overhead, so replacements need benchmarking, not just mechanical substitution) and that Finding 15's fix plausibly forces the floor from 3.9 to 3.10. Recommended minimum version if pursued: 3.10.

Strengths

Confirmed correct/well-designed by direct agent verification, not just absence-of-findings:

  • Reference counting: 78 raw scanner hits, only 1 real (Finding 1). Ownership transfer via Py_SETREF/Py_XSETREF, borrowed-ref safety through Py_TYPE(self) across intervening calls, and stolen-ref handling around PyModule_AddObjectRef are all correct.
  • Error handling: 329 raw scanner hits triaged to zero real findings — every PyArg_Parse* checked, every AttributeError-as-fallback vs. genuine-error distinction correctly made, the raise_uninitialized_wrapper_error() == -1 idiom applied consistently across ~50 call sites.
  • GC lifecycle: all 6 heap types' tp_new/tp_init/tp_dealloc/tp_traverse/tp_clear verified correct, including the subtlety of chaining subclass tp_clear to the base's before adding subclass-owned fields, and exception-preserving Py_DECREF(tp) ordering in every dealloc.
  • Module state / subinterpreter readiness: zero static mutable PyObject* globals (independently verified by 3 different agents via direct grep, not just the scanner), full wrapt_traverse/wrapt_clear coverage of all 37 PyObject*-bearing module-state fields, correct per-interpreter state access via PyType_GetModuleByDef everywhere.
  • Version-guard correctness: every #if PY_VERSION_HEX guard in the file checked against real CPython behavior and found correctly valued — the scanner's own reference data (deprecated_apis.json) had a bug (PyType_FromModuleAndSpec's version_added was wrongly listed as 3.10 instead of 3.9) that version-compat-scanner caught and traced to root cause rather than trusting.
  • PyErr_Clear discipline: 16 of 19 clear sites correctly gated on PyErr_ExceptionMatches; the other 3 use a sound PyErr_WriteUnraisable-then-clear idiom (reports rather than silently swallows) for documented "cannot fail" invariants.
  • Free-threading structural groundwork: heap types (not static PyTypeObject), per-interpreter module state, and multi-phase init are all already in place — the gap is specifically the missing critical sections on top of an otherwise sound foundation, not a structural redesign.

Code Removal Opportunities

Checked <plugin_root>/data/deprecated_apis.json's code_removal_opportunities against the file's >=3.9 floor:

  • PyModule_AddType (3.10+): could replace the WRAPT_ADD_TYPE macro's PyType_FromModuleAndSpec + PyModule_AddObjectRef chain, but only for the no-custom-bases case — all 6 of wrapt's types pass explicit bases (built via PyTuple_Pack), so this doesn't directly apply without restructuring, and would need a 3.10 guard at the current 3.9 floor anyway. Not actionable now.
  • PyImport_ImportModuleAttrString (3.14+): would replace wrapt_exec's PyImport_ImportModule("wrapt.wrappers") + PyObject_GetAttrString + Py_DECREF(module) sequence (_wrappers.c:~4778-4787) — a real textual match, but at 3.14+ only, meaning a version-gated dual path for ~5-8 lines saved. Low priority; revisit once the floor moves past 3.14.
  • PyDict_GetItemRef (3.13+) / PyObject_HasAttrStringWithError (3.13+): neither pattern exists in this file (confirmed by null-safety-scanner and pyerr-clear-auditor — no raw PyDict_GetItem or PyObject_HasAttrString calls). Not applicable.

No net-positive, floor-compatible removal opportunities exist today; both real matches require raising the minimum Python version.

Recommended Action Plan

Immediate

  1. Fix Finding 1 — wrapt_create_type's reference leak on module-init failure (small, mechanical, low-risk fix).

Short-term (highest leverage first)

  1. Findings 3-6 — the free-threading unlocked-mutation races (FunctionWrapperBase fields, PartialCallableObjectProxy.args/.kwargs, self->dict, init_called). Given these share a root cause with the already-documented __wrapped__ bug (see FREE_THREADING_ANALYSIS.md), design one consistent locking mechanism (per-object critical section) rather than patching each field group separately.
  2. Finding 2 — the lazy-init NULL-dereference gap; either fully initialize sibling fields in the lazy path or switch the ambient == Py_None checks to truthiness checks consistently.
  3. Findings 9-10 — the two confirmed C/Python parity gaps; decide which behavior is canonical (live vs. snapshot for __qualname__; strict vs. permissive arity for __enter__) and make both implementations match.
  4. Finding 7 — version-gate wrapt_is_proxy's PyErr_Fetch/PyErr_Restore the same way the three dealloc functions already do, for consistency.
  5. Finding 16 — consolidate the 13 inplace_* operators via a shared static helper; this is the single highest-value code-quality fix in the file (turns 13 independent review/bugfix surfaces into 1).

Longer-term (POLICY)

  1. Finding 22 — either add C-side locking around the lazy __wrapped_get__ path or clearly document the thread-safety contract subclasses must uphold.
  2. Findings 23, 25 — before any stable-ABI migration work, confirm CPython's unknown-module-slot behavior (Finding 23) and decide whether a 3.10 floor bump (Finding 25) is acceptable in exchange for abi3 wheel-count reduction.
  3. Finding 24 — pythoncapi-compat adoption is a discretionary maintainability call, not correctness-driven; low urgency.

Companion document: FREE_THREADING_ANALYSIS.md in this repo has the full, standalone deep-dive on the ObjectProxy.__wrapped__ use-after-free (the headline free-threading bug this report's Findings 3-6 extend to other fields).

Before sharing this report externally (filing issues, opening a PR against wrapt upstream), reach out to the maintainer first about what shape they want it in — a report this size dropped unsolicited is a significant ask. If any finding here is treated as security-relevant, use responsible disclosure rather than a public issue.

C Extension Analysis Report

Extension: yarl._quoting_c

Scope: entire project (.)

Agents Run: type-slot-checker, gil-discipline-checker, resource-lifecycle-checker, parity-checker, module-state-checker, c-complexity-analyzer, version-compat-scanner, stable-abi-checker

refcount-auditor, error-path-analyzer, null-safety-scanner, and pyerr-clear-auditor were skipped by the toolkit's own Cython-mode calibration (95-100% false-positive rate on generator-emitted code, validated against the uvloop baseline with zero FIX-class loss). git-history-analyzer was skipped because the repository is a shallow clone with exactly one visible commit (git rev-parse --is-shallow-repository → true), which leaves it no signal to work with.

Executive Summary

The extension is structurally clean: zero FIX-class findings across every dimension that was run. There are no refcount, GIL, resource-lifecycle, or type-slot defects, and the module already declares free-threading compatibility correctly in its real build (see reports/ft_review.md). The one genuinely actionable signal is C/Python parity drift: _quoting_c.pyx and _quoting_py.py diverge in three ways despite AGENTS.md stating they "must stay behaviourally identical," and one of those divergences (non-ASCII characters in unsafe=) makes the pure-Python _Unquoter emit malformed percent-encoding on its own terms, independent of what the C side does. The other findings are minor code-quality and policy items: an unused legacy DEF constant, a small amount of RFC-3986 dead code shared by both implementations, two nesting hotspots in _do_unquote/_do_quote, and confirmed-negative policy notes on subinterpreter readiness, the exact Cython version pin, and abi3 feasibility.

Extension Profile

  • Module: yarl._quoting_c (1 hand-written .pyx file, yarl/_quoting_c.pyx, 452 lines; no committed generated .c)
  • Code generation: Cython 3.2.5 (exact pin, requirements/cython.txt)
  • Init style: single-phase-equivalent (CYTHON_USE_MODULE_STATE defaults to 0 in standard CPython builds, so Cython's PEP 489 multi-phase scaffolding is behaviourally inert)
  • Python targets: 3.10-3.14 (setup.cfg)
  • Limited API: no (PyUnicode_DATA/KIND/READ are not part of the stable ABI)
  • Types defined: 2 (_Quoter, _Unquoter, both cdef class)
  • Pure-Python fallback: yarl/_quoting_py.py, used under YARL_NO_EXTENSIONS=1, on non-CPython runtimes, or if the C import fails

Key Metrics

Dimension Status FIX CONSIDER Top Finding
Refcount Safety - skipped skipped not run (Cython-mode default skip)
Error Handling - skipped skipped not run (Cython-mode default skip)
NULL Safety - skipped skipped not run (Cython-mode default skip)
GIL Discipline G 0 0 freethreading_compatible already correctly set; see Finding 6 (POLICY)
Resource Lifecycle G 0 0 Writer struct malloc/realloc/free is exhaustively paired, no leaks
Module State G 0 2 Bit tables live as bare C globals outside any module-state struct (Findings 4, 5)
Type Slots G 0 0 No findings; both cdef classes fully initialize in __init__ and never mutate
PyErr_Clear Safety - skipped skipped not run (Cython-mode default skip)
ABI Compliance G 0 0 (1 POLICY) abi3 migration assessed and rejected, see Finding 8
Version Compat G 0 1 (1 POLICY) DEF BUF_SIZE is a deprecated-direction Cython construct (Finding 6*)
C/Python Parity Y 0 3 (+1 cleanup, +1 test-gap) Non-ASCII unsafe= makes the Python _Unquoter emit malformed %-escapes (Finding 1)
Complexity - - 2 _do_unquote and _do_quote both have deep conditional nesting

G = No FIX findings | Y = 1-3 FIX findings | R = 4+ FIX findings. No dimension reached FIX-tier in this review; the parity row is marked Y on the strength of Finding 1 being independently a correctness bug in the Python fallback, even though it is filed as CONSIDER rather than FIX (see Tensions).

* version-compat-scanner's DEF BUF_SIZE item and gil-discipline-checker's hardening note both land in the CONSIDER/POLICY split below with distinct finding numbers; the table cell above is a compressed pointer to both.

Findings by Priority

Must Fix (FIX) — 0

None. Every agent that ran returned either a clean result or CONSIDER/POLICY-tier observations.

Should Consider (CONSIDER) — 7

# Finding File:Line Agent
1 Non-ASCII characters in unsafe=/ignore= are silently accepted by _quoting_py.py's _Unquoter.__init__ (no encode-time check), but _quoting_c.pyx:329 eagerly does self._unsafe.encode('ascii') and raises UnicodeEncodeError at construction for the same input. Beyond the divergence, the Python path produces malformed output on its own terms: when the literal non-ASCII character later appears unescaped in the unquoted text, _do_unquote's hex(ord(ch)).upper()[2:] (_quoting_py.py:200, mirrored at _quoting_c.pyx:432) emits all hex digits of ord(ch) after a single %, e.g. _Unquoter(unsafe='€')('€') returns '%20AC' — four hex digits after one %, not a valid percent-triplet, and not the correct UTF-8 percent-encoding of the character (%E2%82%AC). Verified empirically under python3t.exe against _quoting_py.py directly. yarl/_quoting_py.py:130 (no validation) vs yarl/_quoting_c.pyx:329 (eager ValueError-equivalent); malformed-output site yarl/_quoting_py.py:200 / yarl/_quoting_c.pyx:432 parity-checker
2 _Quoter.__init__ validates safe=/protected= eagerly in C (yarl/_quoting_c.pyx:200,206: raises ValueError("Only safe symbols with ORD < 128 are allowed") for any character with ord(ch) > 127). The Python _Quoter.__init__ performs no such check; it stores safe/protected unvalidated and only fails — with a different exception type and an unrelated-looking message — the first time __call__ is invoked with a non-empty string, via safe.encode("ascii") (yarl/_quoting_py.py:55) raising UnicodeEncodeError. Verified empirically: _Quoter(safe='\xe9') constructs without error in the Python impl, then raises UnicodeEncodeError on first non-empty call. yarl/_quoting_py.py:23-34 vs yarl/_quoting_c.pyx:183-209 parity-checker
3 C's _Quoter.__call__/_Unquoter.__call__ coerce str subclasses via val = str(val) before processing (yarl/_quoting_c.pyx:216-217, yarl/_quoting_c.pyx:342-343), so the result is always a plain str. Python's __call__ never coerces; it keeps the original subclass instance throughout, and its unchanged-fast-path (if ret2 == val: return val, yarl/_quoting_py.py:116-117) returns that same subclass instance by identity. Verified empirically: passing a str subclass instance through the Python _Quoter with no characters needing quoting returns the original object (out is s is True, type(out) is the subclass), whereas C always returns a fresh plain str. yarl/_quoting_py.py:116-118 vs yarl/_quoting_c.pyx:211-220 parity-checker
4 Test-gap underlying Findings 1-3: tests/test_quoting.py::test_quote_str_like/test_unquote_str_like (lines 420-425) compare by ==, which is true for a str subclass regardless of whether the original object or a plain-str copy was returned, so they cannot catch Finding 3. The hypothesis fuzz tests test_fuzz__PyQuoter/test_fuzz__PyUnquoter (lines 507-516) construct only the Python classes with arbitrary safe=/protected=/unsafe=/ignore= text and assert solely that construction succeeds — by design they treat "any text constructs without error" as the expected Python behaviour, which is the divergence in Finding 2, and they never call __call__ on the fuzzed instance, so Finding 1's malformed-output bug is never exercised. There is no fuzz test parametrized across both _PyQuoter/_Quoter and _PyUnquoter/_Unquoter that would catch any of the three findings. tests/test_quoting.py:420-425,507-516 parity-checker (cross-checked against the test file directly)
5 Module-level C globals ALLOWED_TABLE/ALLOWED_NOTQS_TABLE (yarl/_quoting_c.pyx:59-60) live as bare cdef uint8_t[16] arrays outside any module-state struct, populated once by top-level module-exec code (yarl/_quoting_c.pyx:71-79). This is fine under the module's current single-phase-equivalent init (see Finding 9 in Strengths/Tensions) but is a structural item if yarl._quoting_c ever moves to true PEP 489 multi-phase init with CYTHON_USE_MODULE_STATE=1 for subinterpreter support: these tables would need to migrate into PyModule_GetState-backed storage. yarl/_quoting_c.pyx:59-79 module-state-checker
6 The module does not explicitly declare its multi-interpreter posture (no Py_mod_multiple_interpreters slot, no opt-in/opt-out of Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED). Low priority: per Finding 5's analysis, the module's only global state is deterministic and idempotent to recompute, so this is an omission rather than a latent bug. yarl/_quoting_c.pyx (module level) module-state-checker
7 _do_unquote (yarl/_quoting_c.pyx:348-445) and _do_quote (yarl/_quoting_c.pyx:250-309) both have deep conditional nesting — _do_unquote in particular interleaves percent-decode state (pct/buflen), UTF-8 incremental-decode retry-on-UnicodeDecodeError logic, and three different categories of unsafe/ignore/qs post-processing inside one while loop, reaching 4-5 levels of nesting. Both functions are maintainer-visible refactor targets, not bugs; c-complexity-analyzer's standard advice (extract the percent-triplet decode and the unsafe/ignore dispatch into helper cdef functions) applies, but the functions are correct as written and any split needs the Writer/ret-list state threaded through carefully. yarl/_quoting_c.pyx:250-309,348-445 c-complexity-analyzer

Tensions

  • Parity Finding 1 sits at the FIX/CONSIDER boundary. Its output is malformed regardless of what the C implementation does, which argues for FIX. It is filed as CONSIDER because reachability through yarl's own code is effectively nil: every _Unquoter built by yarl/_quoters.py (UNQUOTER, PATH_UNQUOTER, PATH_SAFE_UNQUOTER, QS_UNQUOTER, UNQUOTER_PLUS) is constructed with ASCII-only unsafe=/ignore= literals, and a user would have to call _Unquoter directly with an attacker- or caller-supplied non-ASCII unsafe= string to hit it. Treat it as the most urgent CONSIDER item rather than escalating it without a concrete call site that reaches it.
  • Which implementation is the reference? All three parity findings run one direction: C is stricter (eager ValueError, no subclass passthrough), Python is looser (lazy/no validation, subclass passthrough preserved, malformed output possible). C is the implementation that ships by default and that _quoting_py.py exists to stand in for, not the other way around, so the natural fix direction is to tighten Python to match C's eager-validation and plain-str-return behaviour, not to loosen C. AGENTS.md's "must stay behaviourally identical" requirement for these two files is the rule that makes this the right priority rather than a stylistic preference.
  • module-state-checker's two CONSIDER items (Findings 5, 6) vs. the module's actual size and risk. A 452-line, two-class extension with deterministic, idempotent module-level init has essentially no real exposure to subinterpreter problems today. These findings are correctly filed as low-priority structural notes, not anything requiring near-term action; they would only matter if yarl explicitly took on subinterpreter-per-interpreter-GIL support as a goal.
  • No disagreement between this run and reports/ft_review.md. The one apparent tension across the two reports — gil-discipline-checker's correction of an earlier freethreading_compatible finding in ft_review.md — was a self-inflicted tooling mistake (a grep -A5 window that cut off one line before the directive), not a disagreement between agents. It is fully resolved in ft_review.md; see Finding 8 below for the one residual pointer.

Policy Decisions (POLICY) — 1

# Finding
8 gil-discipline-checker confirmed pyproject.toml:51,110 already set freethreading_compatible = "True" in both Cython directive tables, correcting an earlier mistaken MIGRATE finding in reports/ft_review.md (root cause: a truncated grep -A5, not a code defect). No action needed here; see reports/ft_review.md for the full writeup. The one surviving optional idea is adding # cython: freethreading_compatible=True directly to the top of _quoting_c.pyx as defense-in-depth, in case the file is ever cythonized outside yarl's own pyproject.toml-driven build backend. Purely belt-and-suspenders.

Two further confirmed-negative policy notes from the run, kept out of the numbered findings because no action is implied or possible:

  • Exact Cython pin (version-compat-scanner). requirements/cython.txt pins cython==3.2.5 exactly rather than a range. This is a deliberate reproducibility choice (enforced via PIP_BUILD_CONSTRAINT/UV_BUILD_CONSTRAINT in pyproject.toml), comfortably past the Cython ≥3.1 version that introduced freethreading_compatible. No compatibility gap; noted for completeness.
  • abi3/stable-ABI migration (stable-abi-checker). Verdict: not worthwhile. _quoting_c.pyx uses PyUnicode_DATA, PyUnicode_KIND, and PyUnicode_READ (PEP 393 flexible-string-representation macros) throughout _do_quote_or_skip, _do_quote, and _do_unquote — none of these are part of the limited API / stable ABI, so an abi3 migration would require rewriting the hot path around PyUnicode_ReadChar/PyUnicode_GET_LENGTH-equivalent limited-API calls, for a 452-line extension that already builds fast per-version wheels. It also would not reduce the wheel matrix as much as it might elsewhere: free-threaded CPython builds do not support the limited API at all, so an abi3 wheel still could not cover the free-threaded target, and yarl already ships free-threading-compatible builds (Finding 8 / ft_review.md). Not recommended.

Strengths

  • Writer struct lifecycle is exhaustively correct. _init_writer/_release_writer/_write_char (yarl/_quoting_c.pyx:91-128) track heap_allocated_buf precisely: the initial buffer is a stack-allocated char[BUF_SIZE] owned by the caller, and only a PyMem_Malloc/PyMem_Realloc'd buffer is ever freed via PyMem_Free, gated on that same flag. _do_quote_or_skip calls _release_writer inside a finally block (yarl/_quoting_c.pyx:244-248), so the heap buffer (if any) is freed on every exit path, including exceptions raised mid-_do_quote.
  • Both cdef classes fully initialize in __init__ and never mutate afterward — _Quoter._safe_table/_protected_table and _Unquoter's _ignore/_unsafe/_quoter/_qs_quoter are all set once in __init__ and only read from every other method. This is what makes the nine shared singleton instances in yarl/_quoters.py safe to call concurrently from multiple threads, and is also why freethreading_compatible = True is a correct declaration rather than an aspirational one (cross-referenced from reports/ft_review.md).
  • No manual reference-count or NULL-check patterns to get wrong. Both classes are simple value-holding cdef classes with primitive (bint, uint8_t[16]) instance state; Cython's generated __dealloc__/tp_traverse/tp_clear for these classes need no hand-written analog, which is exactly why type-slot-checker came back with zero findings of any kind.
  • The free-threading declaration is already correct and already wired through the real build, not just present in the source tree: packaging/pep517_backend/_cython_configuration.py turns pyproject.toml's directive tables into the actual cythonize/cython CLI flags used for every real install.

Code Removal Opportunities

Checked <plugin_root>/data/deprecated_apis.json's code_removal_opportunities patterns against this codebase. None apply: the file is 452 lines with no PyType_FromSpec/PyType_Ready/PyModule_AddObject chains, no manual import-then-getattr-then-decref sequences, and no other pattern in the list appears in _quoting_c.pyx. The one cleanup opportunity found in this review is unrelated to the deprecated-API list:

  • Unused RFC 3986 delimiter constants, present in both implementations. GEN_DELIMS, SUB_DELIMS, and RESERVED (yarl/_quoting_c.pyx:17,19-20; mirrored at yarl/_quoting_py.py:8,10-11) are defined but never read anywhere else in either file — only SUB_DELIMS_WITHOUT_QS, UNRESERVED, ALLOWED, and QS actually feed the bit tables / safe string. Verified by grepping both files for each name. Curiously, even within this dead code the two files disagree: C's SUB_DELIMS adds '+?=;' while Python's adds "+&=;" (? vs &) — harmless today since neither SUB_DELIMS nor RESERVED is read, but worth fixing in the same cleanup pass so a future use of these constants doesn't inherit a silent C/Python mismatch. 3 lines removable per file (6 total) if confirmed dead by the maintainers; flagged by parity-checker, independently confirmed here by grep.

Recommended Action Plan

Immediate (FIX items)

None — there are no FIX-tier findings in this review.

Short-term (CONSIDER items)

  1. Finding 1 — tighten _quoting_py.py::_Unquoter to reject non-ASCII unsafe=/ignore= the same way _quoting_c.pyx does (eager encode/range check in __init__), closing the malformed-%-escape output path. Highest-priority CONSIDER item; closest to FIX (see Tensions).
  2. Finding 2 — move _quoting_py.py::_Quoter's safe=/protected= validation into __init__, matching C's eager ValueError instead of a lazy, differently-typed UnicodeEncodeError on first call.
  3. Finding 3 — decide intentionally whether str-subclass passthrough is desired behaviour; if not (matching C), coerce via str(val) at the top of Python's __call__.
  4. Finding 4 — add cross-implementation tests (parametrized over both _Quoter/_PyQuoter and _Unquoter/_PyUnquoter, the way most of tests/test_quoting.py already is) for non-ASCII safe=/unsafe=/protected=/ignore= and for str-subclass identity preservation, so Findings 1-3 are guarded against regression once fixed and so the existing fuzz tests stop encoding the divergence as expected behaviour.
  5. Finding 5/6 — no near-term action; revisit only if subinterpreter-per-interpreter-GIL support becomes an explicit yarl goal.
  6. Finding 7 — optional refactor of _do_unquote/_do_quote into smaller helper cdef functions if/when either function needs a behavioural change anyway; not worth a standalone PR on its own.

Longer-term (POLICY)

  1. Finding 8 — optional: add an in-file # cython: freethreading_compatible=True comment to _quoting_c.pyx as defense-in-depth (see reports/ft_review.md for the full context).
  2. Exact-Cython-pin and abi3-feasibility policy notes require no action; included above for completeness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment