Skip to content

Instantly share code, notes, and snippets.

@devdanzin
Last active June 20, 2026 01:08
Show Gist options
  • Select an option

  • Save devdanzin/9bd9423256ea03c08231d1ebe542db89 to your computer and use it in GitHub Desktop.

Select an option

Save devdanzin/9bd9423256ea03c08231d1ebe542db89 to your computer and use it in GitHub Desktop.
OOM-0019: Abort: double-free in `_PyPegen_raise_error_known_location` (`pegen_errors.c:363`)
# Authoritative gdb backtrace (3.16_ft_debug_asan, commit 15d7406)
# Abort: _Py_NegativeRefcount at Include/refcount.h:520 (Py_XDECREF) ->
# "object has negative ref count" / "<object ... is freed>".
# The over-decref'd object is error_line in _PyPegen_raise_error_known_location:
# its reference was already consumed by the 'N' format in Py_BuildValue
# (pegen_errors.c:346), then the error: path decrefs it again (pegen_errors.c:363).
# The freed slab is reused for the pre-allocated MemoryError instance under OOM,
# hence "object type name: MemoryError".
./Include/refcount.h:520: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7bffb7acd980 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
object type name: MemoryError
Program received signal SIGABRT, Aborted.
#0 __pthread_kill_implementation ./nptl/pthread_kill.c:44
#3 __GI_raise ../sysdeps/posix/raise.c:26
#4 __GI_abort ./stdlib/abort.c:77
#5 fatal_error_exit Python/pylifecycle.c:3517
#6 fatal_error Python/pylifecycle.c:3742
#7 _Py_FatalErrorFunc Python/pylifecycle.c:3765
#8 _PyObject_AssertFailed Objects/object.c:3278
#9 _Py_NegativeRefcount Objects/object.c:275
#10 Py_DECREF ./Include/refcount.h:354
#11 Py_XDECREF ./Include/refcount.h:520 <- decref of already-freed error_line
#12 _PyPegen_raise_error_known_location Parser/pegen_errors.c:363 <- error: Py_XDECREF(error_line) (2nd decref)
#13 RAISE_ERROR_KNOWN_LOCATION Parser/pegen.h:196
#14 _Pypegen_set_syntax_error Parser/pegen_errors.c:404
#15 _PyPegen_run_parser Parser/pegen.c:960
# ... -> ast.parse / compile of invalid syntax under set_nomemory.
#
# Minimal reproducer (start=54) produces frames #8-#14 identically.
# The reference to error_line is consumed at pegen_errors.c:346 by the 'N'
# conversion in Py_BuildValue("(OnnNnn)", ..., error_line, ...); when that
# Py_BuildValue fails under OOM it still consumes the 'N' arg (do_mktuple/
# do_ignore in Python/modsupport.c), and the error: path then decrefs it again.

Abort: double-free in _PyPegen_raise_error_known_location (pegen_errors.c:363)

Py_BuildValue("(OnnNnn)", ...) consumes error_line via the steal-on-failure N format even when it fails under OOM; the error: path then Py_XDECREFs the freed error_line again, driving the refcount negative and tripping _Py_NegativeRefcount.

AI Disclaimer: this gist was drafted by Claude Code, which also generated the reduced reproducer.

Crash report

When the parser raises a located SyntaxError, _PyPegen_raise_error_known_location() passes the freshly built error_line string to Py_BuildValue("(OnnNnn)", ..., error_line, ...) using the reference-stealing N format. Py_BuildValue consumes the N argument's reference even when it fails (by design, to avoid leaks). Under OOM the build fails, the function does goto error, and the error: cleanup runs Py_XDECREF(error_line) on the already-freed object -- a double-free. On debug builds (Py_REF_DEBUG) the second decref drives the refcount negative and aborts via _Py_NegativeRefcount.

Reproducer

import ast, _testcapi, faulthandler
faulthandler.enable()
src = "x y z\n"                 # generic invalid syntax -> located SyntaxError
_testcapi.set_nomemory(54, 0)   # fail every allocation from #54 onward
try:
    ast.parse(src)              # error_line built OK, then Py_BuildValue fails
                                # -> N-arg already consumed -> error: decrefs it again
finally:
    _testcapi.remove_mem_hooks()

Deterministic at start=54 on the free-threaded debug+ASan build (and on the jit debug build at the same start/site). ast.parse is just a convenient way to reach the parser's error path; the OOM budget must be large enough for error_line to be built but small enough that the following Py_BuildValue allocation fails.

Backtrace

#11 Py_XDECREF                            Include/refcount.h:520   <- decref of already-freed error_line
#12 _PyPegen_raise_error_known_location   Parser/pegen_errors.c:363  <- error: Py_XDECREF(error_line)
#13 RAISE_ERROR_KNOWN_LOCATION            Parser/pegen.h:196
#14 _Pypegen_set_syntax_error             Parser/pegen_errors.c:404
#15 _PyPegen_run_parser                   Parser/pegen.c:960

_Py_NegativeRefcount reports <object ... is freed> with object type name: MemoryError -- the freed str slab is reused for the interpreter's pre-allocated MemoryError instance under OOM. The jit ASan trace pins both ends of the double-free: the consuming decref originates at pegen_errors.c:346 (the Py_BuildValue call) and the second decref at pegen_errors.c:363.

Root cause

Parser/pegen_errors.c, _PyPegen_raise_error_known_location():

    tmp = Py_BuildValue("(OnnNnn)", p->tok->filename, lineno, col_number,
                        error_line, end_lineno, end_col_number);   /* L346: 'N' steals error_line */
    if (!tmp) {
        goto error;                                                /* L348 */
    }
    ...
error:
    Py_XDECREF(errstr);
    Py_XDECREF(error_line);                                        /* L363: 2nd decref of error_line */
    return NULL;

The N format consumes the reference to error_line. By design Py_BuildValue consumes N arguments even on failure -- see do_mktuple()/do_ignore() in Python/modsupport.c and the comment at L263: "we can't bail immediately on error as this will leak refcounts on any 'N' arguments." When PyTuple_New(6) (or an earlier conversion) fails under OOM, error_line's sole reference is dropped inside Py_BuildValue, which then returns NULL. The function takes goto error, but error_line was never reset to NULL, so Py_XDECREF(error_line) at L363 decrefs the freed object again. This is a refcounting bug (double-free), not a missing init.

Suggested fix

Reset error_line to NULL immediately after handing it to Py_BuildValue, so the error path's Py_XDECREF is a no-op:

    tmp = Py_BuildValue("(OnnNnn)", p->tok->filename, lineno, col_number,
                        error_line, end_lineno, end_col_number);
    error_line = NULL;   /* 'N' consumed our reference, even on failure */
    if (!tmp) {
        goto error;
    }

Alternatively, pass error_line with the non-stealing O format and keep a single explicit Py_DECREF/Py_XDECREF for it on all paths.

Notes

Found by OOM-injection fuzzing (set_nomemory). The _Py_NegativeRefcount check is gated on Py_REF_DEBUG, so the abort fires on the debug builds (ft_debug_asan and jit, both at start=54, identical site pegen_errors.c:363). Release builds (ft_release, upstream) compile the check out, so a single trigger survives, but the double-free is still real and memory-unsafe there: a full start sweep on the upstream release build was observed to crash with a Segmentation fault. Per the OOM-catalog convention for refcount-assert aborts, the non-debug builds are recorded as n/a.

Three fuzzer vehicles all abort at the identical Include/refcount.h:520 negative-refcount check with a freed MemoryError: two reach the parser via ast.parse (pyclbr._create_tree), and one via multiprocessing.forkserver._handle_preload. Each merely parses/compiles source under OOM and hits the located-SyntaxError path.

This is distinct from OOM-0003 (an uninitialized-field assert in code_dealloc): same family (OOM during code/parse handling) but a different defect (a genuine double-free via the N-format steal-on-failure contract).

Versions

  • main (3.16.0a0, commit 15d7406); aborts on the free-threaded debug+ASan build and on the jit debug build (both Py_REF_DEBUG). Release/upstream builds: check compiled out (n/a); latent double-free (sweep segfault observed on upstream).

Part of python/cpython#151763 — an umbrella tracking 35 OOM-related crash findings.

"""
Minimal reproducer: abort on _Py_NegativeRefcount (Include/refcount.h:520,
"object has negative ref count") caused by a double-free of `error_line` in
_PyPegen_raise_error_known_location() when Py_BuildValue() fails under OOM.
Affected: CPython 3.16.0a0 (main), commit 15d7406.
The _Py_NegativeRefcount check is gated on Py_REF_DEBUG, so the
abort fires on debug builds (ft_debug_asan, jit). Release builds
(ft_release, upstream) compile it out; the underlying double-free
is still present there (latent memory-safety bug -- a cumulative
sweep was observed to segfault upstream).
Crash: SIGABRT, ./Include/refcount.h:520 (Py_XDECREF -> Py_DECREF ->
_Py_NegativeRefcount). "<object ... is freed>",
"object type name: MemoryError".
Requires: a debug build exposing _testcapi.set_nomemory (Py_REF_DEBUG).
Backtrace (gdb):
#11 Py_XDECREF ./Include/refcount.h:520
#12 _PyPegen_raise_error_known_location Parser/pegen_errors.c:363 (Py_XDECREF(error_line))
#13 RAISE_ERROR_KNOWN_LOCATION Parser/pegen.h:196
#14 _Pypegen_set_syntax_error Parser/pegen_errors.c:404
#15 _PyPegen_run_parser Parser/pegen.c:960
Root cause (Parser/pegen_errors.c, _PyPegen_raise_error_known_location):
tmp = Py_BuildValue("(OnnNnn)", p->tok->filename, lineno, col_number,
error_line, end_lineno, end_col_number); // L346
if (!tmp) {
goto error; // L348
}
...
error:
Py_XDECREF(errstr);
Py_XDECREF(error_line); // L363
The 'N' format steals/consumes the reference to `error_line`. Crucially,
Py_BuildValue consumes the 'N' argument even when it FAILS: on a mid-build
error do_mktuple()/do_ignore() in Python/modsupport.c still run do_mkvalue()
for the 'N' slot, which decrefs the argument (see the comment at
modsupport.c:263 "we can't bail immediately on error as this will leak
refcounts on any 'N' arguments"). Under OOM the PyTuple_New(6) (or an earlier
conversion) fails, so error_line's only reference is dropped inside
Py_BuildValue. Py_BuildValue returns NULL, the code does `goto error`, and
`Py_XDECREF(error_line)` at L363 decrefs the *already-freed* object a second
time -> negative refcount -> abort. (`error_line` is never reset to NULL
after L346.)
The freed str's memory is then reused for the interpreter's pre-allocated
MemoryError instance under OOM, which is why the reported "object type name"
is MemoryError and the object is flagged "is freed".
The OOM sweep is needed so that error_line is built successfully (start large
enough) while the subsequent Py_BuildValue allocation fails (start small enough).
Observed crash at start=54 on the FT debug+ASan build; jit reproduces at the
same start and same site.
Likely fix: set `error_line = NULL;` immediately after the Py_BuildValue() call
(before the `if (!tmp)` check) so the error: path's Py_XDECREF(error_line) is a
no-op, or pass error_line with the 'O' format and keep an explicit decref.
Self-sweeping: `python repro.py` runs the trigger under set_nomemory(N, 0) for N in a
sweep, each in a FRESH subprocess (a fresh process avoids cache warm-up shifting the OOM
window), and stops at the first N that crashes. Needs a debug build (the check is compiled
out under NDEBUG). Bare trigger (fixed N=54):
import ast, _testcapi
_testcapi.set_nomemory(54, 0)
ast.parse("x y z\n")
"""
import os
import sys
import subprocess
TRIGGER = r"""
import ast
import _testcapi
import faulthandler
faulthandler.enable()
# Generic "invalid syntax" drives _Pypegen_set_syntax_error ->
# RAISE_SYNTAX_ERROR_KNOWN_LOCATION -> _PyPegen_raise_error_known_location.
_testcapi.set_nomemory({n}, 0)
try:
ast.parse("x y z\n") # Py_BuildValue consumes error_line's ref on failure;
# error: path decrefs it again -> abort
finally:
_testcapi.remove_mem_hooks()
"""
SIGNATURE = "Include/refcount.h:520: _Py_NegativeRefcount"
def main():
env = {**os.environ, "ASAN_OPTIONS": "detect_leaks=0:abort_on_error=0"}
# env["PYTHON_GIL"] = "0" # ONLY if this bug is free-threading-only (it is not)
for n in range(120):
out = subprocess.run([sys.executable, "-c", TRIGGER.format(n=n)],
capture_output=True, text=True, env=env)
if SIGNATURE in out.stdout + out.stderr:
print("reproduced at set_nomemory(%d, 0):" % n)
sys.stdout.write(out.stderr or out.stdout)
return 1
print("no crash in range(120); widen it for your build")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment