Skip to content

Instantly share code, notes, and snippets.

@devdanzin
Last active June 20, 2026 01:08
Show Gist options
  • Select an option

  • Save devdanzin/d3c1d06e95c006a320dbbfffcc210d52 to your computer and use it in GitHub Desktop.

Select an option

Save devdanzin/d3c1d06e95c006a320dbbfffcc210d52 to your computer and use it in GitHub Desktop.
OOM-0010: Abort: `assert(_PyErr_Occurred(tstate))` in `_PyEval_EvalFrameDefault` (`generated_cases.c.h:13817`)
# Authoritative backtrace — free-threaded debug+ASan build
# (~/projects/3.16_ft_debug_asan_cpython/python)
#
# Repro: profiling.sampling.sample.dump_stack(-5) under _testcapi.set_nomemory(5, 0)
# Assertion message (stdout / faulthandler):
#
# python: Python/generated_cases.c.h:13817:
# PyObject *_PyEval_EvalFrameDefault(PyThreadState *, _PyInterpreterFrame *, int):
# Assertion `_PyErr_Occurred(tstate)' failed.
# Fatal Python error: Aborted
#
# Current thread's Python stack (most recent call first):
# File ".../Lib/profiling/sampling/sample.py", line 102 in _new_unwinder
# File ".../Lib/profiling/sampling/sample.py", line 87 in __init__
# File ".../Lib/profiling/sampling/sample.py", line 547 in dump_stack
#
# NOTE: line 13817 here is this debug build's generated_cases.c.h; the catalog
# signature line is 13106 (the build that produced the fuzzer stdout). Both are
# the SAME construct: the eval loop's `error` LABEL, whose debug-only branch is
# assert(_PyErr_Occurred(tstate));
# (the NDEBUG branch instead raises SystemError: "error return without exception set").
Program received signal SIGABRT, Aborted.
#7 __assert_fail ./assert/assert.c:37
#8 _PyEval_EvalFrameDefault Python/generated_cases.c.h:13817 <- LABEL(error): assert(_PyErr_Occurred(tstate))
#9 _PyEval_EvalFrame ./Include/internal/pycore_ceval.h:122
#10 _PyEval_Vector Python/ceval.c:2141 (running _new_unwinder, sample.py:102)
#11 _PyObject_VectorcallDictTstate Objects/call.c:146
#12 _PyObject_Call_Prepend Objects/call.c:504
#13 call_method Objects/typeobject.c:3100
#14 slot_tp_init Objects/typeobject.c:11174 (SampleProfiler.__init__, Python)
#15 type_call Objects/typeobject.c:2484 (SampleProfiler(...) )
# The faulting frame (#8/#10) is the Python frame of profiling.sampling.sample._new_unwinder
# executing the CALL to _remote_debugging.RemoteUnwinder(...) at sample.py:102. That C-level
# constructor returns an error to the eval loop under OOM WITHOUT leaving a live exception, so
# the eval loop jumps to LABEL(error) with _PyErr_Occurred(tstate) false and the assert aborts.
#
# Confirmed via breakpoints on the crashing run: neither the Argument-Clinic wrapper
# (_remote_debugging_RemoteUnwinder___init__) nor its impl (_remote_debugging_RemoteUnwinder___init___impl)
# is entered — the failure happens in the type call / object-creation path before tp_init runs.

Abort: assert(_PyErr_Occurred(tstate)) in _PyEval_EvalFrameDefault (generated_cases.c.h:13817)

Under OOM, constructing _remote_debugging.RemoteUnwinder (via sample._new_unwinder) returns an error result to the eval loop with no live exception, so the LABEL(error) assert that an error must carry an exception fires.

AI Disclaimer: this gist was drafted by Claude Code, which also generated the reduced reproducer.

Crash report

The bytecode interpreter's central error label is taken whenever an opcode returns an error result. On debug builds it asserts the caller actually left a Python exception (assert(_PyErr_Occurred(tstate))); on release builds the same condition is converted into SystemError: "error return without exception set". Under OOM, the CALL of the C-extension type _remote_debugging.RemoteUnwinder(...) -- evaluated inside profiling.sampling.sample._new_unwinder (sample.py:102) -- returns an error to the eval loop without a live exception, so _PyErr_Occurred(tstate) is false and the interpreter aborts.

Reproducer

import faulthandler, _testcapi
import profiling.sampling.sample as sample
faulthandler.enable()
_testcapi.set_nomemory(5, 0)   # fail every allocation from #5 onward
try:
    sample.dump_stack(-5)      # __init__ -> _new_unwinder -> RemoteUnwinder(...) errors w/o exc
finally:
    _testcapi.remove_mem_hooks()

Deterministic at start=5 on the free-threaded debug+ASan build. dump_stack(-5) is just a convenient driver: SampleProfiler.__init__ (sample.py:87) calls _new_unwinder (sample.py:102), which constructs _remote_debugging.RemoteUnwinder; allocation #5 fails inside that constructor.

Backtrace

#8  _PyEval_EvalFrameDefault       Python/generated_cases.c.h:13817   <- LABEL(error): assert(_PyErr_Occurred(tstate))
#10 _PyEval_Vector                 Python/ceval.c:2141                 (running _new_unwinder, sample.py:102)
#13 call_method                    Objects/typeobject.c:3100
#14 slot_tp_init                   Objects/typeobject.c:11174          (SampleProfiler.__init__, Python)
#15 type_call                      Objects/typeobject.c:2484           (SampleProfiler(...) )

The faulting object is not a bad pointer: it is the absence of an exception. The eval loop holds an error result (the RemoteUnwinder(...) CALL returned NULL) but tstate->current_exception is NULL.

13817 is this debug build's generated_cases.c.h; the catalog signature line 13106 is the build that produced the fuzzer stdout. Same construct -- the eval loop's LABEL(error).

Root cause

Python/generated_cases.c.h, LABEL(error) in _PyEval_EvalFrameDefault:

LABEL(error) {
    _PyFrame_StackAssertInvalid(frame);
    #ifdef NDEBUG
    if (!_PyErr_Occurred(tstate)) {
        _PyErr_SetString(tstate, PyExc_SystemError,
                         "error return without exception set");
    }
    #else
    assert(_PyErr_Occurred(tstate));   /* <- aborts here */
    #endif
    ...
}

The assert is correct: it enforces the C-API contract that an error result must be accompanied by a live exception. The defect is in the callee. The CALL at sample.py:102 invokes _remote_debugging.RemoteUnwinder(...); under OOM that returns an error to the eval loop with no exception set.

Confirmed with gdb on the crashing run: neither the Argument-Clinic wrapper _remote_debugging_RemoteUnwinder___init__ nor its impl _remote_debugging_RemoteUnwinder___init___impl is ever entered, so the error originates in the type-call / object-creation path before tp_init runs. Objects/typeobject.c:type_call documents exactly this hazard at its top:

/* type_call() ... can clear it (directly or indirectly) and so the
   caller loses its exception */
assert(!_PyErr_Occurred(tstate));

Under OOM an exception that should accompany the construction failure is either never set or cleared before control returns to the eval loop, leaving the precise "error result, no exception" state the eval-loop assert is meant to catch.

This is a generic invariant violation: any callee that, under OOM, returns an error without setting an exception trips the identical assert. The fuzzer surfaced 13 vehicles with this exact signature across unrelated stdlib paths (see Notes) -- they are siblings sharing one signature, not one bug.

Suggested fix

Audit the OOM error paths of the implicated C constructor(s) so an exception is always live on the way out. For _remote_debugging.RemoteUnwinder, ensure every return -1 / NULL-return sets PyErr_NoMemory() (or otherwise leaves PyErr_Occurred() true) and never PyErr_Clear()s a pending MemoryError while bailing. The eval-loop assert should remain as-is -- it is correctly reporting a contract violation. (Hardening the eval loop to always synthesize a SystemError even on debug builds would only paper over the real callee bug.)

Notes

Found by OOM-injection fuzzing (set_nomemory). This is an assert-based abort on the FT debug build only; the FT release, JIT, and upstream builds compile the assert out and instead surface a clean MemoryError (exit rc 1) -- they do not segfault. Per the OOM-catalog convention for assert-based aborts, the non-debug builds are recorded as n/a.

Signature-cluster split (important). The catalog signature for this cluster is the generic eval-loop error-label assert (generated_cases.c.h:13106 / error return without exception set). 13 fuzzer vehicles match it, but they fall into several distinct underlying "error-without-exception" sites, grouped by Python frame:

  • _remote_debugging.RemoteUnwinder via profiling.sampling.sample._new_unwinder -- this report's representative (python-4/profiling_sampling_sample, python-5/profiling_sampling_sample).
  • profiling.sampling _sort_to_mode (python-5/profiling_sampling_cli, python-7/profiling_sampling_cli).
  • subprocess._args_from_interpreter_flags via multiprocessing.spawn / multiprocessing.util (python-5 x2, python-7 x3).
  • mimetypes._default_mime_types (python-5/mimetypes x2).
  • json.load (python-7/json).
  • builtins (python-7/builtins).

All 13 are listed as vehicles because they share the catalog signature, but the fix must address each callee's OOM error path; the representative root-caused above is the RemoteUnwinder constructor.

Versions

  • main (3.16.0a0, commit 15d7406); aborts on the free-threaded debug+ASan build at start=5. FT release / JIT / upstream builds: assert compiled out, condition surfaces as MemoryError (n/a).

Part of python/cpython#151763 — an umbrella tracking 35 OOM-related crash findings.

"""
Minimal reproducer: abort on assert(_PyErr_Occurred(tstate)) in the eval loop's
error path (Python/generated_cases.c.h, LABEL(error)) under OOM.
Affected: CPython 3.16.0a0 (main), commit 15d7406.
Crash: SIGABRT, Python/generated_cases.c.h (LABEL(error) in
_PyEval_EvalFrameDefault)
Assertion `_PyErr_Occurred(tstate)' failed.
(This build reports line 13817; the fuzzer build reports 13106 --
same construct, different generated line numbering.)
Requires: a free-threaded DEBUG build exposing _testcapi.set_nomemory.
On non-debug builds (NDEBUG) the assert is compiled out; the same
condition is turned into a clean MemoryError and the process exits
normally (rc 1), so only the debug build aborts.
What happens
------------
The eval loop reaches its central LABEL(error) -- taken when an opcode (here a
CALL) returns an error result -- and the debug build asserts that a Python
exception is actually set:
LABEL(error) {
...
#ifdef NDEBUG
if (!_PyErr_Occurred(tstate)) { // release: synthesize one
_PyErr_SetString(tstate, PyExc_SystemError,
"error return without exception set");
}
#else
assert(_PyErr_Occurred(tstate)); // debug: abort
#endif
...
}
Under OOM the failing opcode is the CALL of the C-extension type
``_remote_debugging.RemoteUnwinder(...)`` evaluated inside
``profiling.sampling.sample._new_unwinder`` (sample.py:102). That constructor
returns an error to the eval loop WITHOUT leaving a live exception, so
_PyErr_Occurred(tstate) is false at LABEL(error) and the assert fires.
Confirmed with gdb on the crashing run: neither the Argument-Clinic wrapper
(_remote_debugging_RemoteUnwinder___init__) nor its impl is ever entered -- the
error originates in the type-call / object-creation path before tp_init runs.
``Objects/typeobject.c:type_call`` even documents the hazard at its top:
"type_call() ... can clear it (directly or indirectly) and so the caller loses
its exception". Under OOM an exception raised during construction is cleared (or
never set) before control returns to the eval loop, leaving the "error result,
no exception" state the assert is meant to catch.
This is a generic invariant: *any* callee that returns an error without setting
an exception under OOM trips the same assert. The fuzzer found 13 vehicles that
share this exact signature across several unrelated stdlib paths (RemoteUnwinder,
json.load, subprocess._args_from_interpreter_flags, mimetypes._default_mime_types,
profiling _sort_to_mode, builtins) -- see meta.json "notes" for the split.
Suggested fix
-------------
Make ``_remote_debugging.RemoteUnwinder`` construction always leave an exception
set on every error path under OOM (and audit the other listed callees the same
way): on each ``return -1`` / NULL-return, ensure ``PyErr_NoMemory()`` /
``PyErr_Occurred()`` holds before returning, and do not ``PyErr_Clear()`` a
pending MemoryError while bailing out. The eval-loop assert itself is correct --
it is reporting a real contract violation by the callee.
Self-sweeping: `python repro.py` runs the trigger under set_nomemory(N, 0) for N in a
sweep, each in a FRESH subprocess (a fresh process avoids cache warm-up shifting the OOM
window), and stops at the first N that crashes. Needs a debug build (the check is compiled
out under NDEBUG). Bare trigger (fixed N=5):
import _testcapi
import profiling.sampling.sample as sample
_testcapi.set_nomemory(5, 0)
sample.dump_stack(-5) # __init__ -> _new_unwinder -> RemoteUnwinder(...) errors w/o exc
"""
import os
import sys
import subprocess
TRIGGER = r"""
import _testcapi
import profiling.sampling.sample as sample
_testcapi.set_nomemory({n}, 0)
try:
sample.dump_stack(-5) # __init__ -> _new_unwinder -> RemoteUnwinder(...) errors w/o exc
finally:
_testcapi.remove_mem_hooks()
"""
SIGNATURE = "Assertion `_PyErr_Occurred(tstate)' failed."
def main():
env = {**os.environ, "ASAN_OPTIONS": "detect_leaks=0:abort_on_error=0"}
# env["PYTHON_GIL"] = "0" # ONLY if this bug is free-threading-only
for n in range(80):
out = subprocess.run([sys.executable, "-c", TRIGGER.format(n=n)],
capture_output=True, text=True, env=env)
if SIGNATURE in out.stdout + out.stderr:
print("reproduced at set_nomemory(%d, 0):" % n)
sys.stdout.write(out.stderr or out.stdout)
return 1
print("no crash in range(80); widen it for your build")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment