Skip to content

Instantly share code, notes, and snippets.

@devdanzin
Last active June 20, 2026 01:08
Show Gist options
  • Select an option

  • Save devdanzin/1736971107bea3aa04ce19db21c90845 to your computer and use it in GitHub Desktop.

Select an option

Save devdanzin/1736971107bea3aa04ce19db21c90845 to your computer and use it in GitHub Desktop.
OOM-0013: Abort: builtin breaks result/error contract in `_Py_BuiltinCallFastWithKeywords_StackRef` (`ceval.c:843`)
# Authoritative gdb backtrace (3.16_ft_debug_asan, commit 15d7406)
# Abort: assert((res != NULL) ^ (PyErr_Occurred() != NULL)) in
# _Py_BuiltinCallFastWithKeywords_StackRef (Python/ceval.c:843).
# A builtin C function called via the specialized
# CALL_BUILTIN_FAST_WITH_KEYWORDS path violated the CPython return/error
# contract under OOM (returned NULL without an exception set, or returned
# non-NULL with an exception lingering). Representative vehicle: timeit.py
# __init__ -> compile(setup, dummy_src_name, "exec").
python: Python/ceval.c:843: PyObject *_Py_BuiltinCallFastWithKeywords_StackRef(_PyStackRef, _PyStackRef *, int): Assertion `(res != NULL) ^ (PyErr_Occurred() != NULL)' failed.
Program received signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (...) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (...) at ./nptl/pthread_kill.c:89
#2 __GI___pthread_kill (...) at ./nptl/pthread_kill.c:100
#3 __GI_raise (sig=...) at ../sysdeps/posix/raise.c:26
#4 __GI_abort () at ./stdlib/abort.c:77
#5 __libc_message_impl (...) at ../sysdeps/posix/libc_fatal.c:138
#6 __libc_message_wrapper (...) at ../include/stdio.h:203
#7 __assert_fail (assertion=..., file=..., line=..., function=...) at ./assert/assert.c:37
#8 _Py_BuiltinCallFastWithKeywords_StackRef (callable=..., arguments=..., total_args=...) at Python/ceval.c:843 <- assert (res != NULL) ^ (PyErr_Occurred() != NULL)
#9 _PyEval_EvalFrameDefault (tstate=..., frame=..., throwflag=...) at Python/generated_cases.c.h:2603 <- _CALL_BUILTIN_FAST_WITH_KEYWORDS
#10 _PyEval_EvalFrame (...) at ./Include/internal/pycore_ceval.h:122
#11 _PyEval_Vector (...) at Python/ceval.c:2141
#12 PyEval_EvalCode (co=..., globals=..., locals=...) at Python/ceval.c:679
#13 run_eval_code_obj (...) at Python/pythonrun.c:1369
#14 run_mod (...) at Python/pythonrun.c:1472
# faulthandler Python stack (timeit vehicle):
# File ".../Lib/timeit.py", line 82 in __init__ <- compile(setup, dummy_src_name, "exec")
# NOTE: the abort only lands on ceval.c:843 when the compile() call site has
# been specialized to CALL_BUILTIN_FAST_WITH_KEYWORDS (the fuzzer's repeated
# oom_call loop warms it up). Without specialization the same contract
# violation is caught one frame deeper, in _Py_CheckFunctionResult
# (Objects/call.c:43, "a function returned NULL without setting an exception").

Abort: builtin breaks result/error contract in _Py_BuiltinCallFastWithKeywords_StackRef (ceval.c:843)

A specialized CALL_BUILTIN_FAST_WITH_KEYWORDS dispatch asserts the (res != NULL) ^ (PyErr_Occurred()) xor; under OOM, compile() returns NULL after an allocation failure swallows its MemoryError, so the debug assert fires and aborts.

AI Disclaimer: this gist was drafted by Claude Code, which also generated the reduced reproducer.

Crash report

The specialized CALL_BUILTIN_FAST_WITH_KEYWORDS bytecode dispatches METH_FASTCALL | METH_KEYWORDS builtins through _Py_BuiltinCallFastWithKeywords_StackRef(), which after the call asserts the CPython API contract (res != NULL) ^ (PyErr_Occurred() != NULL) (exactly one of "result" / "error set" must hold). Under OOM, the builtin compile() (reached from timeit.Timer.__init__ at Lib/timeit.py:82) violates that contract -- it returns NULL after an internal allocation failure without the error indicator surviving in the state the wrapper observes -- so the debug-only assert at Python/ceval.c:843 fires and aborts the interpreter.

Reproducer

import _testcapi, faulthandler
faulthandler.enable()

def hot():
    # Specialize this call site to CALL_BUILTIN_FAST_WITH_KEYWORDS.
    return compile("pass", "<timeit-src>", "exec")

for _ in range(2000):        # warm up so the CALL specializes to the builtin-fast path
    hot()

_testcapi.set_nomemory(19, 0)   # fail every allocation from #19 onward
try:
    hot()                       # -> _Py_BuiltinCallFastWithKeywords_StackRef -> assert
finally:
    _testcapi.remove_mem_hooks()

Deterministic at start=19 on the free-threaded debug+ASan build (and on the JIT debug build). The warm-up loop is load-bearing: the abort only lands on ceval.c:843 once the compile() call site is specialized to CALL_BUILTIN_FAST_WITH_KEYWORDS. Without specialization the identical contract violation is caught one frame deeper in _Py_CheckFunctionResult (Objects/call.c:43, "a function returned NULL without setting an exception"). The fuzzer reaches the specialized state naturally via its repeated oom_call sweep over timeit (whose Timer.__init__ calls compile).

Backtrace

#8  _Py_BuiltinCallFastWithKeywords_StackRef  Python/ceval.c:843              <- assert (res != NULL) ^ (PyErr_Occurred() != NULL)
#9  _PyEval_EvalFrameDefault                  Python/generated_cases.c.h:2603 <- _CALL_BUILTIN_FAST_WITH_KEYWORDS
#11 _PyEval_Vector                            Python/ceval.c:2141
#12 PyEval_EvalCode                           Python/ceval.c:679

faulthandler Python stack (timeit vehicle): File ".../Lib/timeit.py", line 82 in __init__ -> compile(setup, dummy_src_name, "exec").

Root cause

Python/ceval.c, _Py_BuiltinCallFastWithKeywords_StackRef() (L827):

    res = cfunc(PyCFunction_GET_SELF(callable_o), args_o, total_args, NULL);
    STACKREFS_TO_PYOBJECTS_CLEANUP(args_o);
    assert((res != NULL) ^ (PyErr_Occurred() != NULL));   /* L843 */
    return res;

The assert encodes the standard rule that a C callable must return non-NULL xor leave an exception set. The defect is not in ceval.c itself -- it is the called builtin. For the compile vehicle, builtin_compile_impl (Python/bltinmodule.c:844) runs the parser/compiler (_Py_CompileStringObjectWithModule, L963) which, on an allocation failure deep inside, returns NULL while the MemoryError it set does not survive in the indicator the wrapper checks (an inner cleanup/PyErr_Clear-style path swallows it, or normalization of the pending MemoryError itself fails). The result is res == NULL and PyErr_Occurred() == NULL, breaking the xor and aborting. This is an OOM-robustness bug in the builtin's error handling, surfaced (not caused) by the debug assert; the same family is hit by the dis and zipfile._path vehicles through other builtins on the same specialized dispatch path.

Suggested fix

The contract must be restored by the builtin, not by relaxing the assert. In builtin_compile_impl ensure every NULL-returning error path leaves an exception set (and every success path leaves none): on the error:/finally: exits, if the compiler returned NULL with no exception, set MemoryError (mirroring _PyErr_NoMemory), e.g.

finally:
    if (result == NULL && !PyErr_Occurred()) {
        PyErr_NoMemory();   /* defensive: never return NULL without an exception */
    }
    return result;

A broader fix is to audit _Py_CompileStringObjectWithModule / the AST-compile path so an OOM there always propagates a live exception. (The ceval.c:843 assert is correct and should stay.)

Notes

Found by OOM-injection fuzzing (set_nomemory). The crash is an abort only on builds with assertions compiled in: the free-threaded debug+ASan build and the JIT debug build both abort at the identical Python/ceval.c:843 assertion. The free-threaded release and upstream (GIL) builds define -DNDEBUG, so the assert is a no-op; there the same OOM path raises a clean MemoryError and exits normally (rc 1). Per the OOM-catalog convention for assert-based aborts, the non-debug builds are recorded as n/a.

Six fuzzer vehicles abort at the identical ceval.c:843 assertion (timeit x2, dis x2, zipfile._path x2); each merely calls a METH_FASTCALL | METH_KEYWORDS builtin (compile, etc.) on an already-specialized call site under OOM. Specialization is what routes the failure to ceval.c:843; the unspecialized variant of the same defect aborts one frame deeper in _Py_CheckFunctionResult (Objects/call.c:43) and is tracked as the related "function returned NULL without setting an exception" signature.

Versions

  • main (3.16.0a0, commit 15d7406); aborts on the free-threaded debug+ASan build and the JIT debug build. Free-threaded release and upstream builds: assertion compiled out, clean MemoryError (n/a).

Part of python/cpython#151763 — an umbrella tracking 35 OOM-related crash findings.

"""
Minimal reproducer: abort on
assert((res != NULL) ^ (PyErr_Occurred() != NULL))
in _Py_BuiltinCallFastWithKeywords_StackRef() when a builtin (compile)
breaks the return/error contract under OOM.
Affected: CPython 3.16.0a0 (main). Aborts on builds with assertions
compiled in: free-threaded debug+ASan and JIT debug.
Release builds (-DNDEBUG) raise a clean MemoryError (see Notes).
Crash: SIGABRT, Python/ceval.c:843
Assertion `(res != NULL) ^ (PyErr_Occurred() != NULL)' failed.
Requires: a debug build exposing _testcapi.set_nomemory.
Backtrace (gdb):
#8 _Py_BuiltinCallFastWithKeywords_StackRef Python/ceval.c:843
(assert (res != NULL) ^ (PyErr_Occurred() != NULL))
#9 _PyEval_EvalFrameDefault Python/generated_cases.c.h:2603
(_CALL_BUILTIN_FAST_WITH_KEYWORDS)
#11 _PyEval_Vector Python/ceval.c:2141
#12 PyEval_EvalCode Python/ceval.c:679
Root cause (Python/ceval.c + Python/bltinmodule.c):
The specialized CALL_BUILTIN_FAST_WITH_KEYWORDS opcode dispatches
METH_FASTCALL|METH_KEYWORDS builtins through
_Py_BuiltinCallFastWithKeywords_StackRef() (ceval.c:827), which asserts
the CPython contract after the call (ceval.c:843):
res = cfunc(self, args_o, total_args, NULL);
assert((res != NULL) ^ (PyErr_Occurred() != NULL));
A C callable must return non-NULL XOR set an exception. Under OOM the
builtin compile() (builtin_compile_impl, bltinmodule.c:844, via
_Py_CompileStringObjectWithModule) fails an internal allocation, returns
NULL, but the MemoryError does not survive in the indicator the wrapper
observes -> res == NULL AND PyErr_Occurred() == NULL -> assert fails.
The defect is in the builtin's OOM error handling, not in ceval.c; the
assert is correct and merely surfaces it.
Why the warm-up loop:
The abort lands on ceval.c:843 only once the compile() call site is
specialized to CALL_BUILTIN_FAST_WITH_KEYWORDS. The for-loop warms it up.
Without specialization the identical contract violation is caught one
frame deeper, in _Py_CheckFunctionResult (Objects/call.c:43,
"a function returned NULL without setting an exception").
The original fuzzer vehicles (timeit, dis, zipfile._path) reach the
specialized state naturally: the fuzzer calls each target repeatedly in
an OOM sweep, and timeit.Timer.__init__ (Lib/timeit.py:82) calls
compile(setup, dummy_src_name, "exec").
Likely fix: make builtin_compile_impl / the AST-compile path never return
NULL without a live exception under OOM (defensively PyErr_NoMemory() on the
finally path). Keep the ceval.c:843 assert.
Self-sweeping: `python repro.py` runs the trigger under set_nomemory(N, 0) for N in a
sweep, each in a FRESH subprocess (a fresh process avoids cache warm-up shifting the OOM
window), and stops at the first N that crashes. Needs a debug build (the check is compiled
out under NDEBUG). Bare trigger (fixed N=19):
import _testcapi
def hot(): return compile("pass", "<timeit-src>", "exec")
for _ in range(2000): hot() # specialize CALL -> BUILTIN_FAST_WITH_KEYWORDS
_testcapi.set_nomemory(19, 0)
hot() # -> assert -> SIGABRT
"""
import os
import sys
import subprocess
TRIGGER = r"""
import _testcapi
def hot():
# Specialize this call site to CALL_BUILTIN_FAST_WITH_KEYWORDS.
return compile("pass", "<timeit-src>", "exec")
# Warm up so the bytecode CALL specializes to the builtin-fast-with-keywords path.
for _ in range(2000):
hot()
_testcapi.set_nomemory({n}, 0) # fail every allocation from #{n} onward
try:
hot() # specialized call -> _Py_BuiltinCallFastWithKeywords_StackRef
# -> compile() fails under OOM -> assert -> SIGABRT
finally:
_testcapi.remove_mem_hooks()
"""
SIGNATURE = "Assertion `(res != NULL) ^ (PyErr_Occurred() != NULL)' failed."
def main():
env = {**os.environ, "ASAN_OPTIONS": "detect_leaks=0:abort_on_error=0"}
# env["PYTHON_GIL"] = "0" # not needed: this is an assert-based abort, not FT-only
for n in range(80):
out = subprocess.run([sys.executable, "-c", TRIGGER.format(n=n)],
capture_output=True, text=True, env=env)
if SIGNATURE in out.stdout + out.stderr:
print("reproduced at set_nomemory(%d, 0):" % n)
sys.stdout.write(out.stderr or out.stdout)
return 1
print("no crash in range(80); widen it for your build")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment