Skip to content

Instantly share code, notes, and snippets.

@devdanzin
Last active June 20, 2026 01:08
Show Gist options
  • Select an option

  • Save devdanzin/34b633230f6d2301ba17dec195ffe4b7 to your computer and use it in GitHub Desktop.

Select an option

Save devdanzin/34b633230f6d2301ba17dec195ffe4b7 to your computer and use it in GitHub Desktop.
OOM-0009: Abort: stale `release1` flag trips an ownership assert in `replace` (`unicodeobject.c:10783`)
# Authoritative gdb backtrace from the free-threaded debug+ASan build
# (~/projects/3.16_ft_debug_asan_cpython/python), running the representative
# fuzzer vehicle ~/crashers/python-4/html-sigabrt-assertion/source.py.
python: Objects/unicodeobject.c:10783: PyObject *replace(PyObject *, PyObject *, PyObject *, Py_ssize_t): Assertion `release1 == (buf1 != PyUnicode_DATA(str1))' failed.
Program received signal SIGABRT, Aborted.
#3 __GI_raise ../sysdeps/posix/raise.c:26
#4 __GI_abort ./stdlib/abort.c:77
#7 __assert_fail ./assert/assert.c:37
#8 replace Objects/unicodeobject.c:10783 <- assert release1 == (buf1 != PyUnicode_DATA(str1))
#9 unicode_replace_impl Objects/unicodeobject.c:12586 (str.replace -> replace())
#10 unicode_replace Objects/clinic/unicodeobject.c.h:1002
#11 _PyCallMethodDescriptorFastWithKeywords_StackRef Python/ceval.c:883
#12 _PyEval_EvalFrameDefault Python/generated_cases.c.h:4205
#13 _PyEval_EvalFrame ./Include/internal/pycore_ceval.h:122
#14 _PyEval_Vector Python/ceval.c:2141
# faulthandler Python stack (from the vehicle's stdout) -- minimization hint:
# File ".../Lib/html/__init__.py", line 19 in escape # s.replace("&", "&amp;")
# File ".../source.py", line 400 in oom_call
# File ".../source.py", line 461 in <module> # html.escape(<UCS-2 string>)
# Faulting state: buf1 == NULL (unicode_askind/PyMem_New failed under OOM) while
# release1 is still 0 -> (buf1 != PyUnicode_DATA(str1)) is 1, so 0 == 1 fails.

Abort: stale release1 flag trips an ownership assert in replace (unicodeobject.c:10783)

When unicode_askind()'s PyMem_New fails under OOM, replace() leaves buf1 == NULL with release1 still 0; the cleanup path's assert(release1 == (buf1 != PyUnicode_DATA(str1))) then sees 0 == 1 and aborts. Triggered by s.replace("&", "&amp;") on a wider-than-ASCII string, as html.escape() does.

AI Disclaimer: this gist was drafted by Claude Code, which also generated the reduced reproducer.

Crash report

str.replace() (the C replace() helper) tracks three "owned scratch buffer" flags -- srelease / release1 / release2 -- that must stay consistent with their pointers sbuf / buf1 / buf2. When a substring or replacement string of a narrower PyUnicode kind has to be widened, the code overwrites the pointer with the result of unicode_askind() (a PyMem_New allocation) and only sets the matching release flag on the next line. Under OOM, unicode_askind() returns NULL; the pointer becomes NULL while the flag is still 0, and the goto error cleanup path then asserts release1 == (buf1 != PyUnicode_DATA(str1)) -- which is 0 == 1 -- aborting the interpreter.

Reproducer

import sys, _testcapi, faulthandler
faulthandler.enable()
s = "轘" * 4                 # UCS-2 (2-byte kind) string
_testcapi.set_nomemory(0, 0)  # fail every allocation from #0 onward
try:
    s.replace("&", "&amp;")   # ASCII "&" must widen to UCS-2 via unicode_askind ->
                              # PyMem_New fails -> buf1=NULL, release1=0 -> error -> assert
finally:
    _testcapi.remove_mem_hooks()

Deterministic at start=0 on builds with assertions enabled (FT debug+ASan, JIT debug). The input must be a wider kind than the ASCII substring so that replace() is forced down the unicode_askind widening path; the very first allocation it makes is the one that must fail. This is exactly what html.escape() does to a non-Latin-1 string (s.replace("&", "&amp;")), which is how the fuzzer found it.

Backtrace

#8  replace              Objects/unicodeobject.c:10783   <- assert release1 == (buf1 != PyUnicode_DATA(str1))
#9  unicode_replace_impl Objects/unicodeobject.c:12586   (str.replace)
#10 unicode_replace      Objects/clinic/unicodeobject.c.h:1002
#11 _PyCallMethodDescriptorFastWithKeywords_StackRef  Python/ceval.c:883
#12 _PyEval_EvalFrameDefault  Python/generated_cases.c.h:4205

Faulting state: buf1 == NULL (the unicode_askind/PyMem_New for the widened substring failed under OOM) while release1 is still 0, so (buf1 != PyUnicode_DATA(str1)) is 1 and the 0 == 1 assert fails. faulthandler's Python stack points at Lib/html/__init__.py:19 (s.replace("&", "&amp;")).

Root cause

Objects/unicodeobject.c, replace(). There are several copies of this pattern (L10583, L10592, L10605, L10608, L10646, L10655, L10662, L10671); the one the reproducer hits is L10644:

    if (kind1 < rkind) {
        /* widen substring */
        buf1 = unicode_askind(kind1, buf1, len1, rkind);   /* L10646: PyMem_New, may fail */
        if (!buf1) goto error;          /* L10647: buf1 == NULL, release1 still 0 */
        release1 = 1;                   /* L10648: only reached on success */
    }

unicode_askind() (L2406) allocates with PyMem_New and returns NULL (via PyErr_NoMemory()) under OOM. On that failure buf1 is set to NULL but release1 is never set to 1, so the flag/pointer invariant is broken. Every cleanup label -- done (L10756-10758), nothing (L10770-10772) and error (L10782-10784) -- then asserts:

    assert(srelease == (sbuf != PyUnicode_DATA(self)));
    assert(release1 == (buf1 != PyUnicode_DATA(str1)));   /* L10783: 0 == 1 -> abort */
    assert(release2 == (buf2 != PyUnicode_DATA(str2)));

With buf1 == NULL, (buf1 != PyUnicode_DATA(str1)) is 1 while release1 == 0, so the assertion fails. This is purely a stale-flag/assert bug: the subsequent if (release1) PyMem_Free(buf1) correctly does not free the NULL pointer, and the function would otherwise return NULL (a clean MemoryError) -- which is exactly what release builds do.

Suggested fix

Make the assertions tolerant of the OOM failure path, or (cleaner) keep the flag in sync by treating a NULL widened buffer as "not owned". The simplest robust fix is to guard each assert against the error pointer being NULL, e.g.:

    assert(release1 == (buf1 != NULL && buf1 != PyUnicode_DATA(str1)));

applied to all three flags at the done / nothing / error labels. (On the success paths buf1 is always non-NULL, so this does not weaken the invariant there.) Equivalently, set buf1 = PyUnicode_DATA(str1) immediately before goto error when unicode_askind returns NULL, so the flag/pointer pair stays consistent.

Notes

Found by OOM-injection fuzzing (set_nomemory). This is a build-agnostic C bug that surfaces only where the assert() is compiled in: it aborts on the free-threaded debug+ASan build and on the JIT debug build (both define assertions). On the FT release and upstream builds -DNDEBUG compiles the assert out, the desync is harmless (if (release1) PyMem_Free skips the NULL), and str.replace correctly raises MemoryError; those builds are therefore recorded as n/a (they do not crash, not even a segfault). Per the OOM-catalog convention for assert-based aborts, only assertion-enabled builds reproduce.

Fourteen fuzzer vehicles across many stdlib modules (html, tomllib, sqlite3.dump, _pylong, shlex, wsgiref.headers, xml.etree.ElementInclude, compression.gzip) all abort at the identical unicodeobject.c:10783 assertion -- each reaches str.replace on a wider-than-ASCII string under OOM, confirming the defect is in replace() itself rather than any one caller.

Versions

  • main (3.16.0a0, commit 15d7406); aborts on the free-threaded debug+ASan build and the JIT debug build. FT release / upstream builds: assertion compiled out, str.replace raises MemoryError cleanly (n/a).

Part of python/cpython#151763 — an umbrella tracking 35 OOM-related crash findings.

"""
Minimal reproducer: abort on assert(release1 == (buf1 != PyUnicode_DATA(str1)))
in replace() (str.replace) when a kind-widening allocation fails under OOM.
Affected: CPython 3.16.0a0 (main). Reproduces as an abort on builds with
assertions enabled (free-threaded debug+ASan, JIT debug). Release
builds define NDEBUG, so the assert is a no-op and str.replace
correctly raises MemoryError instead (see Notes).
Crash: SIGABRT, Objects/unicodeobject.c:10783
Assertion `release1 == (buf1 != PyUnicode_DATA(str1))' failed.
Requires: a build exposing _testcapi.set_nomemory.
Backtrace (gdb):
#8 replace Objects/unicodeobject.c:10783 (assert release1 ...)
#9 unicode_replace_impl Objects/unicodeobject.c:12586 (str.replace)
#10 unicode_replace Objects/clinic/unicodeobject.c.h:1002
#11 _PyCallMethodDescriptorFastWithKeywords_StackRef Python/ceval.c:883
Root cause (Objects/unicodeobject.c):
replace() (L10515) tracks three "owned scratch buffer" flags --
srelease/release1/release2 -- which must stay consistent with their
pointers (sbuf/buf1/buf2). When a substring/replacement of a narrower
PyUnicode kind must be widened, the code does e.g. (L10644):
if (kind1 < rkind) {
buf1 = unicode_askind(kind1, buf1, len1, rkind); // PyMem_New
if (!buf1) goto error; // buf1 is now NULL, release1 STILL 0
release1 = 1;
}
unicode_askind() allocates with PyMem_New and returns NULL under OOM.
On that failure buf1 becomes NULL while release1 is still 0, then
'goto error' jumps to the cleanup labels (error/nothing/done) which all
assert:
assert(release1 == (buf1 != PyUnicode_DATA(str1))); // L10783
Now release1 == 0 but (buf1==NULL) != PyUnicode_DATA(str1) is true (1),
so the assertion fails and the interpreter aborts. The same desync
hazard exists for srelease/sbuf and release2/buf2 (L10583/10592/10605/
10608/10646/10655/10662/10671).
Trigger here: html.escape's first step is "s.replace('&', '&amp;')".
For a non-Latin-1 (UCS-2) input string, the ASCII substring "&" (1-byte
kind) must be widened to 2-byte kind via unicode_askind. Failing that very
first allocation (start=0) hits the desync.
Observed crash at start=0 on the FT debug+ASan and JIT debug builds.
Self-sweeping: `python repro.py` runs the trigger under set_nomemory(N, 0) for N in a
sweep, each in a FRESH subprocess (a fresh process avoids cache warm-up shifting the OOM
window), and stops at the first N that crashes. Needs a debug build (the check is compiled
out under NDEBUG). Bare trigger (fixed N=0):
import _testcapi
s = "轘" * 4
_testcapi.set_nomemory(0, 0)
s.replace("&", "&amp;")
"""
import os
import sys
import subprocess
TRIGGER = r"""
import _testcapi
import faulthandler
faulthandler.enable()
# A UCS-2 (2-byte kind) string: forces the ASCII "&" substring to be widened
# inside replace() via unicode_askind() (a PyMem_New allocation).
s = "轘" * 4
_testcapi.set_nomemory({n}, 0) # fail every allocation from #N onward
try:
s.replace("&", "&amp;") # unicode_askind(str1) fails -> buf1=NULL, release1=0
# -> goto error -> assert release1 == ... -> SIGABRT
finally:
_testcapi.remove_mem_hooks()
"""
SIGNATURE = "release1 == (buf1 != PyUnicode_DATA(str1))"
def main():
env = {**os.environ, "ASAN_OPTIONS": "detect_leaks=0:abort_on_error=0"}
# env["PYTHON_GIL"] = "0" # ONLY if this bug is free-threading-only (it is not)
for n in range(80):
out = subprocess.run([sys.executable, "-c", TRIGGER.format(n=n)],
capture_output=True, text=True, env=env)
if SIGNATURE in out.stdout + out.stderr:
print("reproduced at set_nomemory(%d, 0):" % n)
sys.stdout.write(out.stderr or out.stdout)
return 1
print("no crash in range(80); widen it for your build")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment