Repository navigation
Soft deprecate PyUnicode C API modifying "immutable" str objects #111
Description
Python 3.14 added PyUnicodeWriter C API which is a safe way to create a Unicode object. IMO it's now time to soft deprecate the legacy way to create a str object using PyUnicode_New(): deprecate functions like PyUnicode_CopyCharacters() and PyUnicode_Resize().
The PyUnicode_New() API is more fragile. It's possible that a str object is "used" before it's fully initialized, which make following functions (modifying the str) fail or crash. It can also be exposed in Python by mistake before it's fully initialized.
Soft deprecated functions (6):
- PyUnicode_New()
- PyUnicode_CopyCharacters()
- PyUnicode_Fill()
- PyUnicode_Resize()
- PyUnicode_WRITE()
- PyUnicode_WriteChar()
PyUnicodeWriter has a different design than PyUnicode_New(): it's designed to append multiple strings (in different formats), whereas PyUnicode_New() gives a direct access to an array of characters in a specific format (Py_UCS1, Py_UCS2 or Py_UCS4: 1, 2 or 4 bytes per character).
If PyUnicodeWriter doesn't fit well with existing code, it's possible to allocate an array of characters, write into this array, and then call PyUnicode_FromKindAndData() to create a str object.
Another option is to create a UTF-8 encode string and then call PyUnicode_FromString().
I didn't propose a hard deprecation because I'm too shy to be the one starting such large migration. There are many C extensions relying on the deprecated API.
The pythoncapi-compat project can be used to get the Python 3.14 PyUnicodeWriter API on Python 3.6 - 3.13.
A code search in PyPI top 15,000 projects found 60 projects using these APIs:
Details
- MySQL-python (1.2.5)
- PyICU-binary (2.7.4)
- abi3info (2025.11.29)
- apsw (3.53.4.0)
- backports_datetime_fromisoformat (2.0.3)
- casadi (3.8.1)
- cffi (2.1.1)
- clevercsv (0.8.5)
- cython (3.3.0)
- ddtrace (4.15.0)
- debugpy (1.8.22)
- dnaio (1.2.4)
- gevent (26.9.0)
- habachen (0.6.0)
- hypothesmith (0.3.3)
- jiter (0.17.0)
- jpype1 (1.7.1)
- libipld (3.4.1)
- librt (0.15.0)
- markupsafe (3.0.3)
- minorminer (0.2.22)
- msgspec (0.21.1)
- mwparserfromhell (0.7.2)
- mypy (2.3.1)
- nanobind (3.0.1)
- nuitka (4.2.1)
- onnxoptimizer (0.4.2)
- orjson (3.12.0)
- ormsgpack (1.12.2)
- polyscope (2.6.1)
- pybase64 (1.5.0)
- pydemumble (0.1.3)
- pyfastx (2.3.1)
- pygeohash (3.5.1)
- pyhmmer (0.12.3)
- pyicu (2.16.2)
- pyinstrument (5.1.3)
- pyobjc_core (12.2.2)
- pyodbc (5.3.0)
- pyqt5_sip (12.19.0)
- pyqt6_sip (13.12.0)
- pysam (0.24.1)
- pysha3 (1.0.2)
- pywinpty (3.0.5)
- rcssmin (1.2.2)
- regex (2026.9.10)
- rjsmin (1.2.5)
- safe_pysha3 (1.0.5)
- serpyco_rs (1.22.0)
- simplejson (4.1.2)
- sip (6.16.1)
- sqlmap (1.10.9)
- stringzilla (5.1.2)
- symusic (0.6.0)
- ulid_transform (2.2.9)
- unicodedata2 (17.0.1)
- vercel_runtime (0.23.0)
- whenever (0.10.5)
- wxpython (4.3.1)
- xxhash (4.0.1)
Code search regex: (PyUnicode_New|PyUnicode_CopyCharacters|PyUnicode_Fill|PyUnicode_Resize|PyUnicode_WRITE|PyUnicode_WriteChar)\b.
See also the issue and the pull request.
Vote to soft deprecated the 6 PyUnicode C API functions:
Activity
zooba commentedon Sep 18, 2026
I don't think this ought to be necessary (simply saying "these APIs are better" should be sufficient), but there are plenty of tools out there that won't make their own judgements without us saying something official.
As long as it doesn't start emitting warnings at compile time, I have no problem.
encukou commentedon Sep 18, 2026
Sure.
Note that the
PyUnicodeObjectstruct is still public, you can allocate one and write to itsvoid *anyarray directly. But that still has use cases (which I still didn't propose a good API for, sorry), so it's good to leave it out of scope here.vstinner commentedon Sep 18, 2026
The deprecated APIs are one of the reasons blocking the ability to change how CPython stores strings internally, like the idea of using UTF-8 internally: faster-cpython/ideas#684. PyUnicodeWriter was designed to abstract the internal storage.
It would be better if C extensions don't access directly to PyASCIIObject, PyCompactUnicodeObject and PyUnicodeObject members. But I'm not sure how to do that. Just say "don't do that" in the documentation?
What are the use cases for that?
Long time ago, PyPy once asked to deprecate PEP 393 C API, since most of these APIs require the internal storage to be exactly as PEP 393 designed it (they don't abstract the internal storage).
But well, the scope of such change is wider. Let's stick to the 6 functions for now 😁
encukou commentedon Sep 21, 2026
Well, soft-deprecating them doesn't really help with that.
We can't remove them in the foreseeable future, so AFAICS, UTF-8 would need to be an additional format. The main blocker for that is
PyUnicode_KIND/PyUnicode_DATAbeing infallible.Creating subclasses of
strthat need additional per-instance data.Yup. Waiting on @erlend-aasland now :)
vstinner commentedon Sep 28, 2026
Ping @erlend-aasland
vstinner commentedon Oct 2, 2026
The C API Working Group approved the decision. I close the issue.