Repository navigation
Tracking Issue: Adoption of CPython limited API #31913
Description
Description:
As discussed with @mattip, this issue tracks the long term effort of moving NumPy to the CPython limited API.
With CPython moving toward subinterpreters, free-threading, and forward-compatible wheels, adopting the limited API will give us more flexibility. This will be a gradual process, and the immediate goal is just to tackle the low-hanging fruit.
Important: This work must be entirely internal. It should not change performance, usability, or NumPy's own C-API.
Roadmap:
- Migrate the codebase on a module-by-module basis by replacing restricted C-API calls with
Py_LIMITED_APIalternatives and updating the build configuration to compile those newly compliant files with thePy_LIMITED_APImacro.
Expected Outcomes for this phase:
- Updating C files so many of them can define the
Py_LIMITED_APImacro and still compile. - Forward-compatible abi3 wheels.
Extension modules
| Module | Package | PR | Status |
|---|---|---|---|
_bounded_integers, _common, bit_generator, _mt19937, _philox, _pcg64, _sfc64, _generator, mtrand |
random |
#32112 | Merged |
_pocketfft_umath |
fft |
#31910 | Merged |
lapack_lite |
linalg |
#32127 | Merged |
_umath_linalg |
linalg |
#32127 | Merged |
_rational_tests |
_core |
#32174 | Merged |
_umath_tests |
_core |
#32201 | Merged |
_struct_ufunc_tests |
_core |
#32210 | Merged |
_operand_flag_tests |
_core |
#32211 | Merged |
_reduction_loop_tests |
_core |
#32256 | Merged |
_multiarray_tests |
_core |
#32251 | Merged |
_simd |
_core |
#32244 | Merged |
_multiarray_umath |
_core |
— | Not started |
Supporting infrastructure
| Change | PR | Status |
|---|---|---|
python.allow_limited_api build option and per-module limited_api: kwarg |
#31910 | Merged |
Raise the Limited API target to Python 3.13, retiring the PyMem_Raw* workaround |
#32112 | Merged |
CI: build the debug job with -Dpython.allow_limited_api=true and run the suite |
#32130 | Merged |
Deprecate numpy.linalg.lapack_lite |
#32015 | Merged |
Remaining work
- Port random, fft, and linalg
- Port all five umath test modules
- Land _multiarray_tests (ENH: Port
_multiarray_teststo the Limited API #32251). - Land _simd (ENH: Port
numpy._core._simdto the Limited API #32244) - Guard py_limited_api on free-threaded builds so that Meson fails early instead of hitting CPython's #error. Note that abi3t requires Python 3.15 .
- Adopt the Py_LIMITED_API+0 idiom in ufuncobject.h:186 and _pocketfft_umath.cpp to protect against valueless downstream #defines.
- Port _multiarray_umath. This is currently blocked by heap types which change performance and also change the external API (heap types cannot be base classes of static types).
Reacted by Clément Robert
Activity
da-woods commentedon Jul 8, 2026
Very quick drive-by comment: isolated module state isn't intrinsically linked with the Limited API. It's probably a good thing to have though.
cschanhniem commentedon Jul 8, 2026
da-woods commentedon Jul 8, 2026
prathamhole14 commentedon Jul 16, 2026
Todo List
See: ENH: Add per-extension Limited API toggle and port
_pocketfft_umathto be limited api compatible #31910 (review)prathamhole14 commentedon Jul 28, 2026
Should we have a CI check for LIMITED API builds, similar to PyWavelets/pywt#828 but only for adopted modules?
rgommers commentedon Jul 28, 2026
Adding a CI job, or adapting an existing one, seems fine to me. I think I prefer the latter, and propose the
debugjob inlinux.yml. Note that it needs to be a job usingspin; we cannot go throughmeson-pythonuntil all modules are converted.mattip commentedon Oct 2, 2026
Could you list which non-limited API functions are still used (besides static types)?
prathamhole14 commentedon Oct 2, 2026
Non-limited API still used in
_multiarray_umath, leaving out static types (I asked AI to find these out):PyTuple_GET_ITEM,PyTuple_SET_ITEM,PyTuple_GET_SIZE,PyList_GET_ITEM,PyList_SET_ITEM,PyList_GET_SIZE,PySequence_Fast_GET_SIZE,PySequence_Fast_GET_ITEM,PySequence_Fast_ITEMS,PyBytes_AS_STRING,PyBytes_GET_SIZE,PyFloat_AS_DOUBLE,PyDict_GET_SIZE,PyMemoryView_GET_BUFFER,PyMemoryView_GET_BASE,PyTupleObjectPy_SETREF,Py_XSETREFPy_UNICODE_ISSPACE,Py_UNICODE_ISALNUM,Py_UNICODE_ISALPHA,Py_UNICODE_ISDIGIT,Py_UNICODE_ISLOWER,Py_UNICODE_ISTITLE,Py_UNICODE_ISUPPER,Py_UNICODE_ISDECIMAL,Py_UNICODE_ISNUMERIC,PyUnicode_KIND,PyUnicode_1BYTE_KIND,PyUnicode_2BYTE_KIND,PyUnicode_4BYTE_KIND,PyUnicode_1BYTE_DATA,PyUnicode_2BYTE_DATA,PyUnicode_4BYTE_DATA,PyUnicode_IS_ASCII,PyUnicode_GET_LENGTH,PyUnicode_READ_CHAR,PyUnicode_FromKindAndData,PyUnicode_AsUTF8PyDateTime_IMPORT,PyDateTimeAPI,PyDateTime_Check,PyDateTime_CheckExact,PyDate_Check,PyDate_CheckExact,PyDelta_Check,PyDateTime_FromDateAndTime,PyDate_FromDate,PyDelta_FromDSU,PyDateTime_GET_YEAR,PyDateTime_GET_MONTH,PyDateTime_GET_DAY,PyDateTime_DATE_GET_HOUR,PyDateTime_DATE_GET_MINUTE,PyDateTime_DATE_GET_SECOND,PyDateTime_DATE_GET_MICROSECOND,PyDateTime_DATE_GET_TZINFO,PyDateTime_TimeZone_UTCPyContextVar_New,PyContextVar_Get,PyContextVar_Set,PyContextVar_ResetPyObject_Print,PyTraceMalloc_Track,PyTraceMalloc_Untrack,PyGILState_CheckPy_BEGIN_CRITICAL_SECTION,Py_END_CRITICAL_SECTION,PyMutex,PyMutex_Lock,PyMutex_UnlockPy_complex,PyComplex_AsCComplex,PyComplex_FromCComplexPyTuple_FromArray,PyDict_SetDefaultRef,PyDict_ContainsString,PyLong_FromUnicodeObject,PyLong_IsZero,PyMethod_New,PyObject_LengthHint,PyInterpreterState_Main,PyUnstable_Object_IsUniquelyReferenced,PyUnstable_Object_IsUniqueReferencedTemporary,_Py_HashDouble,Py_TPFLAGS_SEQUENCEadd_docstring):PyCFunctionObject,PyGetSetDescrObject,PyMemberDescrObject,PyMethodDescrObjectneutrinoceros commentedon Oct 2, 2026
why use AI when you have a perfectly good, specialized, reliable and deterministic tool for this job ? (
abi3audit)prathamhole14 commentedon Oct 2, 2026
Thanks for pointing that out, this tool wasn't at the top of my mind. From now on I will use it instead :)
da-woods commentedon Oct 2, 2026
Definitely a good suggestion, but it can be misleading if you haven't compiled with
Py_LIMITED_APIjust because some macros only expand to ABI3 compatible things when it's set. So don't get too hung up on the results before then.Edit addendum: it also won't find macros which again you'll only find when you actually try to compile it.
prathamhole14 commentedon Oct 3, 2026
The biggest group is the accessor macros, 380 places in
_multiarray_umath(PyTuple_GET_ITEM137,PyTuple_SET_ITEM103,PyTuple_GET_SIZE84 and some other).Before opening a PR which approach would you prefer?
npy_config.hunder#ifdef Py_LIMITED_APIlike#define PyTuple_GET_ITEM(op, i) PyTuple_GetItem(op, i)and so on. No call site changes and normal builds unchanged. Same pattern as ENH: Limited API fallbacks forPy_SETREFandpythoncapi-compat#32864NPY_TUPLE_GET_ITEMthat are the CPython macro on normal builds and the function under the Limited API plus a mechanical rename of the 380 call sites.#if defined(Py_LIMITED_API)blocks with error checks as done in ENH: Port_multiarray_teststo the Limited API #32251 this is a lot of change and prone to errors.I'd lean towards 1.