Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809)

Note: PEP 803 and PEP 809 - which are two competing PEPs with large overlap to define a new Stable ABI that will work with both GIL-enabled and free-threaded interpreters - are still in Draft state. This issue tries to summarize what would be needed if one of them got accepted.

NumPy doesn't use the Limited C API and Stable ABI itself, but does support other projects targeting the Stable ABI while using the NumPy headers. Support for that was implemented about two years ago in gh-25531, shipped in NumPy 2.0. It isn't widely used yet, but seems to work fine.

I worked on PyWavelets using the Stable ABI recently (PyWavelets/pywt#828), and decided to try that with PEP 803 and CPython 3.15a5, given that there are branches of Cython (cython#7399) and meson-python (mgorny/meson-python#freethreading-limited-api).

After doing the plumbing to make PyWavelets use the new abi3t, I got an unpleasant surprise from NumPy:

     ....
     [24/32] Compiling C object pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o
      samu: job failed: cc -Ipywt/_extensions/_pywt.abi3.so.p [snip] D_Py_OPAQUE_PYOBJECT -DPy_LIMITED_API=0x030f0000 -o pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o -c pywt/_extensions/_pywt.abi3.so.p/_pywt.c
      In file included from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
                       from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
                       from pywt/_extensions/_pywt.abi3.so.p/_pywt.c:1164:
      /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarraytypes.h:654:9: error: unknown type name 'PyObject_HEAD'; did you mean 'PyObject_DEL'?

PyObject_HEAD went missing - and it's pretty widely used in NumPy:

$ rg PyObject_HEAD | wc -l
78

Which shouldn't have been so surprising, given it's spelled out in https://peps.python.org/pep-0803/#opaque-pyobject. But I didn't quite connect the dots between that section of the PEP and how widely the now-disappearing macros are used in NumPy internals:

$ rg PyObject_HEAD | wc -l
78
$ rg PyObject_EXTRA_INIT | wc -l
0
$ rg PyObject_HEAD_INIT | wc -l
5
$ rg PyObject_VAR_HEAD | wc -l
3
$ rg Py_SET_TYPE | wc -l
38
0
$ rg "\->ob_type" | wc -l
22
$ # sizeof(PyObject) may also be annoying, can't easily grep for that

EDIT: for other projects to use the Limited C API with the NumPy headers, we should count only in include/numpy/, which doesn't look as bad:

$ rg PyObject_HEAD | wc -l
36
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_EXTRA_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_HEAD_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_VAR_HEAD | wc -l
1
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg Py_SET_TYPE | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg "\->ob_type" | wc -l
1

And there's the "They [opaque PyObjects] cannot be embedded in other structures." That applies to PyArrayObject I'd think. That would need to all be replaced by PEP 697 APIs, which is probably not great for performance. And seems to require the work for heap types that we (as maintainers) wanted to avoid doing ourselves for subinterpreters - see gh-24755, the discussion there is probably relevant.

The PR to allow use of the Limited API with NumPy headers from 2 years ago was pretty straightforward, however this looks like quite the project, with potential performance implications.

Taking a step back, the main two changes that are common between PEP 803 and 809 and affect NumPy are:

Use PyModExport as the new entry point for C extension modules, see PEP 793.
Support opaque PyObject (no separate PEP, see https://peps.python.org/pep-0803/#opaque-pyobject for details).

(1) seems straightforward, (2) not so much.

We've already done a decent amount of "internals hiding" ourselves of course, so perhaps in the public API it's not terrible? E.g., the PyArrayObject definition is already opaque:

numpy/numpy/_core/include/numpy/ndarraytypes.h

Lines 829 to 836 in 760778b

    
            * PyArrayObject field access is deprecated as of NumPy 1.7. 
        
            */ 
        
           typedef PyArrayObject_fields PyArrayObject; 
        
           #else 
        
           typedef struct tagPyArrayObject { 
        
                   PyObject_HEAD 
        
           } PyArrayObject; 
        
           #endif

Anyone want to hazard a guess about how much work it would be to avoid PyObject_HEAD et al. in public headers without trying to get rid of it from NumPy internals?

@rgommers

@rgommers any chance you can push your WIP branch of pywavelets somewhere? I'd like to try an experiment with the NumPy headers.

E.g., the PyArrayObject definition is already opaque:

Yeah, I bet we could add a new if statement there that makes PyArrayObject opaque on the opaque PyObject ABI. Hopefully close to no one is accessing e.g. ob_type.

PyArrayObject is almost the simplest one because it doesn't extend PyObject. All the scalar types have a field after PyObject_HEAD containing the scalar data. We also publicly expose the array iterator type, the descriptor type, and the ufunc object type which all extend PyObject. For all those I guess we move the fields into their own structs, expose functions to get the field structs or accessor functions to get each field. We'll also need to make all the static inline functions for PyArrayFields use actual functions in the C API to get the fields.

PyObject_VAR_HEAD is used by the np.void scalar and would need to a similar field struct and accessor function pattern to what I describe above on the limited API.

The use of ob_type is in PyArray_CheckExact:

± rg ob_type
ndarrayobject.h
35:#define PyArray_CheckExact(op) (((PyObject*)(op))->ob_type == &PyArray_Type)

I think we can just use Py_TYPE here. In fact we can just do that right now: #30705

Yeah, I bet we could add a new if statement there that makes PyArrayObject opaque on the opaque PyObject ABI. Hopefully close to no one is accessing e.g. ob_type.

Yeah, that sounds reasonable, the PEP is a bit sparse about it, but I would think PyObject_GetTypeData() probably already works if it returned a PyArrayObjects_FieldsOnly struct that omits the PyObject * part?
We could also add a convenience function for that to our API table or header (if that makes things easier).

I.e. I am thining that we do something like:

// PyArrayObject_fields will include PyObject when not opaque, but omit it otherwise.

static inline PyArrayObject_fields * PyArray_GetFields(PyArrayObject *arr) {
#ifdef NEW_STABLE_API
    // new public API table function or use PyObject_GetTypeData
    // if we need a public API function, will need NumPy version checks here also.
    return _PyArray_GetFields();
#else
    return (PyArrayObject_fields *)arr;
#endif
}

static inline size_t PyArray_NDIM(PyArrayObject *arr) {
    return PyArray_GetFields(arr)->ndim;
}

(Unlike PyObject_GetTypeData() we cannot return an error, I guess. The user has to pass a PyArrayObject *)

Which seems relatively doable and probably even helps us in theory convert at some point. But the priority here isn't converting NumPy (yet), but just enabling this, which seems like churn, but not too bad?

(Yeah, we'll have to make sure to replace every PyArrayObject_fields * cast in the public headers meticulously.)

any chance you can push your WIP branch of pywavelets somewhere?

Sure, here is how to reproduce:

Build/install a recent Python 3.15 alpha (or main branch), ensure it's the python3.15t on your PATH, optionally use a venv, ensure you have non-Python build deps like C/C++ compilers and ninja installed globally, and then:

$ NO_CYTHON_COMPILE=true python3.15t -m pip install git+https://github.com/cython/cython.git@freethreading-limited-api-preview
$ python3.15t -m pip install git+https://github.com/mgorny/meson-python.git@freethreading-limited-api
$  # Install `numpy`, from main or a branch with changes to headers (don't use an editable install):
$ python3.15t -m pip install numpy --no-build-isolation
$ # Now try to install PyWavelets, will fail on missing PyObject_HEAD
$  python3.15t -m pip install git+https://github.com/rgommers/pywt.git@freethreading-limited-api --no-build-isolation

# For easier iteration and not getting swamped with different errors, instead of the above
# one-shot command, clone PyWavelets locally and build with -j1:
$ git clone [email protected]:rgommers/pywt.git
$ cd pywt
$ git checkout freethreading-limited-api
$ python3.15t -m pip install . --no-build-isolation -Ccompile-args="-j1"

@da-woods

Thanks, that works! I'm attempting to see what happens if I make all these types opaque on the opaque PyObject API. Surprisingly, this is the only issue I run into from converting all the scalars:

      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:20021:52: error: incomplete definition of type 'PyDatetimeScalarObject' (aka 'struct PyDatetimeScalarObject')
       20021 |   __pyx_r = ((PyDatetimeScalarObject *)__pyx_v_obj)->obval;
             |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
      /Users/goldbaum/.pyenv/versions/3.15t-dev/lib/python3.15t/site-packages/numpy/_core/include/numpy/arrayscalars.h:189:16: note: forward declaration of 'struct PyDatetimeScalarObject'
        189 | typedef struct PyDatetimeScalarObject PyDatetimeScalarObject;
            |                ^
      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:20055:53: error: incomplete definition of type 'PyTimedeltaScalarObject' (aka 'struct PyTimedeltaScalarObject')
       20055 |   __pyx_r = ((PyTimedeltaScalarObject *)__pyx_v_obj)->obval;
             |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

So I guess Cython needs updates (ping @da-woods - you're probably interested in this issue 🙈). But I guess that's not surprising. And also not that bad, we just need the accessor functions described above, and apparently only for datetimes, at least from Cython's perspective.

Let's see how annoying it is for the array, iterator, descriptor, and ufunc object types...

Oh wait, I was missing some errors because clang's error limit. I also see errors like this:

      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:24634:3: error: invalid application of 'sizeof' to an incomplete type 'PyObject' (aka 'struct _object')
       24634 |   sizeof(PyObject), __PYX_GET_STRUCT_ALIGNMENT_3_3_0a0(PyObject),
             |   ^     ~~~~~~~~~~
      /Users/goldbaum/.pyenv/versions/3.15t-dev/include/python3.15t/pytypedefs.h:18:16: note: forward declaration of 'struct _object'
         18 | typedef struct _object PyObject;
            |                ^
      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:24634:21: error: invalid application of 'alignof' to an incomplete type 'PyObject' (aka 'struct _object')
       24634 |   sizeof(PyObject), __PYX_GET_STRUCT_ALIGNMENT_3_3_0a0(PyObject),
             |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:2929:47: note: expanded from macro '__PYX_GET_STRUCT_ALIGNMENT_3_3_0a0'
       2929 | #define __PYX_GET_STRUCT_ALIGNMENT_3_3_0a0(s) alignof(s)
            |                                               ^      ~~~
      /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/stdalign.h:21:17: note: expanded from macro 'alignof'
         21 | #define alignof _Alignof
            |                 ^
      /Users/goldbaum/.pyenv/versions/3.15t-dev/include/python3.15t/pytypedefs.h:18:16: note: forward declaration of 'struct _object'
         18 | typedef struct _object PyObject;
            |                ^

Which I guess implies that Cython needs to know the alignment of PyObject subtypes?

Based on a quick look, I don't think you're using the WIP Cython branch that supports this (see cython/cython#7399 for a description).

The alignment for example is a sanity check on cimport, but I've turned it off for the "opaque objects" mode because it isn't really possible

That looks promising!

Which I guess implies that Cython needs to know the alignment of PyObject subtypes?

PEP 803 does state that that's no longer possible: Their size and alignment will not be available. Expressions such as sizeof(PyObject) will no longer work.

Based on a quick look, I don't think you're using the WIP Cython branch that supports this (see cython/cython#7399 for a description).

Thanks for the hint! I'll double-check that.

      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:20021:52: error: incomplete definition of type 'PyDatetimeScalarObject' (aka 'struct PyDatetimeScalarObject')
       20021 |   __pyx_r = ((PyDatetimeScalarObject *)__pyx_v_obj)->obval;
             |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
      /Users/goldbaum/.pyenv/versions/3.15t-dev/lib/python3.15t/site-packages/numpy/_core/include/numpy/arrayscalars.h:189:16: note: forward declaration of 'struct PyDatetimeScalarObject'
        189 | typedef struct PyDatetimeScalarObject PyDatetimeScalarObject;
            |                ^
      pywt/_extensions/_cwt.abi3.so.p/_cwt.c:20055:53: error: incomplete definition of type 'PyTimedeltaScalarObject' (aka 'struct PyTimedeltaScalarObject')
       20055 |   __pyx_r = ((PyTimedeltaScalarObject *)__pyx_v_obj)->obval;
             |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

So these are coming from

numpy/numpy/__init__.cython-30.pxd

Lines 1056 to 1070 in 070954a

    
           cdef inline npy_datetime get_datetime64_value(object obj) noexcept nogil: 
        
               """ 
        
               returns the int64 value underlying scalar numpy datetime64 object 
        
               Note that to interpret this as a datetime, the corresponding unit is 
        
               also needed.  That can be found using `get_datetime64_unit`. 
        
               """ 
        
               return (<PyDatetimeScalarObject*>obj).obval 
        
           cdef inline npy_timedelta get_timedelta64_value(object obj) noexcept nogil: 
        
               """ 
        
               returns the int64 value underlying scalar numpy timedelta64 object 
        
               """ 
        
               return (<PyTimedeltaScalarObject*>obj).obval

That's coming pretty directly from your code unfortunately - i.e. Cython doesn't know that PyDatetimeScalarObject and PyTimedeltaScalarObject are Python objects - it's just a struct cast.

I'm planning to add an additional type of cast in Cython to help with this - we need something like that for our own memoryview code for example, so I was going to make it "public". I haven't done that yet.

There's a couple of complications though:

You can't use PyObject_GetTypeData without the GIL and the two functions are nogil functions. I can work around that for Cython internal types, but it isn't so obvious how to do it for external types.
To safely access the members you need to know whether the two types are defined as opaque types or not. The memory layout is different if not. Unfortunately there isn't a good way to tell how they were defined after the fact. Therefore the access to the struct will really need hiding inside Numpy to ensure all users have a consistent view of it (i.e. if I'm compiling pywt assuming that Numpy is using opaque types, but Numpy was built without opaque types, it ends in disaster).

Therefore the access to the struct will really need hiding inside Numpy to ensure all users have a consistent view of it

I think this is what I was referring to above about adding new field accessor functions to NumPy's C API.

@da-woods

One tricky thing to add setters is that it was implemented recently in cython so it would require bumping the minimum cython version.

Ah, good point, I forgot about that. One way to deal with it would be to use Cython's mechanism for supporting multiple Cython versions: https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#versioning

That said, I'd rather not do that because it'll duplicate almost everything else and that's a recipe for bitrot. @da-woods do you know of a way that NumPy can handle this without breaking backward compatibility with older Cython versions and without needing to maintain two copies of our Cython headers?

One option would be to build the two versioned pxd files programmatically using tempita (which I think Numpy vendors? Unless it's scipy I'm thinking of?) or some other simple templating mechanism.

Another option would be to use the include "filename" mechanism to build the pxd files from shared and non-shared chunks. Me feeling is that this will be messy though

(which I think Numpy vendors? Unless it's scipy I'm thinking of?

both:)

It's a bit of a pain though, Cython makes it quite cumbersome to deal with code generation, so let's first make sure we have no better option.

To make sure: are we talking about shipping a numpy/__init__.cython-33.pxd next to numpy/__init__.cython-30.pxd?

To make sure: are we talking about shipping a numpy/__init__.cython-33.pxd next to numpy/__init__.cython-30.pxd?

Yeah - that's what I was thinking, but using tempita to generate the two files from something common.

To make sure: are we talking about shipping a numpy/__init__.cython-33.pxd next to numpy/__init__.cython-30.pxd?

Yup, that's what I had in mind.

We just discussed that leaving out the settings for now might be the easiest option, given that they're unlikely to be needed.

I created #30994 to add getters for cython

@kumaraditya303

as I told Nathan in astropy/astropy#19249, I'm game to convert a bunch of static PyTypes to heap types in numpy, as I have experience doing the same thing in astropy, if it helps moving this forward. Where should I push the changes once I have them ? should I open a PR here, at @kumaraditya303's fork, both, or anywhere else ?

I'm game to convert a bunch of static PyTypes to heap types in numpy [...]

I'm possibly the wrong person to comment (since I don't think I've ever committed anything to Numpy) but I think the aim isn't to convert Numpy to use the stable ABI (or any future freethreading-compatible Stable ABI). I think it's more to make Numpy's public headers compatible anyone using opaque PyObjects so that it doesn't stop anyone else using a future freethreading-compatible Stable ABI.

convert a bunch of static PyTypes to heap types in numpy

Thankfully we don't need to do this in NumPy itself. It's only needed in projects that want to ship limited API wheels. For NumPy we're happy to continue shipping version-specific wheels as we've always done.

Now that said, it would be neat if NumPy could ship limited API wheels, so if you want to try please feel free. I'm pretty sure there are also open PRs in the backlog for similar efforts. No one else is actively working on it now because it's a big project with an unclear payoff. Unclear because we don't know if the performance hit for avoiding APIs outside the limited API will be unacceptable.

Sure. I'll focus on anything blocking astropy itself first, but I'm interested in exploring and contributing to abi3 compliance for numpy itself too.

for completion: my first message stemmed from a confusion I just figured out; I was looking at the wrong build logs and concluded that this numpy fork needed some more work to support building astropy with the new limited API. Instead, it looks like the problems Nathan described are really within astropy itself, so I'm not expecting I'll need to touch numpy after all.

I rebased my branch and created #31091 to add opaque PyObject ABI support, it is still in draft as PEP 803 is not implemented yet in main branch of CPython so it currently uses _Py_OPAQUE_PYOBJECT.

The implementation is merged now and the docs work in tracked in #31259, closing as completed.

Navigation Menu

Repository navigation

Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) #30704

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	* PyArrayObject field access is deprecated as of NumPy 1.7.
	*/
	typedef PyArrayObject_fields PyArrayObject;
	#else
	typedef struct tagPyArrayObject {
	PyObject_HEAD
	} PyArrayObject;
	#endif

	cdef inline npy_datetime get_datetime64_value(object obj) noexcept nogil:
	"""
	returns the int64 value underlying scalar numpy datetime64 object

	Note that to interpret this as a datetime, the corresponding unit is
	also needed. That can be found using `get_datetime64_unit`.
	"""
	return (<PyDatetimeScalarObject*>obj).obval


	cdef inline npy_timedelta get_timedelta64_value(object obj) noexcept nogil:
	"""
	returns the int64 value underlying scalar numpy timedelta64 object
	"""
	return (<PyTimedeltaScalarObject*>obj).obval

Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) #30704

Description

Activity

rgommers commented on Jan 21, 2026

ngoldbaum commented on Jan 21, 2026

ngoldbaum commented on Jan 21, 2026

seberg commented on Jan 22, 2026

rgommers commented on Jan 22, 2026

ngoldbaum commented on Jan 22, 2026

ngoldbaum commented on Jan 22, 2026

da-woods commented on Jan 22, 2026

rgommers commented on Jan 22, 2026

ngoldbaum commented on Jan 22, 2026

da-woods commented on Jan 22, 2026

ngoldbaum commented on Jan 22, 2026

34 remaining items

ngoldbaum commented on Mar 11, 2026

da-woods commented on Mar 11, 2026

rgommers commented on Mar 11, 2026

da-woods commented on Mar 11, 2026

ngoldbaum commented on Mar 11, 2026

rgommers commented on Mar 11, 2026

kumaraditya303 commented on Mar 12, 2026

neutrinoceros commented on Mar 17, 2026

da-woods commented on Mar 17, 2026

ngoldbaum commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

kumaraditya303 commented on Mar 29, 2026

kumaraditya303 commented on May 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions