Document the binary interface of cimported modules (.pxd ABI rules, capsule signature checks)

Describe your issue

Background

While investigating a compatibility breakage between SciPy nightlies and scikit-learn (scikit-learn#33616, a recurrence of scipy#18377), I found it necessary to read the Cython compiler source in detail to understand exactly what constitutes a binary-incompatible change to a .pxd file. That information does not appear to be documented anywhere. This issue proposes adding it.

What actually happens, and where in the source

The Cython documentation describes .pxd files as working "like C header files". This is not actually the case when it comes to binary compatibility of cimport'ing across packages.

There are two separate checking mechanisms, operating at different times and with different rules.

1. Compile-time check: `.pxd` vs `.pyx` declaration mismatch

When a .pxd and a .pyx file share the same name, Cython merges them during compilation. It calls same_c_signature_as_resolved_type (and relatives compatible_signature_with_resolved_type, narrower_c_signature_than_resolved_type) in Cython/Compiler/PyrexTypes.py to decide whether the declarations are compatible. Crucially, type comparison at this stage calls resolve() on both sides, which strips typedefs down to the underlying primitive - so ctypedef int blas_int and int are considered the same type at compile time, and no error is raised.

The merge and error reporting happens in Symtab.ModuleScope.declare_cfunction, with the compile-time error "Function signature does not match previous declaration" emitted there.

2. Runtime check: `cimport` across pre-compiled modules

This is the check that causes the TypeError: C function X has wrong signature error
at import time. The mechanism lives in Cython/Utility/ImportExport.c:

The exporting module (e.g. scipy.linalg.cython_blas) calls __Pyx_ExportFunction during its PyInit_*, which stores each exported function as a PyCapsule in a module-level dict called __pyx_capi__. The capsule's name is the function's C type signature, rendered as a string by CFuncType.signature_string() → empty_declaration_code() → declaration_code("") in PyrexTypes.py.
The importing module calls __Pyx_ImportFunction_<cyversion> → __Pyx_ImportFromPxd_<cyversion>, which retrieves the capsule from __pyx_capi__ and calls PyCapsule_IsValid(cobj, expected_sig). This is a plain C string comparison of the expected signature (baked in at the importer's compile time) against the stored capsule name. There is no typedef resolution, no semantic equivalence - only byte-for-byte string equality.

The practical consequence: a ctypedef int blas_int introduced in a .pxd changes the capsule string for every function that uses blas_int as a parameter type, from e.g. "double (int *, double *, int *)" to "double (__pyx_t_5scipy_6linalg_11cython_blas_blas_int *, double *, __pyx_t_5scipy_6linalg_11cython_blas_blas_int *)", breaking all previously compiled cimporters - even though the underlying C type is identical. The mangled name __pyx_t_5scipy_6linalg_11cython_blas_blas_int encodes the full dotted module path of the .pxd that introduced the typedef, using a length-prefixed scheme (5scipy_, 6linalg_, 11cython_blas_).

The same asymmetry applies to const: CConstOrVolatileType.same_as_resolved_type accepts const T where T was declared (so the compile-time check silently passes), but declaration_code emits "const " into the capsule string, so the runtime check fails.

Documentation gap

The .pxd files tutorial page says only that .pxd files "work like C header files", which is misleading. The sharing declarations page describes the mechanism superficially but gives no guidance on what changes are binary-safe and what are not. Neither page mentions the __pyx_capi__ dict, PyCapsule, or the possibility of runtime TypeError from signature mismatches.

Since .pxd files are the primary ABI surface for any compiled Cython library that is cimported by third parties, this gap regularly causes real-world breakage (the scipy/scikit-learn breakage has now occurred twice).

Proposed documentation additions

The following two tables summarize what I believe to be the complete rules, derived from reading source code (with help of Claude, as declared further down) and validation of some of these rules with SciPy builds.

Complete binary interface of a cimported module

Interface element	Mechanism	Changes that break it	Failure
`cdef` function signature	`PyCapsule` in `__pyx_capi__`, name = C type declaration string	Any change to argument or return types (including `const`, calling convention, typedef introduction/removal); switching `cdef` ↔ `cpdef`; adding optional args; moving the `.pxd` to a different package path	`TypeError` at import time
`cdef` variable type	`PyCapsule` in `__pyx_capi__`, name = C type declaration string	Same as above	`TypeError` at import time
`cdef class` struct size	`tp_basicsize` comparison in `__Pyx_ImportType_<cyversion>`	Adding, removing, or changing `cdef` attributes	`ValueError` at import time
`cdef class` vtable layout	Raw `void` capsule with `name=NULL`, no validation*	Adding, removing, or reordering `cdef` methods	Silent wrong-function-pointer call, undefined behaviour
`cdef inline` functions	Textually inlined into each TU; no runtime check	Signature changes caught at compile time only	Compile-time error
`cdef extern from` declarations	Not exported at all	N/A	Linker error or silent type mismatch
Fused-type specializations	One capsule entry per concrete type, key = `__pyx_fuse_N` + name	Reordering types within a `ctypedef fused` block changes which specialization is at index N	`TypeError` at import time

Capsule signature string for common types

The capsule string is the output of declaration_code("") (non-pyrex, non-display path) on each argument and return type:

Cython type	Capsule string
`int`	`int`
`unsigned int`	`unsigned int`
`long long`	`PY_LONG_LONG` (not `long long` — a Cython-internal macro name)
`unsigned long long`	`unsigned PY_LONG_LONG`
`double`	`double`
`bint`	`int`
`char *`	`char *`
`const double *`	`const double *`
`ctypedef int foo` (private, top-level in `.pxd`)	`__pyx_t_<length-prefixed-module-path>_foo`
`ctypedef int foo` (inside `cdef extern from`)	`foo`
`int[10]` as argument	`int *` (array parameters decay to pointers, as in C)
`object`, `list`, `dict`, …	`PyObject *` (all Python object types are identical in the capsule)
`MyExtClass` (cdef class used as argument)	`struct __pyx_obj_<module>_MyExtClass *`
`double[:]`, `double[::1]`, `double[:, :]` (any memoryview)	`__Pyx_memviewslice` (layout is erased)
`const double[:]`	`__Pyx_memviewslice` (const on memoryview dtype does not appear)
`cpdef` hidden dispatch argument	`int` (the invisible `__pyx_skip_dispatch` arg)
No arguments	`void`

Note that long long appearing as PY_LONG_LONG in capsule strings means Cython's built-in long long type and a ctypedef long long foo (extern) type named foo produce different capsule strings even on a platform where they are the same C type - another source of unexpected incompatibility.

How to inspect a module's current capsule strings

import ctypes, scipy.linalg.cython_blas as mod

_get_name = ctypes.pythonapi.PyCapsule_GetName
_get_name.restype = ctypes.c_char_p
_get_name.argtypes = [ctypes.py_object]

for name, capsule in sorted(mod.__pyx_capi__.items()):
    print(f"{name}: {_get_name(capsule).decode()}")

In scipy/scipy#24938 I added a script to generate regression tests for ABI stability. Unless these string comparisons can be improved, it looks like any package that releases a Cython API with wheels (or anywhere where ABI stability is assumed) should be adding a test along those lines.

AI usage disclosure

This research was carried out with substantial assistance from Claude Sonnet, in particular to dig into the Cython internals of how the compile-time and runtime checks work in detail. I spent most of today digging into this problem to fix the scipy/sklearn incompatibility issue, so I'm fairly sure that the content in this issue is largely correct. It may of course be incomplete, inaccurate or suboptimal - this is nontrivial to understand.

We've definitely had requests to do something about const creating a different output before. I haven't done it mostly because I haven't come up with a good solution. Most changes we do now end up being hard breaks.

In the fairly recent past we've resisted adding documentation for __pyx_capi__ (on the basis that it's an internal implementation detail). But clearly "what matches" is somewhat public.

If we wanted to improve it, the remaining place we have available to store ABI metadata is the context pointer of the capsule. If used, that should be used in a way that can be extended because there isn't a further place to add more data. One possibility would be to be able to manually specify @cython.compatible_signature on a function to manually add a list of signatures that also match (i.e. to manually work around adding const or changing to a typedef). That's very manual but it would likely be fairly simple to implement and let users work around the most common problems.

In the fairly recent past we've resisted adding documentation for __pyx_capi__ (on the basis that it's an internal implementation detail). But clearly "what matches" is somewhat public.

How about as a stopgap, something like a simple .. warning:: on https://cython.readthedocs.io/en/latest/src/tutorial/pxd_files.html that links to this issue? That doesn't commit you to documenting __pyx_capi__ but at least warns users that they have to inspect these signatures (and clean them up) before committing to any public API/ABI.

One possibility would be to be able to manually specify @cython.compatible_signature on a function to manually add a list of signatures that also match

Something like that would really help I think. It's not just signatures though, it may also be the keys of the __pyx_capi__ dict. Example:

>>> import ctypes, scipy.special.cython_special as mod
... 
... _get_name = ctypes.pythonapi.PyCapsule_GetName
... _get_name.restype = ctypes.c_char_p
... _get_name.argtypes = [ctypes.py_object]
... 
... for name, capsule in sorted(mod.__pyx_capi__.items()):
...     print(f"{name}: {_get_name(capsule).decode()}")

<snip>
__pyx_fuse_2smirnovi: double (Py_ssize_t, double, int __pyx_skip_dispatch)
__pyx_fuse_2yn: double (Py_ssize_t, double, int __pyx_skip_dispatch)
agm: double (double, double, int __pyx_skip_dispatch)
bei: double (double, int __pyx_skip_dispatch)

So half the keys end up in there with their plain function name, and half with a mangled __pyx_fuse_xxx name. Even a basic refactoring, addition (say trying to add a second version of a function for an additional type), or removal of a second type will be breaking.

I think to improve this, you'd want a remapping from old name/signature to new name/signature. We already have a concrete case of only the name changing that looks like it needs fixing before the next SciPy release: scipy/scipy#24937

@mdboom

I agree with regard to Cython ABI both the Cython language guarantee and its docs have been super unclear. Previously, I've asked if we could at least document __pyx_capi__ (#3077, #6484, #6487) but it did not go anywhere.

@mdboom from our team has been investigating for quite some time the scope of "Cython ABI" (it is a muddy concept unclear to our C++-expert colleagues) and how to detect potential breakage (NVIDIA/cuda-python#914 (comment), NVIDIA/cuda-python#1030, NVIDIA/cuda-python#1067, NVIDIA/cuda-python#1663), but we did not land on any concrete solution either. It would be really nice if we could figure something out through this discussion 🙂

Previously, I've asked if we could at least document __pyx_capi__ (#3077, #6484, #6487) but it did not go anywhere.

That looks like the right intent, but the wrong ask (it's about the semi-private implementation detail itself rather than the observable effects of it). Focusing on __pyx_capi__ seems counterproductive - the real ask is to document (a) the semantic guarantees of ABI stability when touching .pxd files, and (b) how to test that. The former is about features. And I think the latter can also be done without even touching __pyx_capi__: in the end you can achieve exactly the same thing by just reading the .pxd file itself. The requirement seems to be that you can insert signatures on new lines into a .pxd, and you can reorder existing lines, but you cannot change existing signatures in any way whatsoever. Nor move the .pxd around in the installed tree.

The above tables I posted are about the more granular effects of that. E.g., if you have a double routine and want to add a float version of it, the natural way would be to change it to a fused type float_t that maps to double/float. Which is breaking in the .pxd file. Simply adding a new float signature (even if the .pyx does use fused types) is fine.

But clearly "what matches" is somewhat public.

Surely that is not only somewhat public, it's 100% public. Which is why Numba, SciPy, cuda-python, etc. all keep shooting themselves in the foot with it. Let's document that?

@rgommers

Let's document that?

I agree. I proposed to document mechanism __pyx_capi__ which was declined as private implementation. I think we should start with non-controversial stuff. What about documenting high level rules for cimporting? I think @rgommers did great job with the tables. How about documenting rules without mentioning how are internally implemented?

Compability rules
Here we explain that some changes does not break ABI but other does.

Interface element	Changes that break it	Failure
`cdef` function signature	Any change to argument or return types (including `const`, calling convention, typedef introduction/removal); switching `cdef` ↔ `cpdef`; adding optional args; moving the `.pxd` to a different package path	`TypeError` at import time

type
Here we document how types are exposed by ABI.

Cython type	Type exposed by ABI
`int`	`int`
`unsigned int`	`unsigned int`

Finally, we can document some examples and guidance for the users.

This way we are not explaining internal mechanism (pycapusles, __pyx_capi__, etc.) only high level rules for ABI...

What do you think?

@rgommers

That looks like the right intent, but the wrong ask (it's about the semi-private implementation detail itself rather than the observable effects of it). Focusing on __pyx_capi__ seems counterproductive - the real ask is to document (a) the semantic guarantees of ABI stability when touching .pxd files, and (b) how to test that. The former is about features. And I think the latter can also be done without even touching __pyx_capi__: in the end you can achieve exactly the same thing by just reading the .pxd file itself. The requirement seems to be that you can insert signatures on new lines into a .pxd, and you can reorder existing lines, but you cannot change existing signatures in any way whatsoever. Nor move the .pxd around in the installed tree.

@rgommers I would kindly disagree with you 🙂 I think it is equally important to document __pyx_capi__ in addition to all the nice, undocumented corners discovered and detailed in your OP. This dict cannot be kept undocumented indefinitely, when many projects have to reach into it (ex: Numba, Jax, cuda-core, PETSc, Pythran) in order to 1. do run-time type introspection, and/or 2. reuse function pointers from Cython modules. It is a concrete part of the Cython ABI spec IMHO.

It is a concrete part of the Cython ABI spec IMHO.

I see your point in practice, and it's true today given that there's no other supported way to do this - but conceptually it's not quite right perhaps. What is part of the public ABI is:

The signatures in public .pxd headers (today the literal string signatures, but that could be improved)
Cython's name mangling scheme for those signatures
An address table with function pointers

What __pyx_capi__ provides (unless I'm missing something) is just convenient access to those three things. .pxd headers clearly are already accessible, so it looks like the real ask is to add some supported way to call Cython-compiled functions in public modules.

The way I read previous discussions about documenting __pyx_capi__ is that there wasn't a super clear explanation of those asks. And it got rejected without much discussion, probably because (a) it's an object that starts with an underscore, (b) it isn't the optimal way of designing a supported way of doing this, and (c) it exists on private Cython-generated extension modules as well.

If CPython devs get to be grumpy about Cython digging into CPython internals, I guess Cython devs get to do the same when their users dig into what they consider private. Maybe not optimal, but I think this can also be solved without having to actually use __pyx_capi__ itself, if that's what Cython devs prefer. E.g., through documentation of the name mangling scheme. Or through a new dunder dict (or two) that only exists on public modules.

I think then things could be improved for JIT compilers even. E.g., the Numba docs say "To find out the mangled name of your function you can check the extension module's __pyx_capi__ attribute." - Numba could instead do the unmangling itself, making that feature easier to use as well as more robust (no longer sensitive to use of fused types that would be an ABI break for AOT-compiled packages).

I'll also note that not all projects you linked depend on all of (1), (2) and (3) above. E.g., literal string signatures are critical for cimport ABI compatibility, but the address table doesn't matter. For JAX it's the opposite: it grabs the function pointers, but manually maintains its own function signatures without relying on either the signatures or the name mangling.

@mdboom

What is part of the public ABI is:

The signatures in public .pxd headers (today the literal string signatures, but that could be improved)

Cython's name mangling scheme for those signatures

An address table with function pointers

What __pyx_capi__ provides (unless I'm missing something) is just convenient access to those three things.

Yes.

.pxd headers clearly are already accessible, so it looks like the real ask is to add some supported way to call Cython-compiled functions in public modules.

It's more to it. If a cdef class is generated, we also want to inspect its struct layout and vtable. Today this is not retrievable in the Cython ABI checker that @mdboom is building: NVIDIA/cuda-python#1952 (we use this tool to e.g. ensure that our massive codegen refactoring produces the same ABI: NVIDIA/cuda-python#1900 (comment)).

Numba could instead do the unmangling itself, making that feature easier to use as well as more robust

I am not sure how it is possible without reaching into __pyx_capi__ and/or using Cython itself to reconstruct pieces of the info, as our checker does today? Do we (re-)implement the demangler?

To be clear, I agree with all aspects discussed above regarding documenting the ABI rules (your nice tables of what breaks, how types render as strings, etc., as a contract on .pxd semantics). It is a huge improvement in any case!

My point is in order to fully complete the ABI story, if we do not document __pyx_capi__ by name (treating it as the current impl of that contract) we really need some sort of battery-included introspection API that __pyx_capi__-using projects like ours can migrate to.

Documentation is one thing. The ability of programmatically enforcing/introspecting the contract is another thing. It is much better if everyone can avoid re-implementing a part of Cython or reaching into Cython internal so as to perform static ABI checks and/or fulfill other needs (ex: dynamically accessing function pointers) that requires reaching into __pyx_capi__ today.

@leofang

Documentation is one thing. The ability of programmatically enforcing/introspecting the contract is another thing.

@leofang yes completely agreed. Everything needed either should to be implemented in Cython, or it needs to be documented and committed to as stable so that there can be a single external implementation I'd think. It's hard to draw an analogy to C++, because there the name mangling is determined by the platform and the compiler only implements it, so the demangling utilities are scatted. But that's clearly suboptimal, and Cython is fully in charge of it, so leaving it as an exercise for every user would be the worst of all options.

Everything needed either should to be implemented in Cython…

This was my thought, too. I think we need to do two things:

Short-term: add a way to "freeze" API function declarations given in public .pxd files so that Cython can produce compile time errors if they change in a way that runtime cimports from existing packages would not accept. (Something like a checked-in per-branch textual __pyx_capi__ dump that Cython can validate modules against while translating them.)
Longer-term: augment the cimport mechanism with something like the buffer interface signatures to detect compatible signatures/layouts and distinguish them from real layout incompatibilities at (import) runtime. (To follow Postel's law.)

The second is difficult because it requires platform specific adaptations at C compile time (or even during module setup). But I guess NumPy implies that it can be done.

@scoder

@scoder thanks for your input. Regarding NumPy: it doesn't do anything like your (2), it pretty much does (1). It does that by laying out what goes where in the ABI in a code generator (numpy_api.py), and then takes a hash of that (cversions.txt) that gets checked at build time. It then relies on standard C ABI rules. E.g., if on the user side something uses long long instead of int64_t, that works just fine if those integer types have the same bit width (and if not, the C compiler will tell you). There is no signature checking.

So NumPy essentially has a slightly nicer version of your (1): it's a append-only table of API entries, so that they can be checked with a single hash and appended to but not otherwise altered, and therefore one can use a simple indexed lookup to retrieve the function pointers, rather than using a dict with random order and (possibly mangled) function names as keys.

If we're using laws, then I'd say NumPy aims for adhering to Hyrum's Law rather than Postel's.

For (2): it may be worth considering that there are two very different use cases:

Private sharing between .pyx files within a package only,
Publicly exposing a Cython API to other package.

A large majority of .pxd usage is for private sharing I'd say, and for that use case, everything in this issue is irrelevant. As are the runtime signature checks, because nothing can cause a mismatch.

Exposing a public Cython API is limited to a few handfuls of advanced users like Cython itself, numpy, scipy, pyarrow, cuda-python, mpi4py, and lxml I think. Packages like Numba and JAX are primarily listed here because they consume the Cython APIs of those packages (although Numba lets you import any Cython function so that may be relied on more widely, but likely not in ways that are stable across package versions).

If you designed this from scratch, it would probably make sense to have packages that want to expose a public Cython API use new syntax or a separate step to actually do that export in a more controlled manner, a la numpy's append-only table (or in C with visibility macros or dllexport with MSVC, or ...). That would avoid the need for doing runtime signature checks. Those seem like a lost cause, because if you tried to improve them you're basically trying to second-guess what the C language and C compilers do - that may work for simple cases like adding const, but for arbitrary integer types, fused types, classes, etc., I don't see that working well.

Improve names in fused __pyx_api

Slightly tangential to the "documenting it" bit, but: I think it's fairly easy to improve the situation for fused functions and to give them a dictionary key that depends on the specialized types and not the (slightly arbitrary) order of the types in the fused type. That doesn't break any API because it's just extra keys in the dictionary so we can keep the old ones around as a fallback.

Most other improvements are harder because it involves changing something existing.

Improve names in fused `__pyx_api__` (GH-7656)

Navigation Menu

Repository navigation

Document the binary interface of cimported modules (`.pxd` ABI rules, capsule signature checks) #7603

Describe your issue

Background

What actually happens, and where in the source

1. Compile-time check: `.pxd` vs `.pyx` declaration mismatch

2. Runtime check: `cimport` across pre-compiled modules

Documentation gap

Proposed documentation additions

Complete binary interface of a cimported module

Capsule signature string for common types

How to inspect a module's current capsule strings

AI usage disclosure

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Document the binary interface of cimported modules (.pxd ABI rules, capsule signature checks) #7603

Description

Describe your issue

Background

What actually happens, and where in the source

1. Compile-time check: .pxd vs .pyx declaration mismatch

2. Runtime check: cimport across pre-compiled modules

Documentation gap

Proposed documentation additions

Complete binary interface of a cimported module

Capsule signature string for common types

How to inspect a module's current capsule strings

AI usage disclosure

Activity

da-woods commented on Mar 30, 2026

rgommers commented on Mar 30, 2026

leofang commented on Apr 8, 2026

rgommers commented on Apr 8, 2026

rgommers commented on Apr 8, 2026

matusvalo commented on Apr 8, 2026

leofang commented on Apr 8, 2026

rgommers commented on Apr 9, 2026

leofang commented on Apr 21, 2026

rgommers commented on Apr 21, 2026

scoder commented on Apr 21, 2026

rgommers commented on Apr 22, 2026

da-woods commented on May 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Document the binary interface of cimported modules (`.pxd` ABI rules, capsule signature checks) #7603

1. Compile-time check: `.pxd` vs `.pyx` declaration mismatch

2. Runtime check: `cimport` across pre-compiled modules