Repository navigation
Questions about __pyx_capi__ use/stability. #3077
Description
A common use case when extending Numba is to use an existing Cython extension module to provide access points to the compiled functions contained within. For Numba to do this the __pyx_capi__ attribute must be queried. Examples and discussion here: numba/numba#3086 and API guide here http://numba.pydata.org/numba-doc/latest/extending/high-level.html#importing-cython-functions.
Recently work has started on numba-scipy to add SciPy support to Numba, this PR numba/numba-scipy#12 notes that __pyx_capi__ is a Cython internal.
The crux of all this is I think a few questions:
- Is
__pyx_capi__sufficiently ingrained in Cython that it can be relied upon being present indefinitely. - Is
__pyx_capi__truly internal? Or are there cases in practice where users are parsing/using it (pythran does already?!)? - Is the form of
__pyx_capi__stable/consistent.
Thanks in advance for your help!
Reacted by Leo Fang
Activity
leofang commentedon Nov 5, 2024
+1 for all the questions that @stuartarchibald asked.
We hit this issue recently because we wanted to have a trampoline module (with an old name/path) to "re-export" all Cython symbols from another (new) module to keep a smooth transition to the latter, and it seems a plain
cimportin the trampoline is not enough, and we need to hack at the__pyx_capi__level.da-woods commentedon Nov 6, 2024
I'm reluctant to say it's completely fine to use it. However, we are aware that people are using it. And we've also found in the past that it's really difficult to change it because it's really supposed to match between Cython versions.
So it's likely to be stable, but I wouldn't absolutely promise it.
matusvalo commentedon Nov 6, 2024
I think that it is already used so we should not change it without proper deprecation & replacement in any case. Also it seems to be important for the clients. So I think it is better to document it. I created an issue for that - #6484.
Fixing #6484 should also close this issue because all questions should be clear from the docs.
matusvalo commentedon Nov 6, 2024
I am thinking aloud. As an alternative to providing
__pyx_capi__as part of API does it make sense to provide function (eg.cython.cimport()) which dynamicaly cimports functions similar to numbas?https://github.com/numba/numba/blob/0f363d1b2dd19f2aa1a8cec5f0a99c3dd95512f8/numba/_helperlib.c#L561
da-woods commentedon Nov 7, 2024
I think it'd necessarily have to be visible through a Python interface (otherwise people wouldn't be able to access it easily).
That limits how we chose to return C function pointers back. It'd either be a
PyCapsule(in which case it's very similar to__pyx_capi__), or a Python int, or maybe a ctypes function pointer?stuartarchibald commentedon Nov 7, 2024
Thanks for taking a look at this issue, I'm glad that
__pyx_capi__is going to be considered stable and documented through #6484. Numba has been relying on it for quite a while!To the discussion... from Numba's point of view, the JIT compiler part just needs the address of a symbol and a prototype for calling it. The code linked above in
_helperlib.cis simply there to hide the internal mechanics from the user and expose the address from the retrievedPyCapsule. Making an interface Python accessible would be helpful, Numba's PIXIE project is doing something similar, it exposes everything about the compiled C-extension through a module level dictionary in the extension itself. As to returning function pointers, I don't know how easy it is to handlePyCapsules from Python itself, the situation may have changed since I opened the issue/did the original work, but at the time I seem to recall there was very little available to handle them.scoder commentedon Nov 8, 2024
da-woods commentedon Nov 8, 2024
We've failed to change them before because changing them would mean that you couldn't cimport a module compiled with a different Cython version.
There was an issue (that I can't find right now) where Pandas/scipy/similar wanted to change their function arguments from
inttoconst int(which is considered identical from a C point-of-view) but they couldn't do it because it changed the__pyx_capi__signature, which broke cimports for anyone using the public pxd file.Which to me suggests we're pretty limited in how we can change this ourselves.
So while I don't think this mechanism is likely to change soon, I'm not sure we should formally document it. I think we should just tolerate and be a bit considerate of people using it.