ENH: WASI Build #25859
Description
Proposed new feature or change:
Seeing that WASI just reached Tier 2 status, I was wondering if there were plans for NumPy to build for the WASI target as well?
Seeing that WASI just reached Tier 2 status, I was wondering if there were plans for NumPy to build for the WASI target as well?
Activity
rgommers commentedon Feb 20, 2024
Thanks for asking @XavierGeerinck. Some more details about what that could/should look like would be helpful. I think our priority is better Pyodide/WASM support (mostly fine as is, need to re-enable a CI job and ensure both in-tree and out-of-tree builds work without issues). Is there anything WASI-specific expected from Python packages?
XavierGeerinck commentedon Feb 20, 2024
Thank you for the quick reply! I think the main use case I would be looking at is to be able to use PyPI with Spin which tries to pull the Numpy package but fails as it's not enabled for WASI apparently. They themselves now refer to https://github.com/dicej/wasi-wheels/tree/main/numpy to build it from scratch
rgommers commentedon Feb 21, 2024
Thanks. I had a look at the Spin docs, which look really polished. It's only supporting pure Python code/packages at this point though. From https://www.fermyon.com/blog/spin-python-sdk#future-work (about a year old):
As of this writing, the Spin Python SDK is still experimental and under development. One big missing piece is support for popular Python libraries which rely on native code, such as Pandas and NumPy. Supporting these libraries is challenging because they rely on dynamically linking native code into the CPython interpreter, which isn’t supported in core Wasm or WASI. There are ways to address this, as shown by projects like Pyodide, so we’re confident we’ll be able to do the same for Spin, but that work has not yet been done.
https://discuss.python.org/t/make-wasm-a-1st-class-platform-in-the-python-ecosystem/21798 is the most relevant discussion about wheels on PyPI for WASM and/or WASI. This is not yet done, I don't think I've seen progress since May'23. That discussion says that WASI doesn't allow dynamic linking, making it harder to create wheels for WASI than WASM. There are wheels at https://github.com/dicej/wasi-wheels, but no idea what is going on there since it has no docs.
If there's anything concrete in NumPy to do, I think we'd be happy to consider it. But as far as I can tell, there is nothing for us to do here at the moment. So I suggest we close this issue for the time being.
XavierGeerinck commentedon Feb 21, 2024
Adding Joel (@dicej) from Fermyon (Spin) who actually provided the support for shared library in WASI SDK 21 (https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-21 - add shared library support by @dicej)
As far as I understood Python's stance, they are currently in Tier 2 (while the post linked before was from Tier 3 still) so they should have a fairly mature process in place so far. What is currently still unclear for me is how far WASI its building process is as Joel made customizations to CPython to support it (dicej/cpython@91d0d48)
rgommers commentedon Feb 21, 2024
One quick note on testing: the test suite passes for Pyodide, but that required a lot of tests to be skipped based on:
I assume WASI is going to have pretty much the same restrictions (threading, lack of floating-point error handling, no compile tests), so if we have to adjust
IS_WASMto include WASI too, that is easy to do.dicej commentedon Feb 21, 2024
Hi all. Sorry about the somewhat scattered state of things at the moment. WASI 0.2.0 was just released a few weeks ago, and we're working on a brand new Spin SDK based on
componentize-pyand the temporary CPython fork @XavierGeerinck linked to above, but we're still in the process of updating docs, etc. The new SDK supports arbitrary native extensions (although compiling Fortran to Wasm is still experimental, as the Pyodide folks can certainly confirm).The
wasi-wheelsrepo is a (hopefully temporary) repository of build scripts for cross-compiling various Python libraries towasm32-wasiusing wasi-sdk. Here's an example of how to use them. @brettcannon and I have been discussing the next steps for defining a standard platform tag for WASI, updating thepackaginglibrary to recognize them, working with the PyPI folks to enable them, etc. Once that's all done, we'll be able to start upstreaming support to various libraries like NumPy. I'm not sure if it makes sense to try to start doing that now while things are still in flux, but I'm happy to answer any questions in the meantime.XavierGeerinck commentedon Feb 21, 2024
Thank you for chiming in @dicej! Then I think this issue @rgommers can either be closed or kept open until WASI is ready for broad roll-out if I understood this reply correctly?
Either way, @dicej would it make sense in the meantime to have Python 3.10, 3.11 and 3.12 supported on the
wasi-wheelsrepo? (we have a hard requirement on 3.11 and can't go above it sadly enough). What is the current state of having the patch you applied supported on CPython?dicej commentedon Feb 21, 2024
Sounds reasonable to me. I'd be happy to review a PR for that :)
@brettcannon is in the midst of updating CPython to use WASI-SDK 21, and hopefully he knows autotools better than I do and can enable dynamic linking the "right way", at which point that part of my patch will no longer be needed. The other part of my patch enables full TCP and UDP socket support based on wasi-sockets using a temporary fork of
wasi-libc. That's still in the process of being upstreamed towasi-libc, so we'll need to finish that and make a new WASI-SDK release before it makes sense to get it into CPython. Fortunately, you don't need to wait for any of that to actually use those features sincecomponentize-pyis using all the relevant forks.BTW, I should clarify that the dynamic linking support in WASI-SDK 21 is really just a stub:
dlopenanddlsymalways return errors by default.componentize-pyreplaces those stubs with a useful implementation using component model shared-everything dynamic linking. That implementation doesn't actually load libraries from the filesystem at runtime; instead, it uses build-time-generated lookup tables to resolve symbols in Wasm modules bundled into the component. You can think of this as "pseudo-dynamic-linking". Perhaps someday WASI will also support more traditional dynamic linking that loads libraries from the filesystem, but right now everything either needs to be bundled into a single component, or else imported at the component level.feat: position as narrow pure-Python sandbox, bump 0.3.0