Repository navigation
Description
Just an issue to track the status on the threading support.
Emscripten has Pthreads support via SharedArrayBuffer capability in browsers. The later got disabled due to Spectre etc vulnerabilities, however,
- a few days ago there was an announcement that threading should work again with WASM in Chrome 70 (https://developers.google.com/web/updates/2018/10/wasm-threads)
SharedArrayBufferis available in Firefox since 57 via a config option, but that is disabled by default. The switch to enabled by default can be tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=1477743
There also has been some activity in emscripten on this subject lately.
Because of,
If code is compiled with -s USE_PTHREADS=1 and the current browser does not support multithreading, then an exception will be thrown at page load time. It is not possible to build one binary that would be able to leverage multithreading when available and fall back to single threaded when not. For such backwards compatibility, two separate builds must be done, one with -s USE_PTHREADS=1 and the other with -s USE_PTHREADS=0.
(taken from emscripten docs)
I imagine we won't be able to do much until both Firefox and Chrome have it enabled by default in a stable release. Still it might be worth trying to make it work in a WIP PR to see what are the difficulties that could arise. Opening this issue for future reference.
This would also very partially address #144
Activity
pbazant commentedon Oct 18, 2019
Dear pyodide devs, I'd like to start experimenting with the threading support -- our target browser is Chrome anyway (we need web-bluetooth support). Before I delve into the pyodide codebase: How hard should I expect enabling the threading to be?
rth commentedon Oct 27, 2019
@pbazant I don't have a good understand of much work it would be, but it would be certainly worth trying to build with threading support following emscripten docs and reporting here if you run into any issues :)
Though we likely won't be able to merge it in pyodide until support has been enabled in firefox by default, and it doesn't look like it happened yet.
gatesn commentedon Jan 21, 2020
Looks like this fails in that Emscripten won't allow us to use dynamic loading and PTHREADS in the same build: emscripten-core/emscripten#3494
pbazant commentedon Jan 21, 2020
I actually managed to build Pyodide with threads semi-working. I had to change several compilation options. This caused emscripten to remove many symbols, so I had to force it to keep them by listing them all. Maybe there is a cleaner way.
As you say, this is at the cost of not being able to use dynamic loading.
The problem with the threads was that the non-main threads seem to run only when the main thread is running. Data sharing between the threads seemed to work OK, however! It looked like there was some strange business with the non-main thread being blocked on some mutex. Possibly the GIL behaves in a weird way? I will be investigating this in the future. If I manage to make it fully working, I'll share my results. Thank you all for your answers.
gatesn commentedon Jan 22, 2020
That’s neat - would you be able to push up a branch for me to poke at?
gcatto commentedon Apr 17, 2020
@pbazant I see you mentioned web-bluetooth above. I have a use case where I am just starting to consider that in combination with Pyodide. Have you gotten anywhere?
pbazant commentedon Apr 18, 2020
It's a company project. I think web-bluetooth is usable only from the main thread, so if you run Pyodide in a worker thread (probably a good idea), you must use some communication mechanism between the threads.
Option 1: use postMessage in bith directions(which is tricky because postMessage is tied to the JS event loop and single runPython invocation doesn't yield to the JS loop)
Option 2: use postMessage only to send messages from Pyodide to the main thread but use the Atomics API + SharedArrayBuffer to send data form the main thread to the worker thread (where Pyodide runs).
pbazant commentedon Apr 18, 2020
We postponed making the threads work and use async/await + asyncio in Pyodide as a temporary solution to concurrency. To use asyncio, we inherited from asyncio.BaseEventLoop and did some tweaking (one has to provide a custom selector implementation).
stefnotch commentedon Jul 8, 2020
Good news is that Firefox 79 will apparently support WebAssembly threads.
stefnotch commentedon Jul 29, 2020
Quick update: Firefox 79 has been released and it supports WebAssembly threads and SharedArrayBuffer.
There are some minor restriction regarding
SharedArrayBufferoutside of WebAssembly, which areThe headers in question are the newly introduced
rth commentedon Dec 31, 2020
By @oeway in #958 (comment)
Yes, we can certainly start investigating. Maybe under a feature flag for a start.
There are certainly early adopters but there are actually more users coming from some kind of teaching related platforms (e.g. https://notebook.basthon.fr or https://buildingai.elementsofai.com/), and there 5 month of Firefox support is very short for end users. Also Safari/WebKit based browsers currently have some issues (#721) but there is hope of fixing it, and as far as I can tell threads won't work there yet (Fyrd/caniuse#5200, https://bugs.webkit.org/show_bug.cgi?id=218944)
Ideally we could have deployed it as a separate path e.g.
<version>/full(default without threads),<version>/with_threads(with threads) on the CDN but that would require more CI work (unless we do it semi-manually).dalcde commentedon Dec 31, 2020
rth commentedon Nov 6, 2021
It looks like all major browsers support threading by now. However this would mean dropping support for Safari <14.1, so maybe we could start using it in a long lived feature branch #1932
13 remaining items
joemarshall commentedon Jan 10, 2023
As I see it big technical roadblock is that threads don't work in site environments where cross site isolation isn't enabled. Which annoyingly includes GitHub pages.
The pain with threads is that it's a build time choice in emscripten to enable threading or not, and I'm not sure what happens if you build with threading support and then run without isolation?
There's a risk that it would mean all pyodide packages would need two builds, threads or no threads.
Everything else is just a matter of time to hack things in.
Having said that, I just took a look at emscripten, and pthreads appears to mostly be implemented in JavaScript, which I think might just mean that it is safe to compile packages in pthreads mode even if you have the main pyodide module in two flavours (threads and no threads)?
josephrocca commentedon Jan 10, 2023
(lol in case anyone is confused, my earlier comment was in response to a now-deleted comment from joe that was mostly the same as the one above)
Nice if true!
LijieZhang1998 commentedon Aug 18, 2023
Hi, I'm porting
pymongopackage to pyodide package. I got an error when I executed my sample code in Node.js. How's the status of threading support in pyodide now? Is there any solution that I can use to work around this error. Thank you.error
PythonError: Traceback (most recent call last): File "/lib/python311.zip/_pyodide/_base.py", line 558, in eval_code_async await CodeRunner( File "/lib/python311.zip/_pyodide/_base.py", line 381, in run_async coroutine = eval(self.code, globals, locals) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 10, in File "/lib/python3.11/site-packages/pymongo/mongo_client.py", line 837, in __init__ self._get_topology() File "/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1214, in _get_topology self._topology.open() File "/lib/python3.11/site-packages/pymongo/topology.py", line 192, in open self._ensure_opened() File "/lib/python3.11/site-packages/pymongo/topology.py", line 596, in _ensure_opened self._update_servers() File "/lib/python3.11/site-packages/pymongo/topology.py", line 747, in _update_servers server.open() File "/lib/python3.11/site-packages/pymongo/server.py", line 49, in open self._monitor.open() File "/lib/python3.11/site-packages/pymongo/monitor.py", line 79, in open self._executor.open() File "/lib/python3.11/site-packages/pymongo/periodic_executor.py", line 87, in open thread.start() File "/lib/python311.zip/threading.py", line 957, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new threadtype: 'RuntimeError',
__error_address: 25425896
}
ryanking13 commentedon Aug 21, 2023
It is not supported in Pyodide yet, and AFAIK there's not much progress. I think threading support would require a lot of work both on CPython side and Emscripten side. So, unfortunately, it's unlikely to be supported quickly.
You'll need to find a way to run the package with threading disabled.
allsey87 commentedon Aug 21, 2023
I think Emscripten's dynamic linking will need to become a lot more stable before Pyodide could add support for threading. Dynamic linking does work, however, it gets very flaky if you start using multiple shared modules and mixing in threading/asyncify.
@ryanking13 how much is there to do on the CPython side exactly? The only challenge I can see is packaging the compiled modules for both multi-threaded and single-threaded builds... unless perhaps the threading module uses some low-level pthread features that Emscripten does not support?
ryanking13 commentedon Aug 23, 2023
It's actually hard to tell until someone tries it for themselves. I am just saying based on my experience. If you're lucky, CPython's threading implementation will be so compatible with Emscripten that it will work without you doing anything, but based on my experience, this is usually not the case.
allsey87 commentedon Aug 23, 2023
Perhaps I can give it a go. I already have a multi-threaded build of Pyodide but I haven't tried to enable the
threadingmodule.allsey87 commentedon Aug 23, 2023
For me, a very simple test with Python threads is working, however, I am using a custom runtime so it would be good if someone could test this with the default JS runtime that Emscripten generates. These are roughly the steps that need to be followed:
-pthreadtoCFLAGS_BASE,CXXFLAGS_BASEandLDFLAGS_BASEto Makefile.env.--enable-wasm-pthreadsto the configure script invocation in cpython/Makefile. I am not actually sure if this does anything other than check that we are building forwasm32-unknown-emscripten. Internally, CPython seems to turn on threading when it detects the flags__EMSCRIPTEN_PTHREADS__and__EMSCRIPTEN_SHARED_MEMORY__.With this, I was able to run the following Python code and saw interleaved execution of the two functions:
Note that I have quite a different configuration at this point with Pyodide statically linked to several libraries and use Asyncify. I have no idea whether the changes I listed above will work with the latest Pyodide. My configuration is roughly based on the following:
Emscripten version: 3.1.39
Python version: 3.11.3
Pyodide version: f996914 + python/cpython#103322
lewymati commentedon Nov 3, 2023
@allsey87 I couldn't reproduce. I followed your instructions, but I've got
Didn't expect to load any more file_packager files!error in the browser during start of the new thread.(caused by this line https://github.com/pyodide/pyodide/blob/main/src/js/pyodide.ts#L418 )
I tried removing this
Module.locateFile = ...line, but then browser freezes when running the code.Any chance that there is some missing step?
Skip tests that use async and threading code
G3zz commentedon Jun 5, 2024
On the 0.26.0 branch I get the error below when following these instructions:
jlucaso1 commentedon Sep 28, 2024
This can be a good reference. Porffor compile js to wasm and has promises integration with limited async integration:
https://github.com/CanadaHonk/porffor/blob/main/compiler/builtins/promise.ts
henryiii commentedon Jun 11, 2026
🤖 AI text below 🤖
Some fresh data on this (June 2026): I built an experimental
PYODIDE_PTHREADS=1opt-in build of Pyodide 314.1.0.dev0 on the pinned Emscripten 5.0.3 — main module withMAIN_MODULE=1and-pthread, all vendored libs and package side modules rebuilt with-pthread, CPython configured with--enable-wasm-pthreads. The state of dylink+pthreads is much better than our docs assume:What works (Node 26 and headless Chrome with COOP/COEP):
threading.Threadstart/run/join, locks,queue,concurrent.futures.ThreadPoolExecutortest_thread,test_threading,test_threading_local,test_threadedtempfile,test_thread_local_bytecode,test_importlib.test_threaded_importall pass under--runtime node-pthreadnumpy wheel while a Python thread is running, importing numpy inside a spawned thread (per-thread dylib table sync), and re-dlopening an already-loaded library all work on 5.0.3. (emscripten#26913 is a 5.0.7 regression, fixed post-5.0.7, so an Emscripten upgrade should re-test this.)The one real bug found — CPython upstream: the wasm-gc call trampoline is incompatible with pthreads builds in two ways: (1) its hardcoded wasm binary declares a non-shared
env.memoryimport, so it fails to instantiate against shared memory at boot; (2) more subtly, the trampoline funcptr is created withaddFunctionon the main thread and stored in shared_PyRuntimeState, but every pthread has its ownWebAssembly.Tableand Emscripten only mirrors dlopen'd libraries between thread tables — so the first Python call on a spawned thread dies withRuntimeError: table index is out of boundsin_PyEM_TrampolineCall, surfacing on the main thread as a crypticError: unwind. Falling back to the JS trampoline whenwasmMemory.buffer instanceof SharedArrayBufferfixes it; a per-thread trampoline instantiation would be the real fix. This affects any CPython Emscripten pthreads build, not just Pyodide.Pyodide-side changes needed (all small): guard the
pyodide_preinjected JS (pre.js,pyodide_js_init()) against running in pthread workers; linksystem/lib/html5/callback.c(JSEvents needs_emscripten_run_callback_on_threadandAUTO_NATIVE_LIBRARIES=0excludes it); use thelibGL-mt-getprocaddrembuilder target; disable memory snapshots (they copyHEAP8, now a SAB view); serve COOP/COEP in the test fixtures.What this does not solve: the FFI is not thread-safe (hiwire/JsProxy state is per-realm — spawned threads must not touch
js/pyodide.ffi); jsverror wiring is main-thread-only;-pthreadis an ABI break requiring a full package rebuild (second build matrix); and COOP/COEP makes a threaded build opt-in-only for deployment. JSPI × threads is untested.Happy to share the branch — the whole thing is
ifdef PYODIDE_PTHREADSso the default build is untouched.Un1oR commentedon Jul 8, 2026
Great news!
What steps are expected next to complete this task?
Un1oR commentedon Jul 29, 2026
@henryiii ?