Skip to content

Threading support #237

Open
Open

Description

@rth

Just an issue to track the status on the threading support.

Emscripten has Pthreads support via SharedArrayBuffer capability in browsers. The later got disabled due to Spectre etc vulnerabilities, however,

There also has been some activity in emscripten on this subject lately.

Because of,

If code is compiled with -s USE_PTHREADS=1 and the current browser does not support multithreading, then an exception will be thrown at page load time. It is not possible to build one binary that would be able to leverage multithreading when available and fall back to single threaded when not. For such backwards compatibility, two separate builds must be done, one with -s USE_PTHREADS=1 and the other with -s USE_PTHREADS=0.
(taken from emscripten docs)

I imagine we won't be able to do much until both Firefox and Chrome have it enabled by default in a stable release. Still it might be worth trying to make it work in a WIP PR to see what are the difficulties that could arise. Opening this issue for future reference.

This would also very partially address #144

Activity

  1. pbazant commented on Oct 18, 2019

    @pbazant

    Dear pyodide devs, I'd like to start experimenting with the threading support -- our target browser is Chrome anyway (we need web-bluetooth support). Before I delve into the pyodide codebase: How hard should I expect enabling the threading to be?

  2. rth commented on Oct 27, 2019

    @rth
    MemberAuthor

    @pbazant I don't have a good understand of much work it would be, but it would be certainly worth trying to build with threading support following emscripten docs and reporting here if you run into any issues :)

    Though we likely won't be able to merge it in pyodide until support has been enabled in firefox by default, and it doesn't look like it happened yet.

  3. gatesn commented on Jan 21, 2020

    @gatesn

    Looks like this fails in that Emscripten won't allow us to use dynamic loading and PTHREADS in the same build: emscripten-core/emscripten#3494

  4. pbazant commented on Jan 21, 2020

    @pbazant

    I actually managed to build Pyodide with threads semi-working. I had to change several compilation options. This caused emscripten to remove many symbols, so I had to force it to keep them by listing them all. Maybe there is a cleaner way.
    As you say, this is at the cost of not being able to use dynamic loading.
    The problem with the threads was that the non-main threads seem to run only when the main thread is running. Data sharing between the threads seemed to work OK, however! It looked like there was some strange business with the non-main thread being blocked on some mutex. Possibly the GIL behaves in a weird way? I will be investigating this in the future. If I manage to make it fully working, I'll share my results. Thank you all for your answers.

  5. gatesn commented on Jan 22, 2020

    @gatesn

    That’s neat - would you be able to push up a branch for me to poke at?

  6. gcatto commented on Apr 17, 2020

    @gcatto

    @pbazant I see you mentioned web-bluetooth above. I have a use case where I am just starting to consider that in combination with Pyodide. Have you gotten anywhere?

  7. pbazant commented on Apr 18, 2020

    @pbazant

    @pbazant I see you mentioned web-bluetooth above. I have a use case where I am just starting to consider that in combination with Pyodide. Have you gotten anywhere?

    It's a company project. I think web-bluetooth is usable only from the main thread, so if you run Pyodide in a worker thread (probably a good idea), you must use some communication mechanism between the threads.
    Option 1: use postMessage in bith directions(which is tricky because postMessage is tied to the JS event loop and single runPython invocation doesn't yield to the JS loop)
    Option 2: use postMessage only to send messages from Pyodide to the main thread but use the Atomics API + SharedArrayBuffer to send data form the main thread to the worker thread (where Pyodide runs).

  8. pbazant commented on Apr 18, 2020

    @pbazant

    We postponed making the threads work and use async/await + asyncio in Pyodide as a temporary solution to concurrency. To use asyncio, we inherited from asyncio.BaseEventLoop and did some tweaking (one has to provide a custom selector implementation).

  9. stefnotch commented on Jul 8, 2020

    @stefnotch
    Contributor
  10. stefnotch commented on Jul 29, 2020

    @stefnotch
    Contributor

    Quick update: Firefox 79 has been released and it supports WebAssembly threads and SharedArrayBuffer.


    There are some minor restriction regarding SharedArrayBuffer outside of WebAssembly, which are

    • SharedArrayBuffer objects are in principle always available, but unfortunately the constructor on the global object is hidden, unless the two headers mentioned above are set, for compatibility with web content. There is hope that this restriction can be removed in the future. WebAssembly.Memory can still be used to get an instance.
    • Unless the two headers mentioned above are set, the various postMessage() APIs will throw for SharedArrayBuffer objects. If they are set, postMessage() on Window objects and dedicated workers will function and allow for memory sharing.

    The headers in question are the newly introduced

    • Cross-Origin-Opener-Policy with same-origin as value (protects your origin from attackers)
    • Cross-Origin-Embedder-Policy with require-corp as value (protects victims from your origin)
  11. rth commented on Dec 31, 2020

    @rth
    MemberAuthor

    By @oeway in #958 (comment)

    Maybe it's already the time to investigate? It's going to be very useful, imagine with pthread enabled, we can potentially have a web loop closer to native asyncio, allowing more powerful data analysis with multi-threading, the subsequent packages won't need to patch the threading related parts etc.

    Yes, we can certainly start investigating. Maybe under a feature flag for a start.

    Besides that, the wasm thread features is supported already in Firefox 79 (released 5 month ago) and Chrome 70 (2+ years ago), given the current status of pyodide, most users are early adopters that can afford to upgrade to the latest browser version.

    There are certainly early adopters but there are actually more users coming from some kind of teaching related platforms (e.g. https://notebook.basthon.fr or https://buildingai.elementsofai.com/), and there 5 month of Firefox support is very short for end users. Also Safari/WebKit based browsers currently have some issues (#721) but there is hope of fixing it, and as far as I can tell threads won't work there yet (Fyrd/caniuse#5200, https://bugs.webkit.org/show_bug.cgi?id=218944)

    Ideally we could have deployed it as a separate path e.g. <version>/full (default without threads), <version>/with_threads (with threads) on the CDN but that would require more CI work (unless we do it semi-manually).

  12. dalcde commented on Dec 31, 2020

    @dalcde
    Contributor
  13. rth commented on Nov 6, 2021

    @rth
    MemberAuthor

    It looks like all major browsers support threading by now. However this would mean dropping support for Safari <14.1, so maybe we could start using it in a long lived feature branch #1932

  14. 13 remaining items

  15. joemarshall commented on Jan 10, 2023

    @joemarshall
    Contributor

    As I see it big technical roadblock is that threads don't work in site environments where cross site isolation isn't enabled. Which annoyingly includes GitHub pages.

    The pain with threads is that it's a build time choice in emscripten to enable threading or not, and I'm not sure what happens if you build with threading support and then run without isolation?

    There's a risk that it would mean all pyodide packages would need two builds, threads or no threads.

    Everything else is just a matter of time to hack things in.

    Having said that, I just took a look at emscripten, and pthreads appears to mostly be implemented in JavaScript, which I think might just mean that it is safe to compile packages in pthreads mode even if you have the main pyodide module in two flavours (threads and no threads)?

  16. josephrocca commented on Jan 10, 2023

    @josephrocca
    Contributor

    (lol in case anyone is confused, my earlier comment was in response to a now-deleted comment from joe that was mostly the same as the one above)

    I think might just mean that it is safe to compile packages in pthreads mode even if you have the main pyodide module in two flavours (threads and no threads)?

    Nice if true!

  17. LijieZhang1998 commented on Aug 18, 2023

    @LijieZhang1998

    Hi, I'm porting pymongo package to pyodide package. I got an error when I executed my sample code in Node.js. How's the status of threading support in pyodide now? Is there any solution that I can use to work around this error. Thank you.

    error PythonError: Traceback (most recent call last): File "/lib/python311.zip/_pyodide/_base.py", line 558, in eval_code_async await CodeRunner( File "/lib/python311.zip/_pyodide/_base.py", line 381, in run_async coroutine = eval(self.code, globals, locals) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 10, in File "/lib/python3.11/site-packages/pymongo/mongo_client.py", line 837, in __init__ self._get_topology() File "/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1214, in _get_topology self._topology.open() File "/lib/python3.11/site-packages/pymongo/topology.py", line 192, in open self._ensure_opened() File "/lib/python3.11/site-packages/pymongo/topology.py", line 596, in _ensure_opened self._update_servers() File "/lib/python3.11/site-packages/pymongo/topology.py", line 747, in _update_servers server.open() File "/lib/python3.11/site-packages/pymongo/server.py", line 49, in open self._monitor.open() File "/lib/python3.11/site-packages/pymongo/monitor.py", line 79, in open self._executor.open() File "/lib/python3.11/site-packages/pymongo/periodic_executor.py", line 87, in open thread.start() File "/lib/python311.zip/threading.py", line 957, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread
    at new_error (/Users/lijie.zhang/workspace/pyodide-repo/dist/pyodide.asm.js:9:12562)
    at wasm://wasm/0224d1de:wasm-function[295]:0x158a38
    at wasm://wasm/0224d1de:wasm-function[451]:0x15fe7d
    at _PyCFunctionWithKeywords_TrampolineCall (/Users/lijie.zhang/workspace/pyodide-repo/dist/pyodide.asm.js:9:119469)
    at wasm://wasm/0224d1de:wasm-function[1056]:0x1a321d
    at wasm://wasm/0224d1de:wasm-function[3383]:0x28a37c
    at wasm://wasm/0224d1de:wasm-function[2036]:0x1e4108
    at wasm://wasm/0224d1de:wasm-function[1063]:0x1a3705
    at wasm://wasm/0224d1de:wasm-function[1066]:0x1a39c6
    at wasm://wasm/0224d1de:wasm-function[1067]:0x1a3a68
    at wasm://wasm/0224d1de:wasm-function[3196]:0x268abf
    at wasm://wasm/0224d1de:wasm-function[3197]:0x26e8d0
    at wasm://wasm/0224d1de:wasm-function[1069]:0x1a3b90
    at wasm://wasm/0224d1de:wasm-function[1064]:0x1a3820
    at wasm://wasm/0224d1de:wasm-function[439]:0x15f603
    at Module.callPyObjectKwargs (/Users/lijie.zhang/workspace/pyodide-repo/dist/pyodide.asm.js:9:79884)
    at Module.callPyObject (/Users/lijie.zhang/workspace/pyodide-repo/dist/pyodide.asm.js:9:80266)
    at Timeout.wrapper [as _onTimeout] (/Users/lijie.zhang/workspace/pyodide-repo/dist/pyodide.asm.js:9:56949)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7) {
    

    type: 'RuntimeError',
    __error_address: 25425896
    }

  18. ryanking13 commented on Aug 21, 2023

    @ryanking13
    Member

    How's the status of threading support in pyodide now?

    It is not supported in Pyodide yet, and AFAIK there's not much progress. I think threading support would require a lot of work both on CPython side and Emscripten side. So, unfortunately, it's unlikely to be supported quickly.

    Is there any solution that I can use to work around this error.

    You'll need to find a way to run the package with threading disabled.

  19. allsey87 commented on Aug 21, 2023

    @allsey87

    I think Emscripten's dynamic linking will need to become a lot more stable before Pyodide could add support for threading. Dynamic linking does work, however, it gets very flaky if you start using multiple shared modules and mixing in threading/asyncify.

    I think threading support would require a lot of work both on CPython side and Emscripten side.

    @ryanking13 how much is there to do on the CPython side exactly? The only challenge I can see is packaging the compiled modules for both multi-threaded and single-threaded builds... unless perhaps the threading module uses some low-level pthread features that Emscripten does not support?

  20. ryanking13 commented on Aug 23, 2023

    @ryanking13
    Member

    @ryanking13 how much is there to do on the CPython side exactly? The only challenge I can see is packaging the compiled modules for both multi-threaded and single-threaded builds... unless perhaps the threading module uses some low-level pthread features that Emscripten does not support?

    It's actually hard to tell until someone tries it for themselves. I am just saying based on my experience. If you're lucky, CPython's threading implementation will be so compatible with Emscripten that it will work without you doing anything, but based on my experience, this is usually not the case.

  21. allsey87 commented on Aug 23, 2023

    @allsey87

    Perhaps I can give it a go. I already have a multi-threaded build of Pyodide but I haven't tried to enable the threading module.

  22. allsey87 commented on Aug 23, 2023

    @allsey87

    For me, a very simple test with Python threads is working, however, I am using a custom runtime so it would be good if someone could test this with the default JS runtime that Emscripten generates. These are roughly the steps that need to be followed:

    1. Add -pthread to CFLAGS_BASE, CXXFLAGS_BASE and LDFLAGS_BASE to Makefile.env.
    2. Pass --enable-wasm-pthreads to the configure script invocation in cpython/Makefile. I am not actually sure if this does anything other than check that we are building for wasm32-unknown-emscripten. Internally, CPython seems to turn on threading when it detects the flags __EMSCRIPTEN_PTHREADS__ and __EMSCRIPTEN_SHARED_MEMORY__.
    3. Update the sqlite3 package to build and link the multi-threaded version of sqlite3 as follows:
    diff --git a/packages/sqlite3/meta.yaml b/packages/sqlite3/meta.yaml
    index 5976fbc..d5d26fc 100644
    --- a/packages/sqlite3/meta.yaml
    +++ b/packages/sqlite3/meta.yaml
    @@ -24,15 +24,15 @@ build:
           "Modules/_sqlite/util.c"
         )
     
    -    embuilder build sqlite3 --pic
    +    embuilder build sqlite3-mt --pic
     
         for file in "${FILES[@]}"; do
           emcc $STDLIB_MODULE_CFLAGS -c "${file}" -o "${file/.c/.o}"  \
             -sUSE_SQLITE3 -DMODULENAME=sqlite
         done
     
         OBJECT_FILES=$(find Modules/_sqlite/ -name "*.o")
         emcc $OBJECT_FILES -o $DISTDIR/_sqlite3.so $SIDE_MODULE_LDFLAGS \
    -       -sUSE_SQLITE3 -lsqlite3
    +       -sUSE_SQLITE3 -lsqlite3-mt
     
         cd Lib && tar --exclude=test -cf - sqlite3 | tar -C $DISTDIR -xf -

    With this, I was able to run the following Python code and saw interleaved execution of the two functions:

    import threading
    import sys
    import time
    def print_numbers():
        for i in range(1, 6):
            time.sleep(0.1)
            print('Number:', i)
            sys.stdout.flush()
    def print_letters():
        for letter in 'abcde':
            time.sleep(0.1)
            print('Letter:', letter)
            sys.stdout.flush()
    thread1 = threading.Thread(target=print_numbers)
    thread2 = threading.Thread(target=print_letters)
    thread1.start()
    thread2.start()
    thread1.join()
    thread2.join()
    print('Threads are done!')
    sys.stdout.flush()
    time.sleep(1)

    Note that I have quite a different configuration at this point with Pyodide statically linked to several libraries and use Asyncify. I have no idea whether the changes I listed above will work with the latest Pyodide. My configuration is roughly based on the following:

    Emscripten version: 3.1.39
    Python version: 3.11.3
    Pyodide version: f996914 + python/cpython#103322

  23. lewymati commented on Nov 3, 2023

    @lewymati

    @allsey87 I couldn't reproduce. I followed your instructions, but I've got
    Didn't expect to load any more file_packager files! error in the browser during start of the new thread.
    (caused by this line https://github.com/pyodide/pyodide/blob/main/src/js/pyodide.ts#L418 )
    I tried removing this Module.locateFile = ... line, but then browser freezes when running the code.

    Any chance that there is some missing step?

  24. G3zz commented on Jun 5, 2024

    @G3zz

    For me, a very simple test with Python threads is working, however, I am using a custom runtime so it would be good if someone could test this with the default JS runtime that Emscripten generates. These are roughly the steps that need to be followed:

    1. Add -pthread to CFLAGS_BASE, CXXFLAGS_BASE and LDFLAGS_BASE to Makefile.env.
    2. Pass --enable-wasm-pthreads to the configure script invocation in cpython/Makefile. I am not actually sure if this does anything other than check that we are building for wasm32-unknown-emscripten. Internally, CPython seems to turn on threading when it detects the flags __EMSCRIPTEN_PTHREADS__ and __EMSCRIPTEN_SHARED_MEMORY__.
    3. Update the sqlite3 package to build and link the multi-threaded version of sqlite3 as follows:
    diff --git a/packages/sqlite3/meta.yaml b/packages/sqlite3/meta.yaml
    index 5976fbc..d5d26fc 100644
    --- a/packages/sqlite3/meta.yaml
    +++ b/packages/sqlite3/meta.yaml
    @@ -24,15 +24,15 @@ build:
           "Modules/_sqlite/util.c"
         )
     
    -    embuilder build sqlite3 --pic
    +    embuilder build sqlite3-mt --pic
     
         for file in "${FILES[@]}"; do
           emcc $STDLIB_MODULE_CFLAGS -c "${file}" -o "${file/.c/.o}"  \
             -sUSE_SQLITE3 -DMODULENAME=sqlite
         done
     
         OBJECT_FILES=$(find Modules/_sqlite/ -name "*.o")
         emcc $OBJECT_FILES -o $DISTDIR/_sqlite3.so $SIDE_MODULE_LDFLAGS \
    -       -sUSE_SQLITE3 -lsqlite3
    +       -sUSE_SQLITE3 -lsqlite3-mt
     
         cd Lib && tar --exclude=test -cf - sqlite3 | tar -C $DISTDIR -xf -

    With this, I was able to run the following Python code and saw interleaved execution of the two functions:

    import threading
    import sys
    import time
    def print_numbers():
        for i in range(1, 6):
            time.sleep(0.1)
            print('Number:', i)
            sys.stdout.flush()
    def print_letters():
        for letter in 'abcde':
            time.sleep(0.1)
            print('Letter:', letter)
            sys.stdout.flush()
    thread1 = threading.Thread(target=print_numbers)
    thread2 = threading.Thread(target=print_letters)
    thread1.start()
    thread2.start()
    thread1.join()
    thread2.join()
    print('Threads are done!')
    sys.stdout.flush()
    time.sleep(1)

    Note that I have quite a different configuration at this point with Pyodide statically linked to several libraries and use Asyncify. I have no idea whether the changes I listed above will work with the latest Pyodide. My configuration is roughly based on the following:

    Emscripten version: 3.1.39 Python version: 3.11.3 Pyodide version: f996914 + python/cpython#103322

    On the 0.26.0 branch I get the error below when following these instructions:

    error: undefined symbol: _emscripten_run_callback_on_thread (referenced by $JSEvents, referenced by root reference (e.g. compiled C/C++ code))
    warning: To disable errors for undefined symbols use `-sERROR_ON_UNDEFINED_SYMBOLS=0`
    warning: __emscripten_run_callback_on_thread may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
    Error: Aborting compilation due to previous errors
    
  25. jlucaso1 commented on Sep 28, 2024

    @jlucaso1

    This can be a good reference. Porffor compile js to wasm and has promises integration with limited async integration:
    https://github.com/CanadaHonk/porffor/blob/main/compiler/builtins/promise.ts

  26. henryiii commented on Jun 11, 2026

    @henryiii
    Contributor

    🤖 AI text below 🤖

    Some fresh data on this (June 2026): I built an experimental PYODIDE_PTHREADS=1 opt-in build of Pyodide 314.1.0.dev0 on the pinned Emscripten 5.0.3 — main module with MAIN_MODULE=1 and -pthread, all vendored libs and package side modules rebuilt with -pthread, CPython configured with --enable-wasm-pthreads. The state of dylink+pthreads is much better than our docs assume:

    What works (Node 26 and headless Chrome with COOP/COEP):

    • threading.Thread start/run/join, locks, queue, concurrent.futures.ThreadPoolExecutor
    • CPython stdlib test_thread, test_threading, test_threading_local, test_threadedtempfile, test_thread_local_bytecode, test_importlib.test_threaded_import all pass under --runtime node
    • dlopen under threads, the part we always feared: loading a -pthread numpy wheel while a Python thread is running, importing numpy inside a spawned thread (per-thread dylib table sync), and re-dlopening an already-loaded library all work on 5.0.3. (emscripten#26913 is a 5.0.7 regression, fixed post-5.0.7, so an Emscripten upgrade should re-test this.)
    • 48 numpy matmuls across 12 threads (pool of 8 + deferred spawn) in ~0.6 s in Node.

    The one real bug found — CPython upstream: the wasm-gc call trampoline is incompatible with pthreads builds in two ways: (1) its hardcoded wasm binary declares a non-shared env.memory import, so it fails to instantiate against shared memory at boot; (2) more subtly, the trampoline funcptr is created with addFunction on the main thread and stored in shared _PyRuntimeState, but every pthread has its own WebAssembly.Table and Emscripten only mirrors dlopen'd libraries between thread tables — so the first Python call on a spawned thread dies with RuntimeError: table index is out of bounds in _PyEM_TrampolineCall, surfacing on the main thread as a cryptic Error: unwind. Falling back to the JS trampoline when wasmMemory.buffer instanceof SharedArrayBuffer fixes it; a per-thread trampoline instantiation would be the real fix. This affects any CPython Emscripten pthreads build, not just Pyodide.

    Pyodide-side changes needed (all small): guard the pyodide_pre injected JS (pre.js, pyodide_js_init()) against running in pthread workers; link system/lib/html5/callback.c (JSEvents needs _emscripten_run_callback_on_thread and AUTO_NATIVE_LIBRARIES=0 excludes it); use the libGL-mt-getprocaddr embuilder target; disable memory snapshots (they copy HEAP8, now a SAB view); serve COOP/COEP in the test fixtures.

    What this does not solve: the FFI is not thread-safe (hiwire/JsProxy state is per-realm — spawned threads must not touch js/pyodide.ffi); jsverror wiring is main-thread-only; -pthread is an ABI break requiring a full package rebuild (second build matrix); and COOP/COEP makes a threaded build opt-in-only for deployment. JSPI × threads is untested.

    Happy to share the branch — the whole thing is ifdef PYODIDE_PTHREADS so the default build is untouched.

  27. Un1oR commented on Jul 8, 2026

    @Un1oR

    Great news!
    What steps are expected next to complete this task?

  28. Un1oR commented on Jul 29, 2026

    @Un1oR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions