GH-47329: [Packaging][Python][C++] Build and test PyArrow against Pyodide 314 (`2026_0` ABI) by agriyakhetarpal · Pull Request #47330 · apache/arrow

agriyakhetarpal · 2025-08-13T20:52:26Z

Rationale for this change

This PR allows PyArrow to be built and tested against our Pyodide 2026_0 ABI introduced with the Pyodide 314.0.0 release.

Closes #47329

What changes are included in this PR?

This PR updates the Pyodide version in the build scripts to 314.0.7 and the Emscripten version to 5.0.3, along with associated build-time changes to fix compilation issues.

This PR updates the conda-python-emscripten job to Pyodide 314.0.7, Python 3.14, Node 24, and pyodide-build 0.39.2 and later. The Emscripten version for this job is gathered from the Pyodide distribution, which renders it to be 5.0.3. That matches the version the C++ job uses, since #55417.

The build changes are as follows:

Emscripten builds now use native WebAssembly exceptions and WebAssembly-based setjmp/longjmp (-fwasm-exceptions -sSUPPORT_LONGJMP=wasm), as our 2026_0 ABI requires.
libarrow_python.so and the Cython extension modules are linked with an $ORIGIN RPATH, in order for us to find libarrow_python.so next to the modules that need it.
The host Protobuf build is given the C++ standard explicitly, which fixes an Abseil ABI mismatch when linking with AppleClang on my machine.

Are these changes tested?

Yes, these changes are (to be) tested by the test-conda-python-emscripten and test-ubuntu-22.04-cpp-emscripten jobs on Crossbow.

Are there any user-facing changes?

Yes, downstream packagers trying to build a WASM wheel for Pyodide are now required to use newer Pyodide and Emscripten versions, and the latest stable version of pyodide-build.

GitHub Issue: [Packaging][Python] Build and test against Pyodide 314.0 (2026_0 ABI) #47329

cc: @hoodmane @ryanking13

github-actions · 2025-08-13T20:52:51Z

⚠️ GitHub issue #47329 has been automatically assigned in GitHub to PR creator.

agriyakhetarpal · 2025-08-13T21:44:52Z

@github-actions crossbow submit test-conda-python-emscripten

github-actions · 2025-08-13T21:45:25Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/16950058312

agriyakhetarpal · 2025-08-13T21:50:15Z

Hello, could the maintainers with the necessary permissions please trigger an Emscripten build on this PR? Thank you!

I thought I could do this, as I previously submitted a PR that was accepted and merged: #45072, so I'm not sure if I understand the bot's message above. Is it that the word "contributors" here actually means those with a commit bit? 🤔

agriyakhetarpal · 2025-08-13T21:52:15Z

It looks like the actual error is something else, though, which gets masked by the error message: https://github.com/apache/arrow/actions/runs/16950058312/job/48040535515?pr=47330#step:5:16

fatal: not a git repository (or any of the parent directories): .git

kou · 2025-08-14T03:18:31Z

@github-actions crossbow submit test-conda-python-emscripten

github-actions · 2025-08-14T03:20:39Z

Revision: 9d4bf5a

Submitted crossbow builds: ursacomputing/crossbow @ actions-e6f3074ad3

Task	Status
test-conda-python-emscripten

agriyakhetarpal · 2025-08-14T03:31:03Z

Thanks. We'll need a docker.io/apache/arrow-dev:amd64-conda-python-3.13 image for this, as the required Python version for Pyodide 0.28 builds is 3.13.

agriyakhetarpal · 2025-08-14T03:34:10Z

Could you please submit another build? Thanks!

kou · 2025-08-14T03:36:08Z

@github-actions crossbow submit test-conda-python-emscripten

github-actions · 2025-08-14T03:38:18Z

Revision: 16f90d9

Submitted crossbow builds: ursacomputing/crossbow @ actions-c232acf2ff

Task	Status
test-conda-python-emscripten

agriyakhetarpal · 2025-08-14T05:11:28Z

I am trying to find out the correct place to install libatomic1; it's required by the Brotli and Protocol Buffers Emscripten builds. In the meantime; I changed -fexceptions to -fwasm-exceptions in eebb7a8.

agriyakhetarpal · 2025-08-14T05:20:23Z

I am trying to find out the correct place to install libatomic1; it's required by the Brotli and Protocol Buffers Emscripten builds.

I added this in af2e18d, but I'm not sure if this is the right approach. Could you please verify this commit, and also submit another build? Thanks!

kou · 2025-08-14T05:22:23Z

@github-actions crossbow submit test-conda-python-emscripten

kou · 2025-08-14T05:24:09Z

FYI: You can use UBUNTU=22.04 archery docker run ubuntu-cpp-emscripten on local.

See https://arrow.apache.org/docs/developers/continuous_integration/archery.html how to setup Archery.

github-actions · 2025-08-14T05:24:32Z

Revision: af2e18d

Submitted crossbow builds: ursacomputing/crossbow @ actions-5d3d1c591a

Task	Status
test-conda-python-emscripten

agriyakhetarpal · 2025-08-14T06:23:01Z

Unfortunately, Archery was not helpful for me, as I am on an M-series macOS machine, and that means Docker runtimes under linux/arm64 emulation are irritably slow. :(

I'll try to get access to a Linux-amd64 machine for this. In the meantime, could you please submit another build? I'd like to try out the libatomic1 addition. The -fwasm-exceptions flag was apparently not needed, so I've reverted it.

kou · 2025-08-14T06:44:07Z

@github-actions crossbow submit test-conda-python-emscripten

hoodmane · 2025-08-14T06:44:16Z

The -fwasm-exceptions flag was apparently not needed, so I've reverted it.

You sure about this? I think it's definitely wrong to pass -fexceptions....

Copilot

🔵 Needs a closer look

Human assessment is needed because the change may discard caller-provided linker flags and appears to leave the Node test runner incompatible with the migrated ESM loader.

No open issues 🎉

_{Review effort: Lite}

github-actions · 2026-11-12T07:36:54Z

Revision: 3c80ba4

Submitted crossbow builds: ursacomputing/crossbow @ actions-457023a373

Task	Status
test-conda-python-emscripten
test-ubuntu-22.04-cpp-emscripten

tadeja · 2026-11-12T07:36:55Z

@github-actions crossbow submit emscripten

github-actions · 2026-11-12T07:38:56Z

Revision: 3c80ba4

Submitted crossbow builds: ursacomputing/crossbow @ actions-0f15cbd5a3

Task	Status
test-conda-python-emscripten
test-ubuntu-22.04-cpp-emscripten

tadeja · 2026-11-12T09:51:21Z

@github-actions crossbow submit test-conda-python-emscripten

github-actions · 2026-11-12T09:54:43Z

Revision: 3c80ba4

Submitted crossbow builds: ursacomputing/crossbow @ actions-f18f77b7a1

Task	Status
test-conda-python-emscripten

tadeja · 2026-11-12T12:40:45Z

The previous two runs of test-conda-python-emscripten failed just because of a time limit: The job has exceeded the maximum execution time of 1h0m0s
The last run of test-conda-python-emscripten is now passing! https://github.com/ursacomputing/crossbow/actions/runs/43195482865/job/131105226434

tadeja

I've just updated unneeded testing submodule change on this PR (so it no longer moves backward from Nov 5a26d35 to June 9ff285c).
LGTM!

Copilot

🔵 Needs a closer look

The Pyodide 2026_0 upgrade requires unavailable Crossbow Emscripten and browser/Node validation to confirm WASM exceptions, linker RPATH, ABI, and module-worker runtime behavior.

No open issues 🎉

_{Review effort: Lite}

tadeja · 2026-11-12T13:00:23Z

cc @raulcd, @kou for another review

agriyakhetarpal · 2026-11-12T13:04:51Z

I've just updated unneeded testing submodule change on this PR (so it no longer moves backward from Nov 5a26d35 to June 9ff285c). LGTM!

Ah, thanks a lot @tadeja! 🙈

I have two follow-ups here:

I found a bug on the Pyodide side that I am working to prepare a non-arrow reproducer on, which came from investigating test_read_csv_gil_deadlock, which is that a side module's weak symbols crash when running time.sleep(), when switching stacks with stubs generated from them beneath them.
The testing infrastructure here can rely on pyodide venv, which is much simpler. That makes things more developer-friendly; the NodeDriver can be removed entirely, since the interpreter itself uses Node.js. I tested it locally, and we can enable the ORC tests that have been skipped, since we can access the timezone database dir through the venv.

tadeja · 2026-11-12T13:33:44Z

Ah!

I found a bug on the Pyodide side that I am working to prepare a non-arrow reproducer on, which came from investigating test_read_csv_gil_deadlock, which is that a side module's weak symbols crash when running time.sleep(), when switching stacks with stubs generated from them beneath them.

Does it crash only for a time.sleep() call or can other operations trigger it as well ? Once you have an upstream issue, could you link it here? Perhaps also a short comment for pytest.mark.threading on def test_read_csv_gil_deadlock()

The testing infrastructure here can rely on pyodide venv, which is much simpler.

Sounds like the pyodide venv migration can be done separately. Would you like to open a follow-up issue/PR?

agriyakhetarpal · 2026-11-12T14:07:45Z

Ah!

I found a bug on the Pyodide side that I am working to prepare a non-arrow reproducer on, which came from investigating test_read_csv_gil_deadlock, which is that a side module's weak symbols crash when running time.sleep(), when switching stacks with stubs generated from them beneath them.

Does it crash only for a time.sleep() call or can other operations trigger it as well ? Once you have an upstream issue, could you link it here? Perhaps also a short comment for pytest.mark.threading on def test_read_csv_gil_deadlock()

It can be anything that uses pyodide.ffi.run_sync(), like time.sleep() here, or asyncio.run() or synchronous HTTP with JSPI (which we have enabled). I think Node sockets too. I think it's a bit orthogonal to the test itself, so maybe not worth an inline comment, but I can link it here :) I opened pyodide/pyodide#6551.

The testing infrastructure here can rely on pyodide venv, which is much simpler.

Sounds like the pyodide venv migration can be done separately. Would you like to open a follow-up issue/PR?

Oh, yes, I indeed meant a follow-up PR :) I'm on it and will open one once this one is merged!

Copilot

🔵 Needs a closer look

The review spans tightly coupled ABI, compiler, linker, and loader changes, and its correctness cannot be established until the Crossbow Pyodide 314 build and both runtime test jobs complete.

No open issues 🎉

_{Review effort: Lite}

kou

+1

agriyakhetarpal requested review from assignUser, jonkeane, kou and raulcd as code owners August 13, 2025 20:52

github-actions Bot added Component: Documentation awaiting review labels Aug 13, 2025

This was referenced Aug 13, 2025

Update pyodide marimo-team/marimo#5995

Closed

[Packaging][Python] Build and test against Pyodide 314.0 (2026_0 ABI) #47329

Closed

github-actions Bot added the Component: C++ label Aug 14, 2025

agriyakhetarpal changed the title ~~GH-47329: [Packaging][Python] Build and test PyArrow against the pyodide_2025_0 ABI~~ GH-47329: [Packaging][Python][C++] Build and test PyArrow against the pyodide_2025_0 ABI Aug 14, 2025

github-actions Bot removed the Component: C++ label Aug 14, 2025

Copilot started reviewing on behalf of agriyakhetarpal November 12, 2026 05:49

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

Restore testing submodule rev

099c659

Copilot AI lite review requested due to automatic review settings November 12, 2026 12:45

Copilot started reviewing on behalf of agriyakhetarpal November 12, 2026 12:49

View session

tadeja approved these changes Nov 12, 2026

View reviewed changes

Copilot AI reviewed Nov 12, 2026

View reviewed changes

agriyakhetarpal mentioned this pull request Nov 12, 2026

Fatal SuspendError when time.sleep() (or run_sync()) is called beneath an Emscripten dynamic-linking stub pyodide/pyodide#6551

Open

Merge branch 'main' into feat/pyodide-0.28-builds

dafb6e2

Copilot AI lite review requested due to automatic review settings November 12, 2026 14:09

Copilot started reviewing on behalf of agriyakhetarpal November 12, 2026 14:14

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

kou approved these changes Nov 13, 2026

View reviewed changes

raulcd approved these changes Nov 13, 2026

View reviewed changes

raulcd merged commit 48e4abf into apache:main Nov 13, 2026
56 of 59 checks passed

agriyakhetarpal deleted the feat/pyodide-0.28-builds branch November 13, 2026 12:01

This was referenced Nov 13, 2026

Use pyodide venv for testing PyArrow pyemscripten builds #55586

Open

GH-55586: [Packaging][Python] Use pyodide venv for Emscripten/Pyodide testing infrastructure and unskip several tests #55588

Open

Conversation

agriyakhetarpal commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 13, 2025

Uh oh!

agriyakhetarpal commented Aug 13, 2025

Uh oh!

github-actions Bot commented Aug 13, 2025

Uh oh!

agriyakhetarpal commented Aug 13, 2025

Uh oh!

agriyakhetarpal commented Aug 13, 2025

Uh oh!

kou commented Aug 14, 2025

Uh oh!

github-actions Bot commented Aug 14, 2025

Uh oh!

agriyakhetarpal commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agriyakhetarpal commented Aug 14, 2025

Uh oh!

kou commented Aug 14, 2025

Uh oh!

github-actions Bot commented Aug 14, 2025

Uh oh!

agriyakhetarpal commented Aug 14, 2025

Uh oh!

agriyakhetarpal commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Aug 14, 2025

Uh oh!

kou commented Aug 14, 2025

Uh oh!

github-actions Bot commented Aug 14, 2025

Uh oh!

agriyakhetarpal commented Aug 14, 2025

Uh oh!

kou commented Aug 14, 2025

Uh oh!

hoodmane commented Aug 14, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Nov 12, 2026

Uh oh!

tadeja commented Nov 12, 2026

Uh oh!

github-actions Bot commented Nov 12, 2026

Uh oh!

tadeja commented Nov 12, 2026

Uh oh!

github-actions Bot commented Nov 12, 2026

Uh oh!

tadeja commented Nov 12, 2026

Uh oh!

tadeja left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

tadeja commented Nov 12, 2026

Uh oh!

agriyakhetarpal commented Nov 12, 2026

Uh oh!

tadeja commented Nov 12, 2026

Uh oh!

agriyakhetarpal commented Aug 13, 2025 •

edited

Loading

agriyakhetarpal commented Aug 14, 2025 •

edited

Loading

agriyakhetarpal commented Aug 14, 2025 •

edited

Loading

tadeja left a comment •

edited

Loading