Skip to content

Commit 48e4abf

Browse files
apacheGH-47329: [Packaging][Python][C++] Build and test PyArrow against Pyodide 314 (2026_0 ABI) (apache#47330)
### Rationale for this change This PR allows PyArrow to be built and tested against our Pyodide `2026_0` ABI introduced with [the Pyodide 314.0.0 release](https://blog.pyodide.org/posts/314-release/). Closes apache#47329 ### What changes are included in this PR? This PR updates the Pyodide version in the build scripts to 314.0.7 and the Emscripten version to 5.0.3, along with associated build-time changes to fix compilation issues. This PR updates the `conda-python-emscripten` job to Pyodide 314.0.7, Python 3.14, Node 24, and pyodide-build 0.39.2 and later. The Emscripten version for this job is gathered from the Pyodide distribution, which renders it to be 5.0.3. That matches the version the C++ job uses, since apache#55417. The build changes are as follows: - Emscripten builds now use native WebAssembly exceptions and WebAssembly-based `setjmp`/`longjmp` (`-fwasm-exceptions -sSUPPORT_LONGJMP=wasm`), as our `2026_0` ABI requires. - `libarrow_python.so` and the Cython extension modules are linked with an `$ORIGIN` RPATH, in order for us to find `libarrow_python.so` next to the modules that need it. - The host Protobuf build is given the C++ standard explicitly, which fixes an Abseil ABI mismatch when linking with AppleClang on my machine. ### Are these changes tested? Yes, these changes are (to be) tested by the `test-conda-python-emscripten` and `test-ubuntu-22.04-cpp-emscripten` jobs on Crossbow. ### Are there any user-facing changes? Yes, downstream packagers trying to build a WASM wheel for Pyodide are now required to use newer Pyodide and Emscripten versions, and the latest stable version of pyodide-build. * GitHub Issue: apache#47329 cc: @ hoodmane @ ryanking13 Lead-authored-by: Agriya Khetarpal <[email protected]> Co-authored-by: Tadeja Kadunc <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
1 parent dd619ad commit 48e4abf

9 files changed

Lines changed: 63 additions & 31 deletions

File tree

‎ci/docker/conda-python-emscripten.dockerfile‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,32 +18,39 @@
1818
ARG repo
1919
ARG arch
2020
ARG arch_short
21-
ARG python="3.12"
21+
ARG python="3.14"
2222
FROM --platform=linux/${arch} ${repo}:${arch_short}-conda-python-${python}
2323

2424
ARG selenium_version="4.41.0"
25-
ARG pyodide_version="0.27.1"
25+
ARG pyodide_version="314.0.7"
26+
# pyodide-build is versioned independently of Pyodide
27+
ARG pyodide_build_version="0.39.2"
2628
ARG chrome_version="latest"
27-
ARG required_python_min="(3,12)"
28-
# fail if python version < 3.12
29+
ARG required_python_min="(3,14)"
30+
# fail if python version < 3.14
2931
RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
3032

31-
# install selenium and recent pyodide-build and recent python
33+
RUN apt-get update -y -q && \
34+
apt-get install -y -q --no-install-recommends \
35+
libatomic1 && \
36+
apt-get clean && \
37+
rm -rf /var/lib/apt/lists/*
3238

3339
# needs to be a login shell so ~/.profile is read
3440
SHELL ["/bin/bash", "--login", "-c", "-o", "pipefail"]
3541

42+
# install selenium and recent pyodide-build and recent python
3643
RUN python -m pip install --no-cache-dir selenium==${selenium_version} && \
37-
python -m pip install --no-cache-dir --upgrade pyodide-build>=${pyodide_version}
44+
python -m pip install --no-cache-dir --upgrade "pyodide-build>=${pyodide_build_version}"
3845

3946
# install pyodide dist directory to /pyodide
4047
RUN pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2" && \
4148
wget -q "${pyodide_dist_url}" -O- | tar -xj -C /
4249

43-
# install node 20 (needed for async call support)
50+
# install node 24 (needed for async call support and JSPI)
4451
# and pthread-stubs for build, and unzip needed for chrome build to work
4552
# xz is needed by emsdk to extract node tarballs
46-
RUN conda install nodejs=20 unzip pthread-stubs make xz -c conda-forge
53+
RUN conda install nodejs=24 unzip pthread-stubs make xz -c conda-forge
4754

4855
# install correct version of emscripten for this pyodide
4956
COPY ci/scripts/install_emscripten.sh /arrow/ci/scripts/

‎compose.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -982,10 +982,11 @@ services:
982982
arch_short: ${ARCH_SHORT}
983983
clang_tools: ${CLANG_TOOLS}
984984
llvm: ${LLVM}
985-
pyodide_version: "0.27.1"
985+
pyodide_version: "314.0.7"
986+
pyodide_build_version: "0.39.2"
986987
chrome_version: "latest"
987988
selenium_version: "4.41.0"
988-
required_python_min: "(3,12)"
989+
required_python_min: "(3,14)"
989990
python: ${PYTHON}
990991
shm_size: *shm-size
991992
volumes: *ubuntu-volumes

‎cpp/cmake_modules/SetupCxxFlags.cmake‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -714,26 +714,35 @@ if(MSVC)
714714
endif()
715715

716716
if(CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
717-
# flags are:
717+
# For the Pyodide 2026_0 ABI, the flags are:
718718
# 1) We force *everything* to build as position independent
719-
# 2) And with support for C++ exceptions
720-
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIC -fexceptions")
719+
# 2) We support C++ WASM exceptions, using the native WebAssembly
720+
# exception handling proposal and WebAssembly-based setjmp/longjmp
721+
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIC -fwasm-exceptions -sSUPPORT_LONGJMP=wasm")
721722
# deprecated-literal-operator error is thrown in datetime (vendored lib in arrow)
722723
set(CMAKE_CXX_FLAGS
723-
"${CMAKE_CXX_FLAGS} -fPIC -fexceptions -Wno-error=deprecated-literal-operator")
724+
"${CMAKE_CXX_FLAGS} -fPIC -fwasm-exceptions -sSUPPORT_LONGJMP=wasm -Wno-error=deprecated-literal-operator"
725+
)
724726

725727
# flags for creating shared libraries (only used in pyarrow, because
726728
# Emscripten builds libarrow as static)
727729
# flags are:
728730
# 1) Tell it to use JavaScript / WebAssembly 64 bit number support.
729-
# 2) Tell it to build with support for C++ exceptions
731+
# 2) Tell it to build with support for C++ exceptions (see above)
730732
# 3) Skip linker flags error which happens with -soname parameter
731-
set(ARROW_EMSCRIPTEN_LINKER_FLAGS "-sWASM_BIGINT=1 -fexceptions -Wno-error=linkflags")
732-
set(CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS
733-
"-sSIDE_MODULE=1 ${ARROW_EMSCRIPTEN_LINKER_FLAGS}")
734-
set(CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS
735-
"-sSIDE_MODULE=1 ${ARROW_EMSCRIPTEN_LINKER_FLAGS}")
736-
set(CMAKE_SHARED_LINKER_FLAGS "-sSIDE_MODULE=1 ${ARROW_EMSCRIPTEN_LINKER_FLAGS}")
733+
# 4) Record $ORIGIN as an RPATH, so that Pyodide can locate
734+
# libarrow_python.so next to the extension modules needing it
735+
# Note: the CMAKE_SHARED_LIBRARY_CREATE_<LANG>_FLAGS and
736+
# CMAKE_SHARED_MODULE_CREATE_<LANG>_FLAGS variables are not used by the
737+
# Ninja generator, so the flags must go through the linker-flags
738+
# variables, which apply to SHARED (libarrow_python) and MODULE (Cython
739+
# extension) targets respectively.
740+
set(ARROW_EMSCRIPTEN_LINKER_FLAGS
741+
"-sWASM_BIGINT=1 -fwasm-exceptions -sSUPPORT_LONGJMP=wasm -Wno-error=linkflags")
742+
set(ARROW_EMSCRIPTEN_DYLIB_CREATE_FLAGS
743+
"-sSIDE_MODULE=1 ${ARROW_EMSCRIPTEN_LINKER_FLAGS} '-Wl,-rpath,$ORIGIN'")
744+
set(CMAKE_SHARED_LINKER_FLAGS "${ARROW_EMSCRIPTEN_DYLIB_CREATE_FLAGS}")
745+
set(CMAKE_MODULE_LINKER_FLAGS "${ARROW_EMSCRIPTEN_DYLIB_CREATE_FLAGS}")
737746
if(ARROW_TESTING)
738747
# Deeply nested tests need a larger stack (1 MiB).
739748
set(ARROW_EMSCRIPTEN_TEST_STACK_SIZE 1048576)

‎cpp/cmake_modules/ThirdpartyToolchain.cmake‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2171,6 +2171,10 @@ function(build_protobuf)
21712171
set(PROTOBUF_HOST_CMAKE_ARGS
21722172
"-DCMAKE_CXX_FLAGS="
21732173
"-DCMAKE_C_FLAGS="
2174+
# Explicitly set the C++ standard for the host build. Without this,
2175+
# Abseil may be compiled with the host compiler's default standard,
2176+
# which can be older than C++17.
2177+
"-DCMAKE_CXX_STANDARD=${CMAKE_CXX_STANDARD}"
21742178
"-DCMAKE_INSTALL_PREFIX=${PROTOBUF_HOST_PREFIX}"
21752179
-Dprotobuf_BUILD_TESTS=OFF
21762180
-Dprotobuf_DEBUG_POSTFIX=

‎dev/tasks/tasks.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -493,7 +493,7 @@ tasks:
493493
params:
494494
env:
495495
UBUNTU: 22.04
496-
PYTHON: 3.12
496+
PYTHON: 3.14
497497
image: conda-python-emscripten
498498

499499
test-conda-python-3.11-hypothesis:

‎docs/source/developers/cpp/emscripten.rst‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,9 @@ activate it using the commands below (see https://emscripten.org/docs/getting_st
3434
git clone https://github.com/emscripten-core/emsdk.git
3535
cd emsdk
3636
# replace <version> with the desired EMSDK version.
37-
# e.g. for Pyodide 0.26, you need EMSDK version 3.1.58
37+
# e.g. for Pyodide 314.0.x, you need EMSDK version 5.0.3
3838
# the versions can be found in the Makefile.envs file in the Pyodide repo:
39-
# https://github.com/pyodide/pyodide/blob/10b484cfe427e076c929a55dc35cfff01ea8d3bc/Makefile.envs
39+
# https://github.com/pyodide/pyodide/blob/314.0.7/Makefile.envs
4040
./emsdk install <version>
4141
./emsdk activate <version>
4242
source ./emsdk_env.sh
@@ -49,8 +49,9 @@ versions of emsdk tools.
4949
.. code:: shell
5050
5151
# install Pyodide build tools.
52-
# e.g., for version 0.26 of Pyodide, pyodide-build 0.26 and later work
53-
pip install "pyodide-build>=0.26"
52+
# pyodide-build is versioned independently of Pyodide.
53+
# For Pyodide 314.0.x, pyodide-build 0.39.2 and later work well
54+
pip install "pyodide-build>=0.39.2"
5455
5556
Then build with the ``ninja-release-emscripten`` CMake preset,
5657
like below:

‎python/CMakeLists.txt‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,9 @@ if("$ENV{CMAKE_EXPORT_COMPILE_COMMANDS}" STREQUAL "1")
111111
set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
112112
endif()
113113

114-
if(UNIX)
114+
# On Emscripten, the equivalent $ORIGIN runtime path is added
115+
# through the linker flags in SetupCxxFlags.cmake instead
116+
if(UNIX AND NOT CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
115117
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
116118
# In the event that we are bundling the shared libraries (e.g. in a
117119
# manylinux1 wheel), we need to set the RPATH of the extensions to the

‎python/pyarrow/tests/test_csv.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2202,6 +2202,7 @@ def test_write_csv_decimal(tmpdir, type_factory):
22022202
assert out.column('col').cast(type) == table.column('col')
22032203

22042204

2205+
@pytest.mark.threading
22052206
def test_read_csv_gil_deadlock():
22062207
# GH-38676
22072208
# This test depends on several preconditions:

‎python/scripts/run_emscripten_tests.py‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ def do_GET(self) -> bytes | None:
7676
evt.data.print.forEach((x)=>{window.python_logs.push(x)});
7777
}
7878
}
79-
window.pyworker = new Worker("worker.js");
79+
window.pyworker = new Worker("worker.js", {type: "module"});
8080
window.pyworker.onmessage = capturelogs;
8181
</script>
8282
</head>
@@ -90,8 +90,8 @@ def do_GET(self) -> bytes | None:
9090
self.copyfile(BytesIO(body), self.wfile)
9191
elif self.path.endswith("/worker.js"):
9292
body = b"""
93-
importScripts("./pyodide.js");
94-
onmessage = async function (e) {
93+
import { loadPyodide } from "./pyodide.mjs";
94+
self.onmessage = async function (e) {
9595
const data = e.data;
9696
if (!self.pyodide) {
9797
self.pyodide = await loadPyodide();
@@ -148,8 +148,14 @@ class NodeDriver:
148148
import subprocess
149149

150150
def __init__(self, hostname, port):
151+
# handle different syntax for util-linux script and macOS BSD-style script
152+
if sys.platform == "darwin":
153+
script_command = [shutil.which("script"), "-q", "/dev/null",
154+
shutil.which("node")]
155+
else:
156+
script_command = [shutil.which("script"), "-c", shutil.which("node")]
151157
self.process = subprocess.Popen(
152-
[shutil.which("script"), "-c", shutil.which("node")],
158+
script_command,
153159
stdin=subprocess.PIPE,
154160
shell=False,
155161
bufsize=0,
@@ -320,6 +326,7 @@ def _load_pyarrow_in_runner(driver, wheel_name):
320326
if "pyarrow" not in sys.modules:
321327
await micropip.install("hypothesis")
322328
import pyodide_js as pjs
329+
await pjs.loadPackage("packaging")
323330
await pjs.loadPackage("tzdata")
324331
await pjs.loadPackage("numpy")
325332
await pjs.loadPackage("pandas")

0 commit comments

Comments
 (0)