Skip to content

Commit 7bec4fa

Browse files
apacheGH-45071: [Packaging][Docs] Fix NumPy v2 include directory for Emscripten, and update Pyodide-related documentation (apache#45072)
### Rationale for this change This change would allow building PyArrow correctly with NumPy 1.X and NumPy 2.X, since we are trying to do the latter for pyodide/pyodide#4925. This PR closes apachegh-45071. ### What changes are included in this PR? This PR - issues a correction for the NumPy header files when building under Emscripten - updates Pyodide-related build instructions ### Are these changes tested? Yes, working here: https://github.com/agriyakhetarpal/pyodide-numpy-2.0-rebuilds/actions/runs/12399351376/job/34619554658#step:8:4547 via agriyakhetarpal/pyodide@b651698 that applies a subset of the changes as a patch (the CI job is failing for unrelated reasons, please ignore). ### Are there any user-facing changes? Yes, users trying to build a WASM wheel via Pyodide are now requested to use newer Pyodide and Emscripten versions, and the latest stable version of `pyodide-build` available. * GitHub Issue: apache#45071 Authored-by: Agriya Khetarpal <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
1 parent 9e08c57 commit 7bec4fa

3 files changed

Lines changed: 21 additions & 12 deletions

File tree

‎ci/docker/conda-python-emscripten.dockerfile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,14 +27,14 @@ ARG required_python_min="(3,12)"
2727
# fail if python version < 3.12
2828
RUN echo "check PYTHON>=${required_python_min}" && python -c "import sys;sys.exit(0 if sys.version_info>=${required_python_min} else 1)"
2929

30-
# install selenium and pyodide-build and recent python
30+
# install selenium and recent pyodide-build and recent python
3131

3232
# needs to be a login shell so ~/.profile is read
3333
SHELL ["/bin/bash", "--login", "-c", "-o", "pipefail"]
3434

3535
RUN python -m pip install --no-cache-dir selenium==${selenium_version} && \
36-
python -m pip install --no-cache-dir --upgrade pyodide-build==${pyodide_version}
37-
36+
python -m pip install --no-cache-dir --upgrade pyodide-build>=${pyodide_version}
37+
3838
# install pyodide dist directory to /pyodide
3939
RUN pyodide_dist_url="https://github.com/pyodide/pyodide/releases/download/${pyodide_version}/pyodide-${pyodide_version}.tar.bz2" && \
4040
wget -q "${pyodide_dist_url}" -O- | tar -xj -C /

‎docs/source/developers/cpp/emscripten.rst‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,9 @@ activate it using the commands below (see https://emscripten.org/docs/getting_st
3333
git clone https://github.com/emscripten-core/emsdk.git
3434
cd emsdk
3535
# replace <version> with the desired EMSDK version.
36-
# e.g. for Pyodide 0.24, you need EMSDK version 3.1.45
36+
# e.g. for Pyodide 0.26, you need EMSDK version 3.1.58
37+
# the versions can be found in the Makefile.envs file in the Pyodide repo:
38+
# https://github.com/pyodide/pyodide/blob/10b484cfe427e076c929a55dc35cfff01ea8d3bc/Makefile.envs
3739
./emsdk install <version>
3840
./emsdk activate <version>
3941
source ./emsdk_env.sh
@@ -46,8 +48,8 @@ versions of emsdk tools.
4648
.. code:: shell
4749
4850
# install Pyodide build tools.
49-
# e.g. for version 0.24 of Pyodide:
50-
pip install pyodide-build==0.24
51+
# e.g., for version 0.26 of Pyodide, pyodide-build 0.26 and later work
52+
pip install "pyodide-build>=0.26"
5153
5254
Then build with the ``ninja-release-emscripten`` CMake preset,
5355
like below:
@@ -69,8 +71,7 @@ go to ``arrow/python`` and run
6971
pyodide build
7072
7173
It should make a wheel targeting the currently enabled version of
72-
Pyodide (i.e. the version corresponding to the currently installed
73-
``pyodide-build``) in the ``dist`` subdirectory.
74+
Pyodide in the ``dist`` subdirectory.
7475

7576

7677
Manual Build
@@ -85,9 +86,8 @@ you will need to override. In particular you will need:
8586

8687
#. ``CMAKE_TOOLCHAIN_FILE`` set by using ``emcmake cmake`` instead of just ``cmake``.
8788

88-
#. You will quite likely need to set ``ARROW_ENABLE_THREADING`` to ``OFF``
89-
for builds targeting single threaded Emscripten environments such as
90-
Pyodide.
89+
#. You will need to set ``ARROW_ENABLE_THREADING`` to ``OFF`` for builds
90+
targeting single-threaded Emscripten environments such as Pyodide.
9191

9292
#. ``ARROW_FLIGHT`` and anything else that uses network probably won't
9393
work.

‎python/CMakeLists.txt‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,8 +166,17 @@ if($ENV{PYODIDE})
166166
# modules (at least under Pyodide it does).
167167
set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE})
168168
set(Python3_LIBRARY $ENV{CPYTHONLIB})
169-
set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include)
170169
set(Python3_EXECUTABLE)
170+
execute_process(COMMAND ${Python3_EXECUTABLE} -c
171+
"import numpy; print(numpy.__version__)"
172+
OUTPUT_VARIABLE PYODIDE_NUMPY_VERSION
173+
OUTPUT_STRIP_TRAILING_WHITESPACE)
174+
string(REGEX MATCH "^([0-9]+)" PYODIDE_NUMPY_MAJOR_VERSION ${PYODIDE_NUMPY_VERSION})
175+
if(PYODIDE_NUMPY_MAJOR_VERSION GREATER_EQUAL 2)
176+
set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/_core/include)
177+
else()
178+
set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include)
179+
endif()
171180
set(ENV{_PYTHON_SYSCONFIGDATA_NAME} $ENV{SYSCONFIG_NAME})
172181
# we set the c and cxx compiler manually to bypass pywasmcross
173182
# which is pyodide's way of messing with C++ build parameters.

0 commit comments

Comments
 (0)