Skip to content

Commit ba5ca20

Browse files
authored
GH-48172: [Python] Add cp315 to build (#48191)
### Rationale for this change Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow. Note: - ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded. - Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~ - [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01. ### What changes are included in this PR? - Adding cp315/cp315t to the build matrix - Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available. ### 2026-08-13 edit - Bump libcst to 1.9.0 - Update manylinux image to 2026.08.08-1 (for 3.15rc1) - Add -rc to PYTHON_IMAGE_TAG - Disable pandas tests if Pandas isn't installed: ### Are these changes tested? - Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal) ### Are there any user-facing changes? No functional changes, other than allowing the user to use Python 3.15. * GitHub Issue: #48172 ### Progress Notes (2026-08-15) Each platform uses different sources for Python builds, each with different release schedules and paths: - [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1. - musllinux: Uses Alpine system python built from docker-library/python#1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35). - musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`. - macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary - Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile` For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1. Lead-authored-by: Paul Timmins <[email protected]> Co-authored-by: [email protected] <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
1 parent 71c98fc commit ba5ca20

19 files changed

Lines changed: 74 additions & 30 deletions

‎.env‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,5 +98,6 @@ VCPKG="9b965a116838c6cdcd36bca60d1b81b030c8ab8d" # 2026.05.27 (not release, u
9898
# ci/docker/python-*-windows-*.dockerfile or the vcpkg config.
9999
# This is a workaround for our CI problem that "archery docker build" doesn't
100100
# use pulled built images in dev/tasks/python-wheels/github.windows.yml.
101-
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-07-24
102-
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-07-24
101+
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-08-14
102+
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-08-14
103+

‎ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,8 @@ RUN apk update && \
3535
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
3636
RUN set -e; \
3737
case "${python_version}" in \
38-
3.14) python_patch_version="3.14.5";; \
38+
3.14) python_patch_version="3.14.7";; \
39+
3.15) python_patch_version="3.15.0rc1";; \
3940
esac && \
4041
curl -L -o python.tar.zst \
4142
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \

‎ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ RUN apk update && \
3737
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
3838
RUN set -e; \
3939
case "${python_version}" in \
40-
3.14) python_patch_version="3.14.5";; \
40+
3.14) python_patch_version="3.14.7";; \
41+
3.15) python_patch_version="3.15.0rc1";; \
4142
esac && \
4243
curl -L -o python.tar.zst \
4344
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \

‎ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,17 @@ FROM ${base}
2626

2727
ARG python=3.14
2828

29+
# PYTHON_RELEASE is the python.org ftp directory, which for a pre-release is the
30+
# final release it leads to, e.g. 3.15.0/python-3.15.0rc1-amd64.exe
2931
# hadolint ignore=SC1072
30-
RUN (if "%python%"=="3.14" setx PYTHON_VERSION "3.14.5")
32+
RUN (if "%python%"=="3.14" setx PYTHON_VERSION "3.14.7" && setx PYTHON_RELEASE "3.14.7") & \
33+
(if "%python%"=="3.15" setx PYTHON_VERSION "3.15.0rc1" && setx PYTHON_RELEASE "3.15.0")
3134

3235
SHELL ["powershell", "-NoProfile", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
3336
RUN $version = $env:PYTHON_VERSION; \
37+
$release = $env:PYTHON_RELEASE; \
3438
$filename = 'python-' + $version + '-amd64.exe'; \
35-
$url = 'https://www.python.org/ftp/python/' + $version + '/' + $filename; \
39+
$url = 'https://www.python.org/ftp/python/' + $release + '/' + $filename; \
3640
Invoke-WebRequest -Uri $url -OutFile $filename; \
3741
Start-Process -FilePath $filename -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait
3842

‎ci/docker/python-wheel-windows-test-vs2022.dockerfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,9 @@ ARG python=3.11
3131
RUN (if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PYTHON_CMD "py -3.11") & \
3232
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.10" && setx PYTHON_CMD "py -3.12") & \
3333
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.14" && setx PYTHON_CMD "py -3.13") & \
34-
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.6" && setx PYTHON_CMD "py -3.14")
34+
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.7" && setx PYTHON_CMD "py -3.14") & \
35+
(if "%python%"=="3.15" setx PYTHON_VERSION "3.15.0-rc1" && setx PYTHON_CMD "py -3.15")
36+
3537

3638
# hadolint ignore=DL3059
3739
RUN choco install -r -y --pre --no-progress --force python --version=%PYTHON_VERSION%

‎ci/scripts/install_python.sh‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,11 @@ declare -A versions
2828
versions=([3.11]=3.11.9
2929
[3.12]=3.12.10
3030
[3.13]=3.13.14
31-
[3.14]=3.14.6
32-
[3.14t]=3.14.6)
31+
[3.14]=3.14.7
32+
[3.14t]=3.14.7
33+
[3.15]=3.15.0rc1
34+
[3.15t]=3.15.0rc1
35+
)
3336

3437
if [ "$#" -ne 2 ]; then
3538
echo "Usage: $0 <platform> <version>"
@@ -46,11 +49,14 @@ full_version=${versions[$2]}
4649
if [ "$platform" = "macOS" ]; then
4750
echo "Downloading Python installer..."
4851

52+
# Match python.org directory by truncating any pre-release suffix:
53+
# 3.15.0rc1 is https://www.python.org/ftp/python/3.15.0/python-3.15.0rc1-macos11.pkg
54+
release_version="${full_version%%[abrc]*}"
4955
fname="python-${full_version}-macos11.pkg"
50-
wget "https://www.python.org/ftp/python/${full_version}/${fname}"
56+
wget "https://www.python.org/ftp/python/${release_version}/${fname}"
5157

5258
echo "Installing Python..."
53-
if [[ $2 == "3.14t" ]]; then
59+
if [[ $2 == *t ]]; then
5460
# Extract the base version without 't' suffix
5561
base_version="${version%t}"
5662
# See https://github.com/python/cpython/issues/120098#issuecomment-2151122033 for more info on this.
@@ -78,7 +84,7 @@ EOF
7884
rm "$fname"
7985

8086
python="/Library/Frameworks/Python.framework/Versions/${version}/bin/python${version}"
81-
if [[ $2 == "3.14t" ]]; then
87+
if [[ $2 == *t ]]; then
8288
base_version="${version%t}"
8389
python="/Library/Frameworks/PythonT.framework/Versions/${base_version}/bin/python${base_version}t"
8490
fi

‎ci/scripts/python_wheel_macos_build.sh‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,15 @@ else
4848
exit 1
4949
fi
5050

51+
wheel_build_binary_flag="--only-binary=:all:"
52+
if python -c "import sys; sys.exit(0 if sys.version_info[:2] == (3, 15) else 1)"; then
53+
# TODO: 3.15 workaround due to pyyaml (via libcst). Remove once pyyaml ships 3.15 wheels.
54+
wheel_build_binary_flag=""
55+
fi
56+
5157
pip install \
5258
--force-reinstall \
53-
--only-binary=:all: \
59+
${wheel_build_binary_flag} \
5460
--upgrade \
5561
-r "${source_dir}/python/requirements-wheel-build.txt"
5662
pip install "delocate>=0.10.3"

‎ci/scripts/python_wheel_unix_test.sh‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,12 @@ export PYARROW_TEST_GANDIVA=OFF
4747
export PYARROW_TEST_GCS=${ARROW_GCS}
4848
export PYARROW_TEST_HDFS=ON
4949
export PYARROW_TEST_ORC=ON
50-
export PYARROW_TEST_PANDAS=ON
50+
# TODO: To be removed once pandas provides wheels for Python 3.15
51+
if python -c "import sys; sys.exit(0 if sys.version_info < (3, 15) else 1)"; then
52+
export PYARROW_TEST_PANDAS=ON
53+
else
54+
export PYARROW_TEST_PANDAS=OFF
55+
fi
5156
export PYARROW_TEST_PARQUET=ON
5257
export PYARROW_TEST_PARQUET_ENCRYPTION=ON
5358
export PYARROW_TEST_SUBSTRAIT=${ARROW_SUBSTRAIT}

‎ci/scripts/python_wheel_windows_test.bat‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ set PYARROW_TEST_GANDIVA=OFF
2626
set PYARROW_TEST_GCS=ON
2727
set PYARROW_TEST_HDFS=ON
2828
set PYARROW_TEST_ORC=ON
29-
set PYARROW_TEST_PANDAS=ON
29+
@REM TODO: To be removed once pandas provides wheels for Python 3.15
30+
%PYTHON_CMD% -c "import sys; sys.exit(0 if sys.version_info < (3, 15) else 1)"
31+
if errorlevel 1 (set PYARROW_TEST_PANDAS=OFF) else (set PYARROW_TEST_PANDAS=ON)
3032
set PYARROW_TEST_PARQUET=ON
3133
set PYARROW_TEST_PARQUET_ENCRYPTION=ON
3234
set PYARROW_TEST_SUBSTRAIT=ON

‎compose.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1178,7 +1178,7 @@ services:
11781178
args:
11791179
arch: ${ARCH}
11801180
arch_short: ${ARCH_SHORT}
1181-
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2026.05.31-3
1181+
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2026.08.08-1
11821182
manylinux: 2_28
11831183
python: ${PYTHON}
11841184
python_abi_tag: ${PYTHON_ABI_TAG}
@@ -1203,7 +1203,7 @@ services:
12031203
args:
12041204
arch: ${ARCH}
12051205
arch_short: ${ARCH_SHORT}
1206-
base: quay.io/pypa/musllinux_1_2_${ARCH_ALIAS}:2026.05.31-3
1206+
base: quay.io/pypa/musllinux_1_2_${ARCH_ALIAS}:2026.08.08-1
12071207
musllinux: 1_2
12081208
python: ${PYTHON}
12091209
python_abi_tag: ${PYTHON_ABI_TAG}
@@ -1244,7 +1244,7 @@ services:
12441244
args:
12451245
base: "alpine:${ALPINE_LINUX}"
12461246
python_version: ${PYTHON}
1247-
build_date: "20260510" # python-build-standalone release date
1247+
build_date: "20260807" # python-build-standalone release date
12481248
arch: ${ARCH_ALIAS}
12491249
context: .
12501250
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile
@@ -1288,7 +1288,7 @@ services:
12881288
args:
12891289
base: "alpine:${ALPINE_LINUX}"
12901290
python_version: ${PYTHON}
1291-
build_date: "20260510" # python-build-standalone release date
1291+
build_date: "20260807" # python-build-standalone release date
12921292
arch: ${ARCH_ALIAS}
12931293
context: .
12941294
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile

0 commit comments

Comments
 (0)