Skip to content

Commit bb81a6e

Browse files
rokkouraulcd
authored
GH-49102: [CI] Add type checking infrastructure and CI workflow for type annotations (#48618)
### Rationale for this change This is the first in series of PRs adding type annotations to pyarrow and resolving #32609. ### What changes are included in this PR? This PR establishes infrastructure for type checking: - Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows - Configures type checkers to validate stub files (excluding source files for now) - Adds PEP 561 `py.typed` marker to enable type checking - Updates wheel build scripts to include stub files in distributions - Creates initial minimal stub directory structure - Updates developer documentation with type checking workflow ### Are these changes tested? No. This is mostly a CI change. ### Are there any user-facing changes? This does not add any actual annotations (only `py.typed` marker) so user should not be affected. * GitHub Issue: #32609 * GitHub Issue: #49102 Lead-authored-by: Rok Mihevc <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Rok Mihevc <[email protected]>
1 parent a82edf9 commit bb81a6e

15 files changed

Lines changed: 472 additions & 3 deletions

‎.pre-commit-config.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,7 @@ repos:
337337
?^ci/scripts/python_sdist_build\.sh$|
338338
?^ci/scripts/python_sdist_test\.sh$|
339339
?^ci/scripts/python_wheel_unix_test\.sh$|
340+
?^ci/scripts/python_test_type_annotations\.sh$|
340341
?^ci/scripts/r_build\.sh$|
341342
?^ci/scripts/r_revdepcheck\.sh$|
342343
?^ci/scripts/release_test\.sh$|
@@ -377,6 +378,7 @@ repos:
377378
# TODO: Remove this when we fix all lint failures
378379
files: >-
379380
(
381+
?^ci/scripts/python_test_type_annotations\.sh$|
380382
?^dev/release/05-binary-upload\.sh$|
381383
?^dev/release/binary-recover\.sh$|
382384
?^dev/release/post-03-binary\.sh$|

‎ci/conda_env_python.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ cython>=3.1
2424
cloudpickle
2525
fsspec
2626
hypothesis
27+
libcst>=1.8.6
2728
numpy>=1.16.6
2829
pytest
2930
pytest-faulthandler

‎ci/scripts/python_build.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ export PYARROW_PARALLEL=${n_jobs}
8181
: "${CMAKE_PREFIX_PATH:=${ARROW_HOME}}"
8282
export CMAKE_PREFIX_PATH
8383
export LD_LIBRARY_PATH=${ARROW_HOME}/lib:${LD_LIBRARY_PATH}
84+
export DYLD_LIBRARY_PATH=${ARROW_HOME}/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
8485

8586
# https://github.com/apache/arrow/issues/41429
8687
# TODO: We want to out-of-source build. This is a workaround. We copy
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
set -ex
21+
pyarrow_dir=${1}
22+
23+
if [ -n "${ARROW_PYTHON_VENV:-}" ]; then
24+
# shellcheck source=/dev/null
25+
. "${ARROW_PYTHON_VENV}/bin/activate"
26+
fi
27+
28+
# Install library stubs. Note some libraries contain their own type hints so they need to be installed.
29+
pip install fsspec pandas-stubs scipy-stubs types-cffi types-psutil types-requests types-python-dateutil
30+
31+
# Install type checkers
32+
pip install mypy pyright ty
33+
34+
# Run type checkers
35+
cd "${pyarrow_dir}"
36+
mypy
37+
pyright
38+
ty check

‎ci/scripts/python_wheel_validate_contents.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ def validate_wheel(path):
3838
for info in f.filelist), \
3939
f"{filename} is missing from the wheel."
4040
print(f"The wheel: {wheels[0]} seems valid.")
41-
41+
# TODO(GH-32609): Validate some docstrings were generated and added.
4242

4343
def main():
4444
parser = argparse.ArgumentParser()

‎compose.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1539,7 +1539,8 @@ services:
15391539
/arrow/ci/scripts/python_build.sh /arrow /build &&
15401540
pip install -e /arrow/dev/archery[numpydoc] &&
15411541
archery numpydoc --allow-rule GL10,PR01,PR03,PR04,PR05,PR10,RT03,YD01 &&
1542-
/arrow/ci/scripts/python_test.sh /arrow"]
1542+
/arrow/ci/scripts/python_test.sh /arrow &&
1543+
/arrow/ci/scripts/python_test_type_annotations.sh /arrow/python"]
15431544

15441545
conda-python-dask:
15451546
# Possible $DASK parameters:

‎docs/source/developers/python/development.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,74 @@ The test groups currently include:
101101
* ``s3``: Tests for Amazon S3
102102
* ``tensorflow``: Tests that involve TensorFlow
103103

104+
Type Checking
105+
=============
106+
107+
PyArrow provides type stubs (``*.pyi`` files) for static type checking. These
108+
stubs are located in the ``pyarrow-stubs/`` directory and are automatically
109+
included in the distributed wheel packages.
110+
111+
Running Type Checkers
112+
---------------------
113+
114+
We support multiple type checkers. Their configurations are in
115+
``pyproject.toml``.
116+
117+
**mypy**
118+
119+
To run mypy on the PyArrow codebase:
120+
121+
.. code-block::
122+
123+
$ cd arrow/python
124+
$ mypy
125+
126+
The mypy configuration is in the ``[tool.mypy]`` section of ``pyproject.toml``.
127+
128+
**pyright**
129+
130+
To run pyright:
131+
132+
.. code-block::
133+
134+
$ cd arrow/python
135+
$ pyright
136+
137+
The pyright configuration is in the ``[tool.pyright]`` section of ``pyproject.toml``.
138+
139+
**ty**
140+
141+
To run ty (note: currently only partially configured):
142+
143+
.. code-block::
144+
145+
$ cd arrow/python
146+
$ ty check
147+
148+
Maintaining Type Stubs
149+
-----------------------
150+
151+
Type stubs for PyArrow are maintained in the ``pyarrow-stubs/``
152+
directory. These stubs mirror the structure of the main ``pyarrow/`` package.
153+
154+
When adding or modifying public APIs:
155+
156+
1. **Update the corresponding ``.pyi`` stub file** in ``pyarrow-stubs/``
157+
to reflect the new or changed function/class signatures.
158+
159+
2. **Include type annotations** where possible. For Cython modules or
160+
dynamically generated APIs such as compute kernels add the corresponding
161+
stub in ``pyarrow-stubs/``.
162+
163+
3. **Run type checkers** to ensure the stubs are correct and complete.
164+
165+
The stub files are automatically copied into the built wheel during the build
166+
process and will be included when users install PyArrow, enabling type checking
167+
in downstream projects and for users' IDEs.
168+
169+
Note: ``py.typed`` marker file in the ``pyarrow/`` directory indicates to type
170+
checkers that PyArrow supports type checking according to :pep:`561`.
171+
104172
Doctest
105173
=======
106174

‎python/MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ include ../NOTICE.txt
44

55
global-include CMakeLists.txt
66
graft pyarrow
7+
graft pyarrow-stubs
78
graft cmake_modules
89

910
global-exclude *.so
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""Type stubs for PyArrow.
19+
20+
This is a placeholder stub file.
21+
Complete type annotations will be added in subsequent PRs.
22+
"""
23+
24+
from typing import Any
25+
26+
# TODO(GH-48970): remove __getattr__ before release as this
27+
# will annotate non-existing attributes as Any.
28+
# https://github.com/apache/arrow/issues/48970
29+
def __getattr__(name: str) -> Any: ...

‎python/pyarrow/py.typed‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

0 commit comments

Comments
 (0)