Skip to content

Commit 044cb36

Browse files
committed
apacheGH-55263: [Python][Annotations] Core data structures and internal types
Squash of apache#55264 (5a026a0) on top of main, to check the stacked compute stubs against it. Not for merging.
1 parent 48e4abf commit 044cb36

45 files changed

Lines changed: 4497 additions & 84 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/dev.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,12 @@ jobs:
6767
- name: Run pre-commit
6868
run: |
6969
pre-commit run --all-files --color=always --show-diff-on-failure
70+
- name: Run pre-commit for type stubs
71+
env:
72+
# The previous step already ran the ruff hooks.
73+
SKIP: ruff-check,ruff-format
74+
run: |
75+
pre-commit run --hook-stage manual python-stubs --all-files --color=always --show-diff-on-failure
7076
7177
release:
7278
name: Source Release and Merge Script on ${{ matrix.runs-on }}

‎.pre-commit-config.yaml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,70 @@ repos:
145145
(
146146
?^python/pyarrow/vendored/|
147147
)
148+
# Linting, formatting and type checking of the type stubs. The type checkers
149+
# don't run on commit, only on request (the Lint CI job runs them too):
150+
# pre-commit run --hook-stage manual python-stubs --all-files
151+
# The type checkers run with uvx, in the environment of
152+
# python/requirements-stubs.txt. They don't need PyArrow installed: the
153+
# modules that aren't annotated yet, e.g. pyarrow.compute, are treated as Any
154+
# (see python/pyproject.toml).
155+
- repo: https://github.com/astral-sh/ruff-pre-commit
156+
rev: v0.17.2
157+
hooks:
158+
- id: ruff-check
159+
alias: python-stubs
160+
name: Python Stubs Lint (ruff)
161+
args:
162+
- "--fix"
163+
files: >-
164+
^python/pyarrow-stubs/
165+
types:
166+
- pyi
167+
- id: ruff-format
168+
alias: python-stubs
169+
name: Python Stubs Format (ruff)
170+
files: >-
171+
^python/pyarrow-stubs/
172+
types:
173+
- pyi
174+
- repo: local
175+
hooks:
176+
- id: mypy
177+
alias: python-stubs
178+
name: Python Stubs Type Check (mypy)
179+
stages:
180+
- manual
181+
language: python
182+
entry: bash -c "cd python && exec uvx --with-requirements requirements-stubs.txt mypy"
183+
pass_filenames: false
184+
files: >-
185+
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
186+
additional_dependencies: &uv
187+
- uv==0.13.7
188+
- id: pyright
189+
alias: python-stubs
190+
name: Python Stubs Type Check (pyright)
191+
stages:
192+
- manual
193+
language: python
194+
entry: uvx --with-requirements python/requirements-stubs.txt pyright -p python/pyproject.toml
195+
pass_filenames: false
196+
files: >-
197+
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
198+
additional_dependencies: *uv
199+
- id: ty
200+
alias: python-stubs
201+
name: Python Stubs Type Check (ty)
202+
stages:
203+
- manual
204+
language: python
205+
# Unset the VIRTUAL_ENV of the hook, so that ty uses the environment
206+
# of uvx.
207+
entry: env -u VIRTUAL_ENV -u CONDA_PREFIX uvx --with-requirements python/requirements-stubs.txt ty check --project python
208+
pass_filenames: false
209+
files: >-
210+
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
211+
additional_dependencies: *uv
148212
- repo: https://github.com/MarcoGorelli/cython-lint
149213
rev: v0.21.1
150214
hooks:

‎ci/scripts/python_test_type_annotations.sh‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,20 @@ if [ -n "${ARROW_PYTHON_VENV:-}" ]; then
2525
. "${ARROW_PYTHON_VENV}/bin/activate"
2626
fi
2727

28-
# Install library stubs. Note some libraries contain their own type hints so they need to be installed.
29-
pip install fsspec pandas-stubs scipy-stubs types-cffi types-psutil types-requests types-python-dateutil
28+
cd "${pyarrow_dir}"
3029

31-
# Install type checkers
32-
pip install mypy pyright ty
30+
# Install type checkers and library stubs. Note some libraries contain their
31+
# own type hints so they need to be installed.
32+
pip install -r requirements-stubs.txt
3333

3434
# Run type checkers
35-
cd "${pyarrow_dir}"
3635
mypy
3736
pyright --stats
3837
ty check --verbose --output-format concise
38+
39+
# The type annotation tests are type checked above; also run them to verify
40+
# the expected annotated types against the runtime types.
41+
pytest -r s pyarrow-stubs-tests
42+
43+
# Check that the stubs match the signatures of the installed PyArrow.
44+
python scripts/run_stubtest.py

‎ci/scripts/python_wheel_validate_contents.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ def validate_wheel(path):
6060
), "pyarrow/py.typed is present in the wheel."
6161

6262
source_root = Path(__file__).resolve().parents[2]
63-
stubs_dir = source_root / "python" / "pyarrow-stubs" / "pyarrow"
63+
stubs_dir = source_root / "python" / "pyarrow-stubs"
6464
assert stubs_dir.exists(), f"Stub source directory not found: {stubs_dir}"
6565

6666
expected_stub_files = {
@@ -75,7 +75,7 @@ def validate_wheel(path):
7575
}
7676

7777
assert not (wheel_stub_files == expected_stub_files), (
78-
"Wheel .pyi files do not differ from python/pyarrow-stubs/pyarrow.\n"
78+
"Wheel .pyi files do not differ from python/pyarrow-stubs.\n"
7979
f"Missing in wheel: {sorted(expected_stub_files - wheel_stub_files)}\n"
8080
f"Unexpected in wheel: {sorted(wheel_stub_files - expected_stub_files)}"
8181
)

‎docs/source/developers/python/development.rst‎

Lines changed: 99 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -105,46 +105,102 @@ Type Checking
105105
=============
106106

107107
PyArrow provides type stubs (``*.pyi`` files) for static type checking. These
108-
stubs are located in the ``pyarrow-stubs/`` directory and are automatically
109-
included in the distributed wheel packages.
108+
stubs are located in the ``pyarrow-stubs/`` directory, which is laid out as a
109+
:pep:`561` stub-only package for ``pyarrow``, so type checkers and IDEs pick
110+
it up from the ``arrow/python`` directory. The stubs aren't included in the
111+
distributed wheel packages until they are complete (see
112+
`GH-49831 <https://github.com/apache/arrow/issues/49831>`_).
113+
114+
Run all the commands in this section from the ``arrow/python`` directory,
115+
which holds the type checker configurations in ``pyproject.toml``:
116+
117+
.. code-block::
118+
119+
$ cd arrow/python
110120
111121
Running Type Checkers
112122
---------------------
113123

114124
We support multiple type checkers. Their configurations are in
115125
``pyproject.toml``.
116126

127+
The easiest way to run all of them is with ``pre-commit``, which installs
128+
the type checkers and stub packages pinned in ``requirements-stubs.txt`` (with
129+
uv). This also lints and formats the stubs with ruff:
130+
131+
.. code-block::
132+
133+
$ pre-commit run --hook-stage manual python-stubs --all-files
134+
135+
The type checkers only run on request, while ruff also runs on commit. The
136+
Lint CI job runs all of them as well. CI also runs the type checkers, the type
137+
stub tests and stubtest (see below) against the PyArrow it builds, with
138+
``ci/scripts/python_test_type_annotations.sh``.
139+
140+
To run a type checker directly instead, install the pinned type checkers and
141+
stub packages in your PyArrow development environment:
142+
143+
.. code-block::
144+
145+
$ pip install -r requirements-stubs.txt
146+
147+
Checking the PyArrow tests also needs their dependencies from
148+
``requirements-test.txt``.
149+
117150
**mypy**
118151

119152
To run mypy on the PyArrow codebase:
120153

121154
.. code-block::
122155
123-
$ cd arrow/python
124156
$ mypy
125157
126158
The mypy configuration is in the ``[tool.mypy]`` section of ``pyproject.toml``.
159+
mypy only uses the stubs when it checks them too, so pass ``pyarrow-stubs``
160+
along with any other files you check:
161+
162+
.. code-block::
163+
164+
$ mypy pyarrow-stubs pyarrow/tests/test_array.py
127165
128166
**pyright**
129167

130168
To run pyright:
131169

132170
.. code-block::
133171
134-
$ cd arrow/python
135172
$ pyright
136173
137174
The pyright configuration is in the ``[tool.pyright]`` section of ``pyproject.toml``.
138175

139176
**ty**
140177

141-
To run ty (note: currently only partially configured):
178+
To run ty:
142179

143180
.. code-block::
144181
145-
$ cd arrow/python
146182
$ ty check
147183
184+
**stubtest**
185+
186+
mypy's stubtest compares the stubs with the signatures of the installed
187+
PyArrow. Run it with:
188+
189+
.. code-block::
190+
191+
$ python scripts/run_stubtest.py
192+
193+
Stubs that aren't written yet are skipped. The known mismatches are listed in
194+
``pyarrow-stubs-tests/stubtest_allowlist.txt``; remove the entries of the
195+
mismatches you fix, as stubtest fails on unused entries.
196+
197+
Type Checking in an IDE
198+
-----------------------
199+
200+
Editors also run the type checkers from the folder they open. If you open the
201+
whole ``arrow`` repository in VS Code, point the type checker at ``arrow/python``
202+
instead.
203+
148204
Maintaining Type Stubs
149205
-----------------------
150206

@@ -169,6 +225,43 @@ in downstream projects and for users' IDEs.
169225
Note: ``py.typed`` marker file in the ``pyarrow/`` directory indicates to type
170226
checkers that PyArrow supports type checking according to :pep:`561`.
171227

228+
Testing Type Stubs
229+
------------------
230+
231+
The tests in ``pyarrow-stubs-tests/`` pin down the annotated types of public
232+
APIs. Each check pairs :func:`typing.assert_type`, which every type checker
233+
verifies statically, with a runtime ``isinstance`` check:
234+
235+
.. code-block:: python
236+
237+
from typing import TYPE_CHECKING
238+
239+
import pyarrow as pa
240+
241+
from _check import assert_type, check
242+
243+
def test_list_type() -> None:
244+
lst = pa.list_(pa.int32())
245+
check(assert_type(lst, pa.ListType), pa.ListType)
246+
check(assert_type(lst.value_type, pa.DataType), pa.DataType)
247+
248+
if TYPE_CHECKING:
249+
pa.list_(1) # type: ignore[call-overload]
250+
251+
The type checkers check these files as part of the regular runs above, and
252+
pytest runs them against the installed PyArrow to make sure the stubs agree
253+
with what PyArrow actually returns:
254+
255+
.. code-block::
256+
257+
$ pytest pyarrow-stubs-tests
258+
259+
Code under ``if TYPE_CHECKING:`` is never executed. It holds the usage that
260+
should fail type checking, which is asserted with a
261+
``# type: ignore[<mypy code>]`` comment: unused ignores are reported as errors
262+
in these tests, so the test fails once the usage is no longer rejected. ty
263+
doesn't honor mypy error codes, so it doesn't check invalid usage.
264+
172265
Doctest
173266
=======
174267

‎python/CMakeLists.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1079,15 +1079,15 @@ endif()
10791079
#
10801080
# TODO(GH-32609): Reintroduce type stubs into PyArrow wheels.
10811081
# GH-49831: Temporarily do not install pyarrow-stubs into wheels.
1082-
# Stubs live in pyarrow-stubs/pyarrow/ during development but are installed
1082+
# Stubs live in pyarrow-stubs/ during development but are installed
10831083
# alongside the package so type checkers can find them (PEP 561).
10841084
# The stubs are currently incomplete, and some type checkers consume .pyi files
10851085
# even without the py.typed marker. Re-enable this when the stubs are complete.
10861086
if(PYARROW_REQUIRE_STUB_DOCSTRINGS)
10871087
message(FATAL_ERROR "PYARROW_REQUIRE_STUB_DOCSTRINGS cannot be used while "
10881088
"pyarrow-stubs are omitted from wheels (GH-49831).")
10891089
endif()
1090-
# set(PYARROW_STUBS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/pyarrow-stubs/pyarrow")
1090+
# set(PYARROW_STUBS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/pyarrow-stubs")
10911091
# if(EXISTS "${PYARROW_STUBS_SOURCE_DIR}")
10921092
# install(DIRECTORY "${PYARROW_STUBS_SOURCE_DIR}/"
10931093
# DESTINATION "."
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""Helpers for the type annotation tests in this directory.
19+
20+
Each test pairs a static and a runtime check of the same expression::
21+
22+
check(assert_type(pa.list_(pa.int32()), pa.ListType), pa.ListType)
23+
24+
``assert_type`` is verified by every type checker we run in CI (mypy,
25+
pyright and ty) and is a no-op at runtime. ``check`` runs under pytest and
26+
verifies that the value really is an instance of the given class, so the
27+
stubs can't drift away from what pyarrow actually returns.
28+
29+
Code that type checkers should *reject* goes under ``if TYPE_CHECKING:``
30+
with a ``# type: ignore[<mypy code>]`` comment. The block never runs, and
31+
mypy and pyright are configured to report unused ignores in these tests, so
32+
the test fails if the usage stops being an error.
33+
"""
34+
35+
from typing import Any, TypeVar, assert_type
36+
37+
__all__ = ["assert_type", "check"]
38+
39+
_T = TypeVar("_T")
40+
41+
42+
def check(actual: _T, klass: type[Any] | tuple[type[Any], ...]) -> _T:
43+
"""Assert that ``actual`` is an instance of ``klass`` and return it."""
44+
if not isinstance(actual, klass):
45+
raise AssertionError(
46+
f"Expected an instance of {klass}, got {type(actual)}: {actual!r}"
47+
)
48+
return actual
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
import sys
19+
import types
20+
21+
import pyarrow as pa
22+
23+
24+
def _stub_only_type(name: str) -> type[pa.DataType]:
25+
# The factories of the stub-only DataType subclasses (Int32Type, ...)
26+
# return plain DataType instances at runtime.
27+
if name.endswith("Type") and not name.startswith("_"):
28+
return pa.DataType
29+
raise AttributeError(name)
30+
31+
32+
# pyarrow._stubs_typing only exists in the type stubs. Provide it at runtime,
33+
# so that the tests can import the stub-only types without a TYPE_CHECKING
34+
# block.
35+
_stubs_typing = types.ModuleType("pyarrow._stubs_typing")
36+
setattr(_stubs_typing, "__getattr__", _stub_only_type)
37+
sys.modules["pyarrow._stubs_typing"] = _stubs_typing

0 commit comments

Comments
 (0)