Skip to content

BUG: #1786 resume supporting numpy 2.5 - #1802

Merged
Dr-Irv merged 18 commits into
pandas-dev:mainfrom
cmp0xff:chore/yw/numpy-2-5
Jul 17, 2026
Merged

Dr-Irv merged 18 commits into
pandas-dev:mainfrom
cmp0xff:chore/yw/numpy-2-5

Conversation

@cmp0xff

@cmp0xff cmp0xff commented Jun 29, 2026

Copy link
Copy Markdown
Contributor
  • Partially addresses mypy raises errors after numpy 2.5.0 #1786
  • Tests added (Please use assert_type() to assert the type of any return value)
  • If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

@cmp0xff
cmp0xff force-pushed the chore/yw/numpy-2-5 branch from 6eb620e to 10b0629 Compare June 29, 2026 18:07
@cmp0xff cmp0xff changed the title fix: #1786 resume supporting numpy 2.5 BUG: #1786 resume supporting numpy 2.5 Jun 29, 2026
@cmp0xff
cmp0xff marked this pull request as ready for review June 29, 2026 19:05

@loicdiridollou loicdiridollou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a lot of work and I am a bit worried of loosing track about all the changes and exceptions that were carved out here to make it compliant with numpy 2.5.
Is it something that is bound to stay with numpy? Or later get fixed in one way or another?

Comment thread tests/arrays/test_datetime_array.py Resolved
@cmp0xff
cmp0xff marked this pull request as draft July 9, 2026 08:58
@cmp0xff
cmp0xff marked this pull request as ready for review July 13, 2026 07:58
@cmp0xff
cmp0xff force-pushed the chore/yw/numpy-2-5 branch from e63894f to 23a4987 Compare July 13, 2026 15:02
@cmp0xff
cmp0xff requested a review from loicdiridollou July 14, 2026 09:35

@loicdiridollou loicdiridollou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I think this is good to go, it is a ton of splits but there does not seem to be a better setup than this so thanks @cmp0xff for doing all the heavy lifting!
@Dr-Irv please merge if you are onboard with the changes

@cmp0xff
cmp0xff requested a review from Dr-Irv July 15, 2026 14:05

@Dr-Irv Dr-Irv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I figured out most of the problems and created a mypy issue that explains why we have to do the tests for python 3.12 in various places.

for pd.array, if you comment out:

@overload
def array(
    data: Sequence[Any], dtype: BuiltinObjectDtypeArg | None = None, copy: bool = True
) -> NumpyExtensionArray: ...

then we get the correct results for all of the pd.array() calls where you needed that test.

And there are other places too (e.g., Timedelta.__eq__(), that have the same issue. So if they fix that in mypy, we won't need any of the funky tests.

I think it is worth changing the tests that say if sys.version_info >= (3, 12): to use a constant NUMPY_GE_251 that is defined in tests/__init__.py to be sys.version_info >= (3, 12), then it is clear that what we are testing for is the newer version of numpy, not for the difference in the python version.

Comment thread tests/scalars/test_scalars.py Resolved
Comment thread tests/arrays/test_datetime_array.py Resolved
Comment thread tests/arrays/test_datetime_array.py Resolved
Comment thread tests/arrays/test_datetime_array.py Outdated
DatetimeArray,
)
if sys.version_info >= (3, 12):
# TODO: pandas-dev/pandas-stubs#1786

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change reference to python/mypy#21733

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread tests/arrays/test_datetime_array.py Resolved
Comment thread tests/scalars/test_scalars.py Resolved
Comment thread tests/series/test_series.py Resolved
Comment thread tests/series/test_series.py Resolved
Comment thread tests/test_dtypes.py Resolved
Comment thread tests/test_dtypes.py Resolved

@cmp0xff cmp0xff left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Dr-Irv , I planned to postpone further tunings to the next PRs, but you have already done much of the research.

I think it is worth changing the tests that say if sys.version_info >= (3, 12): to use a constant NUMPY_GE_251 that is defined in tests/__init__.py to be sys.version_info >= (3, 12), then it is clear that what we are testing for is the newer version of numpy, not for the difference in the python version.

Unfortunately type checkers do not use such a variable to skip conditional branches, they skip only when seeing if sys.version_info >= (3, 12) explicitly, not when seeing if NUMPY_GE_251 where NUMPY_GE_251 = sys.version_info >= (3, 12).

Comment thread tests/scalars/test_scalars.py Resolved
Comment thread tests/test_dtypes.py Resolved
Comment thread tests/test_dtypes.py Resolved
Comment thread tests/arrays/test_integer.py Resolved
Comment thread tests/scalars/test_scalars.py Resolved
Comment thread tests/arrays/test_datetime_array.py Resolved
Comment thread tests/arrays/test_datetime_array.py Outdated
dt_nat = cast(list[np.datetime64 | NaTType], [np_dt, pd.NaT])
check(assert_type(pd.array(dt_nat), DatetimeArray), DatetimeArray)
if sys.version_info >= (3, 12):
# TODO: pandas-dev/pandas-stubs#1786

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread tests/scalars/test_scalars.py Outdated
eq_1d1 = check(assert_type(td == arr_1d, np_1darray_bool), np_1darray_bool)
ne_1d1 = check(assert_type(td != arr_1d, np_1darray_bool), np_1darray_bool)
assert (eq_1d1 != ne_1d1).all()
# TODO: pandas-dev/pandas-stubs#1786 facebook/pyrefly#3977

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread tests/scalars/test_scalars.py Resolved
Comment thread tests/arrays/test_datetime_array.py Resolved
@cmp0xff
cmp0xff requested a review from Dr-Irv July 16, 2026 07:42

@Dr-Irv Dr-Irv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately type checkers do not use such a variable to skip conditional branches, they skip only when seeing if sys.version_info >= (3, 12) explicitly, not when seeing if NUMPY_GE_251 where NUMPY_GE_251 = sys.version_info >= (3, 12).

OK, but I think we need better comments here as opposed to just cross-referencing the mypy issue or a pyrefly issue.

Every time that you have

if sys.version_info >= (3, 12):

there are one of 2 cases:

  1. We need the test because numpy >= 2.5 has broken behavior with mypy as referred to in the issue, and with python 3.12, we are using numpy >= 2.5
  2. We need the test because numpy >= 2.5 gives us the right behavior, and we are using numpy=2.4.6 with python 3.11 so we have to add ignores to deal with that.

So I'd prefer a comment in each case (in addition to the cross referencing comments) that says which of those 2 cases is applicable.

Could be as simple as doing

if sys.version_info >= (3, 12):   # numpy  >= 2.5

so that we remember the reason is because of the numpy versioning, NOT because of the python version.

@cmp0xff cmp0xff left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explained in more detail in 367f567

Comment thread tests/arrays/test_datetime_array.py Outdated
DatetimeArray,
)
if sys.version_info >= (3, 12):
# TODO: pandas-dev/pandas-stubs#1786

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread tests/arrays/test_datetime_array.py Outdated
dt_nat = cast(list[np.datetime64 | NaTType], [np_dt, pd.NaT])
check(assert_type(pd.array(dt_nat), DatetimeArray), DatetimeArray)
if sys.version_info >= (3, 12):
# TODO: pandas-dev/pandas-stubs#1786

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread tests/scalars/test_scalars.py Outdated
eq_1d1 = check(assert_type(td == arr_1d, np_1darray_bool), np_1darray_bool)
ne_1d1 = check(assert_type(td != arr_1d, np_1darray_bool), np_1darray_bool)
assert (eq_1d1 != ne_1d1).all()
# TODO: pandas-dev/pandas-stubs#1786 facebook/pyrefly#3977

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmp0xff
cmp0xff requested a review from Dr-Irv July 17, 2026 16:32

@Dr-Irv Dr-Irv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @cmp0xff

@Dr-Irv
Dr-Irv merged commit 933d405 into pandas-dev:main Jul 17, 2026
14 checks passed
@cmp0xff
cmp0xff deleted the chore/yw/numpy-2-5 branch July 18, 2026 16:21
@cmp0xff cmp0xff linked an issue Aug 2, 2026 that may be closed by this pull request
cmp0xff added a commit to cmp0xff/pandas-stubs that referenced this pull request Aug 28, 2026
* fix: partially support numpy 2-5

* fix: typing

* fix: typing

* fix: rebase

* fix: pyrefly

* fix: tests

* fix: installed stubs

* fix: https://github.com/pandas-dev/pandas-stubs/actions/runs/28395061571

* remove _dist discrepency

* force numpy 2.5 for python >= 3.12 for local stubs; without this change, for python 3.12 and 3.13, numpy 2.4 will be installed

* conditional warnings

* pyrefly issue

* python/mypy#21733

* conditional warnings revisited

* deduplicate + black

* regressions in pyrefly

* pandas-dev#1802 (review)

* revert a non-functional change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mypy raises errors after numpy 2.5.0

3 participants