TYP: regression between 2.1.3 and 2.2.0 (mypy only)

Describe the issue:

The following code seems correct, and passes type checking under 2.1.3 but not 2.2.0:

import numpy as np

rng = np.random.default_rng(27446968)
n_states = 5

P = rng.random(size=(n_states, n_states))
p = rng.random(n_states)

p = P.T @ p

I suppose mypy is flagging that the float width can increase, but since float64 is used by random, that won't happen in this scenario.

Reproduce the code example:

N/A

Error message:

foo.py:10: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[floating[Any]]]", variable has type "ndarray[tuple[int, ...], dtype[float64]]")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Python and NumPy Versions:

Python 3.13, mypy 1.13, numpy 2.2.0

Runtime Environment:

No response

Context for the issue:

No response

We're seeing these failures in http://github.com/google/jax/ as well.

Much simpler reproducer:

import numpy as np

x = np.zeros(1)
x = x / 1

results in:

test.py:4: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[floating[Any]]]", variable has type "ndarray[tuple[int], dtype[float64]]")  [assignment]

If I try:

import numpy as np

x = np.zeros(1, dtype=float)
x = x / 1

I get:

test.py:4: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[floating[Any]]]", variable has type "ndarray[tuple[int], dtype[Any]]")  [assignment]

Trying:

import numpy as np

x = np.zeros(1, dtype=np.float32)
x = x / 1

gives:

test.py:4: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[floating[Any]]]", variable has type "ndarray[tuple[int], dtype[floating[_32Bit]]]")  [assignment]

Similarly, this:

import numpy as np

x = np.zeros(1, dtype=np.float64)
x = x / 1

gives:

test.py:4: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[floating[Any]]]", variable has type "ndarray[tuple[int], dtype[float64]]")  [assignment]

I have no idea how to annotate this in a way that will work.

@jorenham

Looks like this was introduced in #27767: we need tuple[int, ...] rather than tuple[int] in a number of places. cc/ @jorenham

That's unrelated to this issue. Both the left- and the right-hand-side have tuple[int, ...] as shape type.

Since numpy 2.2.0, numpy.float64 is a proper subtype of numpy.floating (and builtins.float). Before, it was simply a type alias, but that was invalid.
What you're seeing is a direct result of that, and it's correct behavior. To illustrate:

from typing import Any
import numpy as np

f: np.floating[Any]
f8: np.float64

f = f8  # accepted
f8 = f  # rejected on >=2.2.0, accepted on <2.2.0

So

The following code seems correct, and passes type checking under 2.1.3 but not 2.2.0:
import numpy as np

rng = np.random.default_rng(27446968)
n_states = 5

P = rng.random(size=(n_states, n_states))
p = rng.random(n_states)

p = P.T @ p
I suppose mypy is flagging that the float width can increase, but since float64 is used by random, that won't happen in this scenario.

Pyright gives a more helpful error message in this case:

Type "floating[Any]" is not assignable to declared type "float64"
  "floating[Any]" is not assignable to "float64"

So the result of rng.random is a float64 array, which causes p to be inferred as a npt.NDArray[np.float64]. But P.T @ p isn't typed as narrowly as it could be (but technically it's valid), so it is inferred as npt.NDArray[np.floating]. And because floating is the supertype of float64, it cannot be assigned to it. And that is correct behavior.

To make this example work without typing errors, this is what you could do

import numpy as np
import numpy.typing as npt

rng = np.random.default_rng(27446968)
n_states = 5

P = rng.random(size=(n_states, n_states))

p: npt.NDArray[np.floating]
p = rng.random(n_states)
p = P.T @ p

@adamjstewart

Much simpler reproducer:

import numpy as np

x = np.zeros(1)
x = x / 1

This is yet another issue, and it has to do with shape-typing instead. While technically this isn't a bug, I'll admit that it's very annoying.

The np.zeros(1) returns an array with shape type tuple[int] and scalar type float64, and that actually the best possible way to annotate this.

But currently, the division operator doesn't take shape-typing into account, so x / 1 will ignore the shape-type of the input, and returns the "generic" shape type tuple[int, ...].
But since x already has a more-specific shape-type, it cannot be overwritten with one that is less specific.
It also isn't very precise when it comes to the scalar type, and returns np.floating instead of float64. Technically, that is correct, but I fully agree that it's sub-optimal.

Anyway, this is how you can fix it to work with both mypy and pyright:

import numpy as np
import numpy.typing as npt

x: npt.NDArray[np.floating]
x = np.zeros(1)
x = x / 1

You're other examples, @adamjstewart, can also be fixed in the same way.

Feel free to open a separate issue about these "suboptimally annotated arithmetic operator types".

We're seeing these failures in http://github.com/google/jax/ as well.

Could you be more specific?

We're seeing these failures in http://github.com/google/jax/ as well.

Could you be more specific?

I temporarily pinned NumPy 2.1 yesterday to fix the CI errors, but jax-ml/jax#25381 shows the errors under NumPy 2.2. In particular, this one prompted me to go searching for this issue:

jax/_src/mesh_utils.py:600: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[Any]]", variable has type "ndarray[tuple[int], dtype[Any]]")  [assignment]

I now see that this is probably the intended behavior: it's no longer safe to assume that an array is an array; every array annotation must now be specialized on its shape (as I said in the other thread, I suspect this will break a lot of valid code).

I found this issue when running into the following problem and chatted with hauntsaninja about it:

import numpy
from numpy.typing import NDArray

def f() -> NDArray[numpy.float64]:
    return numpy.zeros(2, dtype=numpy.float64) + numpy.float64(1.0)  # error: Incompatible return value type (got "ndarray[tuple[int, ...], dtype[floating[Any]]]", expected "ndarray[tuple[int, ...], dtype[float64]]")  [return-value]

First, a general remark. Statements like this

So for instance, mypy doesn't support @Property setters, ... But this isn't documented, and they even actively try to make it look like it's supported...

are not only wrong but insulting. Wrong, because Mypy has supported property setters for ages. It only does not support different getter and setter types (I guess this is what jorenham refers to). Insulting (and also wrong), because it suggests that people who contributed to Mypy are trying to mislead the Python community. In the case of different setter types, there was a change in the perception of the desired behaviour. This change is now tricky to implement because Mypy's property support is not consistently programmed with its descriptor support for historical reasons. I do not want to go into more details, but I want to clarify there is no active deception but only limited time. And usually, one (or at least me) finds fixing problems in their free time more attractive than documenting them.

But now to the actual topic. In my (and, I think, also hauntsaninja's) perception, this issue is about incomplete transitions in shape typing (tuple[int] vs tuple[int, ...]) and inheritance (e.g. floating vs float64) in Numpy and that Mypy reports the resulting inconsistencies more frequently than other type checkers, primarily because of --allow-redefinition is not enabled by default. However, jorenham says in his second least comment that neither of these is the topic of this issue. Could you please clarify what defect you mean, then?

The only other hint pointing to another concrete topic I could find is this one:

This is because mypy (unlike pyright) can't understand tuple[int, int, Unpack[tuple[int, ...]], or in words, "a shape type that is at least 1-dimensional".

But, again, I have the feeling you might have encountered a bug but are overgeneralising. Mypy supports Unpack:

from typing import Unpack

x: tuple[int, int, Unpack[tuple[int, ...]]]

x = (1,)  #  error: Incompatible types in assignment (expression has type "tuple[int]", variable has type "tuple[int, int, *tuple[int, ...]]")  [assignment]
x = (1, 2, 3, 4)
reveal_type(x)  # note: Revealed type is "tuple[builtins.int, builtins.int, builtins.int, builtins.int]"

I work with Numpy and Mypy (and sometimes try to improve the latter a little), so I want them to work well together. I hope for a constructive discussion.

I found this issue when running into the following problem and chatted with hauntsaninja about it:

import numpy
from numpy.typing import NDArray

def f() -> NDArray[numpy.float64]:
    return numpy.zeros(2, dtype=numpy.float64) + numpy.float64(1.0)  # error: Incompatible return value type (got "ndarray[tuple[int, ...], dtype[floating[Any]]]", expected "ndarray[tuple[int, ...], dtype[float64]]")  [return-value]

I can reproduce this on both pyright and mypy, and this indeed used to be accepted on numpy==2.1.3. So could you open a new issue for this?

Wrong, because Mypy has supported property setters for ages. It only does not support different getter and setter types (I guess this is what jorenham refers to).

When I say that mypy doesn't support something, I mean that within the context of typing, because that's what we're talking about here, and that's what mypy is meant to check.
So for mypy to support something, it should do what is expected of it, and prevent type-unsafe situations like these:

class A:
    @property
    def thing(self) -> str:
        return "thing"
    @thing.setter
    def thing(self, thing: int, /) -> None:
        assert isinstance(thing, int)

a = A()
a.thing = "unsafe"  # mypy allows this

https://mypy-play.net/?mypy=latest&python=3.12&gist=c74a929091e366135306081ea0965098

Mypy explicitly allows this type-unsafe behaviour, but rejects e.g. a.thing = 3.14. And it also allows defining the thing setter with a different type, even though it is known to be unsupported. So this is why the only conclusion I can draw, is that mypy actively hides the fact that is doesn't (fully) support property setters.

because it suggests that people who contributed to Mypy are trying to mislead the Python community

I never mentioned individuals or a group of contributors. I'm purely talking about the software itself. So I don't see why you think that I suggesting this.

I thought it would go without saying, but I applaud anyone who contributes to open source, and actively try to motivate people to do so. Mypy is of course no exception to it.
All I'm trying to say is that, there are currently better alternatives to mypy.
And of course doesn't mean that I think that there's some conspiracy going on or something 🤷🏻‍♂️.

this issue is about incomplete transitions in shape typing (tuple[int] vs tuple[int, ...])

No. I've explained this several times now. This issue isn't about shape-typing. And shape-typing isn't yet supported in numpy, so you shouldn't expect it to work.

... and inheritance (e.g. floating vs float64) in Numpy

This issue isn't about inheritance. It's about how a = f(); a = g() was accepted in mypy when f: () -> int and g: () -> int, but isn't anymore now that we've narrowed f to () -> bool, while keeping g the same. This is the same situations as when you'd do a = True; a = 1, which also gets rejected by mypy, and only mypy.

Mypy reports the resulting inconsistencies more frequently than other type checkers, primarily because of --allow-redefinition is not enabled by default

So that's why --allow-redefinition has nothing do do with this.

The only other hint pointing to another concrete topic I could find is this one:

This is because mypy (unlike pyright) can't understand tuple[int, int, Unpack[tuple[int, ...]], or in words, "a shape type that is at least 1-dimensional".

I just checked this again with the latest mypy, and it seems to work now. So it appears my knowledge was outdated, and it's supported now 👌🏻.

I can reproduce this on both pyright and mypy, and this indeed used to be accepted on numpy==2.1.3. So could you open a new issue for this?

Yes, gladly,

So this is why the only conclusion I can draw, is that mypy actively hides the fact that is doesn't (fully) support property setters.

property is very special and requires a lot of special casing. Even Pyright, which, as far as I know, has better support for property, does not try to catch everything. For example, neither Mypy nor Pyright will warn you about this obvious bug:

from __future__ import annotations

def get(self: C) -> int:
    return 1

class C:
    g = property(get)

y = C().g + "x"

There was a general attempt to improve the situation for all type checker maintainers and Python users (by turning property into a descriptor), but that led to nothing (so far) and was even rejected by the Pyright team because of the anticipated work (which I find unfortunate but it's understandable, of course).

As hauntsaninja mentioned, Static type checking in Python is far from perfect and things are complicated, so oversimplified "by the way accusations" are usually not helpful.

But coming back to the main point of this issue (which you seem to have the authority to decide). From the perspective of all numpy users: What would be Mypy's preferred behaviour? I still guess they would want --allow-redefinition enabled by default. I do not know if the limitations of this option are for a good reason or just because the implementation is incomplete (but I could investigate). Also, my preliminary impression is that there should be a way to handle explicitly typed code differently:

x = True
x = 1  # maybe okay

y: bool = True
y = 1  # maybe not okay

What do you think?

There was a general attempt to improve the situation for all type checker maintainers and Python users (by turning property into a descriptor), but that led to nothing (so far) and was even rejected by the Pyright team because of the anticipated work (which I find unfortunate but it's understandable, of course).

Yes, I agree that this would be the best way. IMO special casing should only be used in, well, special cases. But @property being nothing but a descriptor, so it's not special (sorry @property; but we still love you regardless).

As hauntsaninja mentioned, Static type checking in Python is far from perfect and things are complicated, so oversimplified "by the way accusations" are usually not helpful.

I completely agree that static type checking is very difficult, especially in a gradual type system like the one Python uses.
That's why it matters so much that we use, and recommend, the best tool to help us with this. And the way to do this, is to lay out the pros and cons of the available tools.

And it's no secret that mypy has a lot more issues than pyright has. See https://github.com/erictraut/mypy_issues for example, for a thorough analysis of this. So what

So the way I see it, is that I'm not saying something new, and not accusing anyone of anything.
I'm just trying to help the numpy developers and users choose the type-checker that's best suited for the job.

Admittedly, I have my frustrations in dealing with mypy for five years or so, and we've seen here that I'm not the only one. But by no means did I intend to insult anyone; my frustrations are with the tool itself, and I having nothing but respect for those that try to make it better.

But coming back to the main point of this issue (which you seem to have the authority to decide). From the perspective of all numpy users: What would be Mypy's preferred behaviour? I still guess they would want --allow-redefinition enabled by default. I do not know if the limitations of this option are for a good reason or just because the implementation is incomplete (but I could investigate). Also, my preliminary impression is that there should be a way to handle explicitly typed code differently:
x = True
x = 1  # maybe okay

y: bool = True
y = 1  # maybe not okay
What do you think?

I agree that this would be the preferred behaviour in this case.

The x type should be determined by looking at how it's used globally. But as far as I can tell, mypy currently decides on the type after looking at x = True, but doesn't consider the following x = 1 after that for this.

The y: bool declaration is explicit, so there's no for the type-checker to infer y. That indeed makes it invalid to y = 1, since 1 isn't assignable to bool, just like isinstance(1, bool) is False.

Coincidentally, pyright already supports this: It accepts x = 1, and only rejects y = 1:

Type "Literal[1]" is not assignable to declared type "bool"
  "Literal[1]" is not assignable to "bool"  (reportAssignmentType)

https://basedpyright.com/?typeCheckingMode=all&code=B4AgvCAqBOCuCmBYAUKCBGEIDEIC2AhgJ4BG8IA9gNbEopEBcIJFFANuFHEskZ5jnzEyIAHYUALpRpEgA

But coming back to the main point of this issue (which you seem to have the authority to decide)

That's not really the way things work here. I'm just "the typing guy", and most of what I do in open source is related to static typing in Python. And just like any other NumPy maintainer, I try to do what's best for numpy and its users. In the end, we all decide together.

with allow-redefinition mypy will allow any assignments, as long as the old value has been used at least once:

a: bool = True
a = 1 # error

# but
b: bool = True
b = int(b)  # no error

where as pyright will never allow any redefinitions:

a: bool = True  # error: declaration is obscured
a: int = 1

pyright will allow a variable without an annotation to have assignments that can change the type, as it's not considered:

a = 1
a = ""  # no error

the minuta of the issues that mypy has with this can be demonstrated:

a: str = "a"
print(a)  # use `a` to allow mypy redefinition
a: int
reveal_type(a)  # mypy: int, runtime: "a"

also, mypy doesn't support reassignments across branching logic:

a = True
if bool():
    a = 1  # mypy: error expected bool
else:
    a = "a"  # mypy: error expected bool
reveal_type(a)  # mypy: bool, pyright: Literal[1, "a"]

additionally, I never intended to mislead anyone into thinking I was a mypy collaborator/maintainer. I didn't state as much but I can understand that it could be interpreted that way. I have spent many years developing (based)mypy every day, so I would consider my experience quite a bit more than "has contributed several ci improvements"

additionally-additionally, when I said "mypy is beyond saving", I wasn't trying to personally insult anyone, what I meant was that the issues in mypy are so deep in the implementation that it would require rewriting it to resolve them

I have great respect for open source, Python, mypy and all of its collaborators and contributors. I detest the vitriolic, spiteful and downright nasty not niceness of certain maintainers that exist in this world, so I'm sorry if anything I said was taken personally

Thanks for the clarifying words. I am very happy we left the stumblers behind us and seem to have started a closer collaboration. Numpy and Mypy working well together would be fantastic, and I look forward to Numpy providing shape-typing support for arrays. It's great that you are working on this!

the minuta of the issues that mypy has with this can be demonstrated:
a: bool = True
print(a)  # use `a` to allow mypy redefinition
a: int
reveal_type(a)  # mypy: int, pyright: Literal[True]

I screened through the above comments again and have the impression that users are most bothered that Mypy reports not-so-relevant (from the user perspective) array inconsistencies for unannotated assignments. We internally discussed ways to make Mypy less strict for such cases, even when different scopes are involved. I am (cautiously) optimistic we can make some progress on this soon.

Regarding the given example, I don't see how it relates to this issue. Could you please give an example? (Personally, I like narrowing but don't like and never use allow-redefinition, so I have never really thought it through.)

@jakevdp

To reiterate what @jakevdp said, and what I agree with: "un-annotated user code that executes correctly should not error when type-checked". I care about that case in the context of teaching, and the Scientific Python lecture notes.

@stefanv

@stefanv was avoiding mypy a viable solution here for you or not? I am not sure if Scientific Python lectures is a typical use-case or not.
But if it isn't a very common use-case and/or pyright (or another work around yet to be discovered!) is feasible than that would be good to know.

EDIT: I suppose this PR answers the question: scipy-lectures/scientific-python-lectures#813 (mostly works, but some other issues remaining).

scipy-lectures/scientific-python-lectures#813 (mostly works, but some other issues remaining).

Anything I can help with?

@stefanv

@stefanv was avoiding mypy a viable solution here for you or not? I am not sure if Scientific Python lectures is a typical use-case or not.
But if it isn't a very common use-case and/or pyright (or another work around yet to be discovered!) is feasible than that would be good to know.

Concerning mypy vs pyright I have had similar experience to @jorenham while adding type annotations in SymPy. There are many situations where I can add what seem like the right annotations and everything works with pyright but then I run mypy and it doesn't work and I need to go change the code or the annotations in awkward ways to satisfy mypy.

I think it is worth remembering though that there are different consumers of the type checkers and that these different type checkers are typically used by different people. Downstream of NumPy there are lots of libraries that will for the most part use mypy if they do use a type checker. In this role mypy is a command line development tool that is used somewhat like a linter so the kind of people using it are also the kind of people who might run ruff or flake8 etc on their libraries in CI.

On the other hand there are many "end users" who use NumPy directly but probably do not run any type checker explicitly and certainly would not use mypy. However pyright is a popular editor plugin and is e.g. part of the default Python plugin for vscode. The kind of person who is doing a typical end user calculation with NumPy probably does not add type hints to their code but will often benefit from things like autocompletion provided by pyright based on the type annotations/stubs that are provided by NumPy.

I think that the typical person reading @stefanv's lecture notes does not run mypy. At most they have pyright running in their editor. If thinking about the "end user" experience it is more important to focus on pyright being able to infer the types correctly for users of NumPy's main public API. Importantly you want those users to have no type hints in their own code but for pyright to be able to infer the types of all the numpy objects. Then users can do things like hover their mouse over x and see something like ndarray or type x. and see suggestions for possible array methods and so on. These features are actually very useful in an editor and now that I have pyright running in my editor (vim) I use these a lot and would recommend others to try them out.

For downstream libraries that use type checkers it will be problematic as NumPy adds annotations because it will be a long process during which the annotations are incomplete and continuously changing. Whenever new annotations or stubs are added it will "break" some downstream library's type checking. In some cases that might be a genuine regression and it would be reasonable for NumPy to change/revert the annotations. Otherwise though downstream libraries will just need to adapt and fix their own type annotations. There isn't any way to do the "gradual typing" thing without going through a long process of churn so either NumPy gives up on typing altogether or this downstream library typing churn has to be accepted.

There will be many downstream libraries who are using mypy in combination with the NumPy stubs regardless of whether NumPy "blesses" any other type checker. If, like in the OP issue, someone has problems that are specific to mypy then I think it is perfectly reasonable for NumPy to point out that the problem is specific to mypy and that mypy has configuration options and also that they can use a different checker if they want.

The problematic behaviour that mypy is showing here is just that mypy disallows rebinding the type like this:

data = '123'
data = int(data) # mypy errors but fine with pyright

Personally I don't want to use a checker that behaves like that and I much prefer pyright's handling of this case.

If someone else is happy with this style of type checking then they can feel free to use mypy but as NumPy (and other libraries) add more annotations it is going to break their mypy CI jobs. This is unavoidable since fewer things will be inferred as Any and mypy will inevitably consider more assignments to be incompatible. There is a simple fix to make mypy happy:

data = '123'
data_int = int(data)

Or they can configure mypy or they can use a different checker.

If NumPy wants to use a type checker on its own internal code then it can choose whichever one it wants. If pyright seems best then just use pyright. It doesn't matter what anyone else thinks or what checker they use. I haven't used basedpyright but having a lot of experience of mypy and pyright I would recommend pyright over mypy.

Several types in numpy 2.2+ are defined as typeAliases with no underl…

The new mypy 1.16.0 release introduces the optional --allow-redefinition-new flag. When running the original example in this post with it enabled (using numpy==2.2.0), mypy will no longer report it as an error.

Note that you'll also need to enable --local-partial-types for it to work.

https://mypy.readthedocs.io/en/stable/changelog.html#flexible-variable-redefinitions-experimental

Navigation Menu

Repository navigation

TYP: regression between 2.1.3 and 2.2.0 (mypy only) #27957

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

95 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TYP: regression between 2.1.3 and 2.2.0 (mypy only) #27957

Description

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

Activity

jakevdp commented on Dec 10, 2024

jakevdp commented on Dec 10, 2024

nabenabe0928 commented on Dec 10, 2024

adamjstewart commented on Dec 10, 2024

jorenham commented on Dec 10, 2024

jorenham commented on Dec 10, 2024

jorenham commented on Dec 10, 2024

jorenham commented on Dec 10, 2024

jakevdp commented on Dec 10, 2024

95 remaining items

tyralla commented on Dec 27, 2024

jorenham commented on Dec 27, 2024

tyralla commented on Dec 27, 2024

jorenham commented on Dec 27, 2024

jorenham commented on Dec 28, 2024

KotlinIsland commented on Dec 28, 2024

tyralla commented on Dec 29, 2024

stefanv commented on Dec 29, 2024

seberg commented on Dec 30, 2024

jorenham commented on Dec 31, 2024

oscarbenjamin commented on Jan 6, 2025

jorenham commented on May 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions