Repository navigation
Description
Describe the issue:
The commonly used @overload pattern with _ArrayLike{}_co parameter types can lead to unexpected type inferences if the return types are incompatible with one another other.
An example of this is the numpy.dtype constructor:
It correctly infers dtype(bool) and dtype(int) as dtype[np.bool] and dtype[signedinteger[Any]], respectively.
But in the case that either bool or int can be passed, e.g as dtype(int_co) with int_co: type[bool] | tuple[int], the resulting type is inferred as dtype[signedinteger[Any]], which is incompatible with dtype[bool].
The correct type in this case would be dtype[bool | signedinteger[Any]].
Reproduce the code example:
# overlapping_overload_issue.pyi
import numpy as np
x: type[bool]
y: type[int]
z: type[bool] | type[int]
reveal_type(np.dtype(x)) # OK: dtype[bool]
reveal_type(np.dtype(y)) # OK: dtype[signedinteger[Any]]
reveal_type(np.dtype(z)) # WRONG: dtype[signedinteger[Any]]But the last revealed type doesn't include bool, i.e. it should have been dtype[bool | signedinteger[Any]].
This isn't limited to only bool and int; all possible non-trivial unions of bool, int, float or complex have this problem, see https://typing.readthedocs.io/en/latest/spec/special-types.html#special-cases-for-float-and-complex
And this also isn't limited to dtype.__new__:
The same issue can be observed with the arithmetic operator methods (e.g. __add__) of numpy.ndarray, numpy.bool, and all subclasses of numpy.number, just to name a few.
Error message:
With Pyright / Pylance, enabling the reportOverlappingOverload setting will in such cases result in an error message (unless there's a # type: ignore comment), e.g. for numpy.dtype.__new__ Pylance reports:
Overload 1 for "__new__" overlaps overload 51 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 3 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 4 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 5 and returns an incompatible type
Python and NumPy Versions:
2.1.0.dev0+git20240724.3df4283
3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0]
Type-checker version and settings:
basedpyright 1.15.0 (pyright 1.1.373)
Additional typing packages.
typing-extensions 4.12.2
Activity
Jacob-Stevens-Haas commentedon Jul 25, 2024
Is the solution to use parametrized alias, not union aliases? e.g.
would be replaced with:
I think the dtype case (for overloads 2, 3, 4, 5) related but a different story though? It looks like
zmatches overload 2. Lettingzbe a typevar seems to work, and could collapse some of the overloads:jorenham commentedon Jul 25, 2024
@Jacob-Stevens-Haas I think you're on the right track. The
numpy._typing._Arrayalias is close to what you are suggesting, but also covers non-ndarrayarray-likes.But unfortunately, that'll only work for types that are expressible using NumPy scalars;
there's also the array-likes of builtin scalars, e.g. (nested) sequences of
bool,int,float, orcomplex.The big issue with the builtin scalars is that e.g.
def f(x: float)also accepts anintor abool, even thoughissubclass(int, float) is Falseat runtime 😓.So solving this issue mainly hinges on gfiguring out how to tell type-checkers that a function accepts e.g. a
floatthat isn't anint, which isn't possible.However, there's a workaround for this, which could look like
... but it'll probably be wise to add more methods, as to minimize the false-positive likelihood.
But this code can (at least in Pyright 1.1.373) distinguish between the exact builtin scalar types.
See this pyright playground link for a working example, including tests.
But either way, there's no easy solution - we can't just "fix" the
_ArrayLike(Int|Bool|Float|...)_coaliases or something like that 🤷🏻 .Edit
This doesn't work for mypy, and I haven't been able to figure out anything that does 😞.
jorenham commentedon Aug 10, 2024
To give a more concrete example; consider a simple function that transforms a Python builtin scalar into a
numpy.genericone:For the sake of simplicity, let's limit ourselves to just the numeric scalar types.
Stubs with incompatible overlapping overloads
At first glance, it doesn't look like there's anything wrong with this.
And some simple tests would show that it's working as expected, e.g.:
reveal_type(scalar(True))givesnp.bool✅reveal_type(scalar(42))givesnp.int_✅reveal_type(scalar(1 / 137))givesnp.float64✅reveal_type(scalar((-1) ** .5))givesnp.complex128✅But directly passing a constant to a function isn't something that's done very often.
In practice, we almost always pass a variable, which aren't limited to be an instance of a specific class:
The type-checker outputs
But that's clearly wrong:
x_npcould also be anp.bool, anp.int_, or anp.float64!Why is it wrong?
The source of this issue lies in what the typing spec calls "Special cases for
floatandcomplex":So in other words: type-checkers must assume that
issubclass(int, float) and issubclass(float, complex), even though this isn't the case at runtime, nor are they type-compatible (i.e. in the way that protocols can be).This makes it impossible to distinguish between e.g. "float | int" and "float", as they are treated the same.
Correct stubs
A less user-unfriendly approach would be to use
| Any:Note that
Anywill make this type-unsafe. But I'd argue that in this case it's the lesser of two evils.