Skip to content

TYP: Overlapping overloads with incompatible return types #27032

Open

Description

@jorenham

Describe the issue:

The commonly used @overload pattern with _ArrayLike{}_co parameter types can lead to unexpected type inferences if the return types are incompatible with one another other.

An example of this is the numpy.dtype constructor:
It correctly infers dtype(bool) and dtype(int) as dtype[np.bool] and dtype[signedinteger[Any]], respectively.
But in the case that either bool or int can be passed, e.g as dtype(int_co) with int_co: type[bool] | tuple[int], the resulting type is inferred as dtype[signedinteger[Any]], which is incompatible with dtype[bool].
The correct type in this case would be dtype[bool | signedinteger[Any]].

Reproduce the code example:

# overlapping_overload_issue.pyi

import numpy as np

x: type[bool]
y: type[int]
z: type[bool] | type[int]

reveal_type(np.dtype(x))  # OK: dtype[bool]
reveal_type(np.dtype(y))  # OK: dtype[signedinteger[Any]]
reveal_type(np.dtype(z))  # WRONG: dtype[signedinteger[Any]]

But the last revealed type doesn't include bool, i.e. it should have been dtype[bool | signedinteger[Any]].

This isn't limited to only bool and int; all possible non-trivial unions of bool, int, float or complex have this problem, see https://typing.readthedocs.io/en/latest/spec/special-types.html#special-cases-for-float-and-complex

And this also isn't limited to dtype.__new__:
The same issue can be observed with the arithmetic operator methods (e.g. __add__) of numpy.ndarray, numpy.bool, and all subclasses of numpy.number, just to name a few.

Error message:

With Pyright / Pylance, enabling the reportOverlappingOverload setting will in such cases result in an error message (unless there's a # type: ignore comment), e.g. for numpy.dtype.__new__ Pylance reports:

Overload 1 for "__new__" overlaps overload 51 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 3 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 4 and returns an incompatible type
Overload 2 for "__new__" overlaps overload 5 and returns an incompatible type

Python and NumPy Versions:

2.1.0.dev0+git20240724.3df4283
3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0]

Type-checker version and settings:

basedpyright 1.15.0 (pyright 1.1.373)

Additional typing packages.

typing-extensions 4.12.2

Activity

  1. Jacob-Stevens-Haas commented on Jul 25, 2024

    @Jacob-Stevens-Haas
    Contributor

    Is the solution to use parametrized alias, not union aliases? e.g.

    _ArrayInt_co: TypeAlias = NDArray[np.bool | integer[Any]]
    

    would be replaced with:

    from typing import TypeVar
    Int_co = TypeVar("Int_co", integer[Any], np.bool)
    _ArrayInt_co: TypeAlias = NDArray[Int_co]

    I think the dtype case (for overloads 2, 3, 4, 5) related but a different story though? It looks like z matches overload 2. Letting z be a typevar seems to work, and could collapse some of the overloads:

    from typing import TypeVar
    z = TypeVar("z", bound=type[complex])
    def identity(typ: z) -> z: ...
    
    reveal_type(np.dtype(identity(bool))) # OK: dtype[bool]
    reveal_type(np.dtype(identity(int)))# OK: dtype[signedinteger[Any]]
    reveal_type(np.dtype(identity(float)))  # OK: dtype[floating[_64Bit]]
    reveal_type(np.dtype(identity(complex)))  # OK: dtype[complexfloating[_64Bit, _64Bit]]
  2. jorenham commented on Jul 25, 2024

    @jorenham
    MemberAuthor

    @Jacob-Stevens-Haas I think you're on the right track. The numpy._typing._Array alias is close to what you are suggesting, but also covers non-ndarray array-likes.
    But unfortunately, that'll only work for types that are expressible using NumPy scalars;
    there's also the array-likes of builtin scalars, e.g. (nested) sequences of bool, int, float, or complex.

    The big issue with the builtin scalars is that e.g. def f(x: float) also accepts an int or a bool, even though issubclass(int, float) is False at runtime 😓.
    So solving this issue mainly hinges on gfiguring out how to tell type-checkers that a function accepts e.g. a float that isn't an int, which isn't possible.

    However, there's a workaround for this, which could look like

    from typing import Protocol, final
    
    @final
    class OnlyBool(Protocol):
        def __new__(cls, o: int, /) -> bool: ...
    
    @final
    class OnlyInt(Protocol):
        def __new__(cls, o: str, /, base: int) -> int: ...
    
    @final
    class OnlyFloat(Protocol):
        def __new__(cls, o: str, /) -> float: ...
        def hex(self) -> str: ...
    
    @final
    class OnlyComplex(Protocol):
        def __new__(cls, real: float, imag: float) -> complex: ...

    ... but it'll probably be wise to add more methods, as to minimize the false-positive likelihood.
    But this code can (at least in Pyright 1.1.373) distinguish between the exact builtin scalar types.
    See this pyright playground link for a working example, including tests.

    But either way, there's no easy solution - we can't just "fix" the _ArrayLike(Int|Bool|Float|...)_co aliases or something like that 🤷🏻 .


    Edit

    This doesn't work for mypy, and I haven't been able to figure out anything that does 😞.

  3. jorenham commented on Aug 10, 2024

    @jorenham
    MemberAuthor

    To give a more concrete example; consider a simple function that transforms a Python builtin scalar into a numpy.generic one:

    import numpy as np
    
    def scalar(x, /):
        if isinstance(x, np.generic | np.ndarray):
            raise TypeError(type(x))
        return np.array(x)[()]

    For the sake of simplicity, let's limit ourselves to just the numeric scalar types.

    Stubs with incompatible overlapping overloads

    from typing import overload
    import numpy as np
    
    @overload
    def scalar(x: bool, /) -> np.bool: ...  # ok
    @overload
    def scalar(x: int, /) -> np.int_: ...  # wrong
    @overload
    def scalar(x: float, /) -> np.float64: ...  # wrong
    @overload
    def scalar(x: complex, /) -> np.complex128: ...  # wrong

    At first glance, it doesn't look like there's anything wrong with this.
    And some simple tests would show that it's working as expected, e.g.:

    • reveal_type(scalar(True)) gives np.bool ✅
    • reveal_type(scalar(42)) gives np.int_ ✅
    • reveal_type(scalar(1 / 137)) gives np.float64 ✅
    • reveal_type(scalar((-1) ** .5)) gives np.complex128 ✅

    But directly passing a constant to a function isn't something that's done very often.
    In practice, we almost always pass a variable, which aren't limited to be an instance of a specific class:

    import random
    
    x_py = random.choice([bool, int, float, complex])()
    reveal_type(x_py)
    x_np = scalar(x_py)
    reveal_type(x_np)

    The type-checker outputs

    Type of "x_py" is "bool | complex | float | int"
    Type of "x_np" is "complex128"
    

    But that's clearly wrong: x_np could also be a np.bool, a np.int_, or a np.float64!

    Why is it wrong?

    The source of this issue lies in what the typing spec calls "Special cases for float and complex":

    Python’s numeric types complex, float and int are not subtypes of each other, but to support common use cases, the type system contains a straightforward shortcut: when an argument is annotated as having type float, an argument of type int is acceptable; similar, for an argument annotated as having type complex, arguments of type float or int are acceptable.

    So in other words: type-checkers must assume that issubclass(int, float) and issubclass(float, complex), even though this isn't the case at runtime, nor are they type-compatible (i.e. in the way that protocols can be).

    This makes it impossible to distinguish between e.g. "float | int" and "float", as they are treated the same.

    Python-typing float|int meme

    Correct stubs

    from typing import overload
    import numpy as np
    
    @overload
    def scalar(x: bool, /) -> np.bool: ...  # ok :)
    @overload
    def scalar(x: int, /) -> np.int_ | np.bool: ...  # ok :|
    @overload
    def scalar(x: float, /) -> np.float64 | np.int_ | np.bool: ...  # ok :\
    @overload
    def scalar(x: complex, /) -> np.complex128 | np.float64 | np.int_ | np.bool: ...  # ok :(

    A less user-unfriendly approach would be to use | Any:

    @overload
    def scalar(x: bool, /) -> np.bool: ...
    @overload
    def scalar(x: int, /) -> np.int_ | Any: ...
    @overload
    def scalar(x: float, /) -> np.float64 | Any: ...
    @overload
    def scalar(x: complex, /) -> np.complex128 | Any: ...

    Note that Any will make this type-unsafe. But I'd argue that in this case it's the lesser of two evils.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions