Skip to content

TYP: Return type of np.dtype is Any? #28034

New issue
New issue
Closed
Closed
TYP: Return type of np.dtype is Any?#28034
Assignees
jorenham
Labels
41 - Static typing

Description

@jhamman
jhamman
opened on Dec 18, 2024

Describe the issue:

Numpy 2.2 introduced new return type semantics for np.dtype. Mypy is identifying the return type of np.dtype(Any) as Any. Example and mypy logs below 👇

Reproduce the code example:

from typing import Any
import numpy as np

def parse_dtype(dtype: Any) -> np.dtype[Any]:
    reveal_type(np.dtype(dtype))
    return np.dtype(dtype)

Error message:

❯ mypy ./test.py
/Users/jhamman/scratch/test.py:5: note: Revealed type is "Any"
/Users/jhamman/scratch/test.py:6: error: Returning Any from function declared to return "dtype[Any]"  [no-any-return]
Found 1 error in 1 file (checked 1 source file)

Python and NumPy Versions:

2.2.0
3.11.6 | packaged by conda-forge | (main, Oct 3 2023, 10:37:07) [Clang 15.0.7 ]

Runtime Environment:

[{'numpy_version': '2.2.0',
  'python': '3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:37:07) '
            '[Clang 15.0.7 ]',
  'uname': uname_result(system='Darwin', node='Joes-MacBook-Air-766.local', release='24.2.0', version='Darwin Kernel Version 24.2.0: Fri Dec  6 18:51:28 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T8112', machine='arm64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP'],
                      'not_found': ['ASIMDFHM']}}]

Context for the issue:

Xrefs:

Activity

  1. jhamman
    added
    00 - Bug
    on Dec 18, 2024

  2. jakevdp commented on Dec 18, 2024

    @jakevdp
    jakevdp
    on Dec 18, 2024
    Contributor

    It looks like there's no overload that matches np.dtype.__new__(cls, dtype: Any):

    numpy/numpy/__init__.pyi

    Line 1167 in 5a2da5e

    def __new__(

  3. jhamman commented on Dec 18, 2024

    @jhamman
    jhamman
    on Dec 18, 2024
    Author

    Thanks @jakevdp! Nice to bump into you again :)

    This did seem to work fine in Numpy 2.1 so smells like I regression to me.

    pip install -U numpy==2.1
mypy ~/Downloads/test.py
/Users/jhamman/scratch/test.py:5: note: Revealed type is "numpy.dtype[Any]"
Success: no issues found in 1 source file
  4. charris
    added
    41 - Static typing
    on Dec 18, 2024
  5. jakevdp
    assigned
    jorenham
    on Dec 18, 2024

  6. jakevdp commented on Dec 18, 2024

    @jakevdp
    jakevdp
    on Dec 18, 2024
    Contributor

    The problematic change is likely #27139

  7. jorenham
    changed the title [-]Return type of np.dtype is Any?[/-] [+]TYP: Return type of `np.dtype` is `Any`?[/+] on Dec 19, 2024

  8. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    What did you expect would be the outcome?

  9. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    It looks like there's no overload that matches np.dtype.__new__(cls, dtype: Any):

    That's intentional, because otherwise something invalid like np.dtype(NotImplemented) would result in a false negative.

  10. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    repro:

    # [...]/issue_28034.pyi

from typing import Any, reveal_type

import numpy as np

no_idea_what_this_is: Any
reveal_type(np.dtype(no_idea_what_this_is))

    relevant pyproject.toml config:

    dependencies = [
    "numpy>=2.2.0",
    "mypy[faster-cache]>=1.13.0",
    "pyright>=1.1.391",
]

[tool.mypy]
python_version = "3.12"
strict = true
disable_bytearray_promotion = true
disable_memoryview_promotion = true

[tool.pyright]
pythonPlatform = "All"
pythonVersion = "3.12"
typeCheckingMode = "strict"

enableTypeIgnoreComments = false
reportUnnecessaryTypeIgnoreComment = true
reportAny = false
reportUnusedCallResult = false

    output of uv run mypy [...]/issue_28034.pyi:

    [...]/issue_28034.pyi:6: note: Revealed type is "Any"
Success: no issues found in 1 source file

    output of uv run pyright [...]/issue_28034.pyi:

    [...]/issue_28034.pyi
 [...]/issue_28034.pyi:6:13 - information: Type of "np.dtype(no_idea_what_this_is)" is "dtype[Unknown]"
0 errors, 0 warnings, 1 information

    So this appears to be yet another mypy bug 😑. And in this case I can't think of any way to work around it without breaking other things.

    FYI, this is another example of why you shouldn't use Any if you can avoid it, and one of the (many) reasons for why tools like flake8-pyi and basedpyright report usages of Any as an error.

  11. jorenham
    added
    57 - Close?Issues which may be closable unless discussion continued
    and removed
    00 - Bug
    on Dec 19, 2024

  12. jakevdp commented on Dec 19, 2024

    @jakevdp
    jakevdp
    on Dec 19, 2024
    Contributor

    Is there a valid annotation that means "any dtype"?

  13. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    Is there a valid annotation that means "any dtype"?

    np.dtype[np.generic]

  14. jakevdp commented on Dec 19, 2024

    @jakevdp
    jakevdp
    on Dec 19, 2024
    Contributor

    OK, so then am I correct in understanding that the recommended approach here would be something like this?

    def parse_dtype(dtype: Any) -> np.dtype[np.generic]:
    return np.dtype(dtype)  # type: ignore

  15. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    OK, so then am I correct in understanding that the recommended approach here would be something like this?

    def parse_dtype(dtype: Any) -> np.dtype[np.generic]:
    return np.dtype(dtype)  # type: ignore

    That works! Or even better, you could remove # type: ignore, as this already is valid :)

  16. jakevdp commented on Dec 19, 2024

    @jakevdp
    jakevdp
    on Dec 19, 2024
    Contributor

    Ah thanks – from the discussion above, I thought that the np.dtype(dtype) line would be problematic because dtype is Any and np.dtype does not have any overloads for that. But perhaps I misinterpreted things.

  17. jorenham commented on Dec 19, 2024

    @jorenham
    jorenham
    on Dec 19, 2024
    Member

    Ah thanks – from the discussion above, I thought that the np.dtype(dtype) line would be problematic because dtype is Any and np.dtype does not have any overloads for that. But perhaps I misinterpreted things.

    Yea that's understandable. Any is just very annoying to deal with, and doesn't behave like any other type (pun intended), as everything can be assigned to Any, and Any can be assigned to everything.

  18. jhamman commented on Dec 20, 2024

    @jhamman
    jhamman
    on Dec 20, 2024
    Author

    That works! Or even better, you could remove # type: ignore, as this already is valid :)

    I don't think this is true.

    from typing import Any
import numpy as np

def parse_dtype(dtype: Any) -> np.dtype[np.generic]:
    reveal_type(np.dtype(dtype))
    return np.dtype(dtype)
    /Users/jhamman/scratch/test.py:5: note: Revealed type is "Any"
/Users/jhamman/scratch/test.py:6: error: Returning Any from function declared to return "dtype[generic[Any]]"  [no-any-return]

  19. jorenham commented on Dec 20, 2024

    @jorenham
    jorenham
    on Dec 20, 2024
    Member

    That works! Or even better, you could remove # type: ignore, as this already is valid :)

    I don't think this is true.

    from typing import Any
import numpy as np

def parse_dtype(dtype: Any) -> np.dtype[np.generic]:
    reveal_type(np.dtype(dtype))
    return np.dtype(dtype)
    /Users/jhamman/scratch/test.py:5: note: Revealed type is "Any"
/Users/jhamman/scratch/test.py:6: error: Returning Any from function declared to return "dtype[generic[Any]]"  [no-any-return]

    That check isn't enabled by default if I remember correctly. It's called disallow_any_explicit or disallow_any_expr I believe

  20. jorenham commented on Dec 20, 2024

    @jorenham
    jorenham
    on Dec 20, 2024
    Member

    Either way, even with that check enabled, if you replace the dtype: Any with dtype: npt.DTypeLike it'll also work:

    import numpy as np
import numpy.typing as npt

def parse_dtype(dtype: npt.DTypeLike) -> np.dtype[np.generic]:
    return np.dtype(dtype)
  21. jorenham
    closed this as completedon Dec 25, 2024
  22. jorenham
    removed
    57 - Close?Issues which may be closable unless discussion continued
    on Dec 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

41 - Static typing

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions