Skip to content

TYP: np.ndarray.tolist return type seems broken in numpy 2.2.0 #27944

Closed
1 of 1 issue completed
Closed

Description

@neutrinoceros

Describe the issue:

While upgrading CMasher's CI to numpy 2.2.0, I got a new typechecking error that seems spurious to me. No error is raised with numpy 2.1.3

Reproduce the code example:

import numpy as np

def to_rgb(in_) -> tuple[float, float, float]:
    # minimal mock for matplotlib.colors.to_rgb
    return (1. ,2., 3.)

def foo(name: str, data: list[tuple[float, float, float]]) -> None:
    data_arr = np.array(data)

    if data_arr.dtype.kind == "S":
        cm_data = [str(x) for x in data_arr]
        colorlist = [to_rgb(_) for _ in cm_data]
    else:
        data_arr = np.atleast_2d(data_arr)
        colorlist = data_arr.tolist()

Error message:

cmasher.py:15: error: Incompatible types in assignment (expression has type "str | list[str] | list[list[str]] | list[list[list[Any]]]", variable has type "list[tuple[float, float, float]]")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Python and NumPy Versions:

2.2.0
3.13.1 (main, Dec  4 2024, 16:25:14) [Clang 16.0.0 (clang-1600.0.26.4)]

Type-checker version and settings:

mypy==1.13.0

no custom settings

Additional typing packages.

Included with mypy, but not used here (as far as I'm aware)

typing-extensions==4.12.2

Activity

  1. changed the title [-]TYP: `np.ndarray.tolist` return type seems broken in numpy 2.2[/-] [+]TYP: `np.ndarray.tolist` return type seems broken in numpy 2.2.0[/+] on Dec 9, 2024
  2. added this to the 2.2.1 release milestone on Dec 9, 2024
  3. jorenham commented on Dec 10, 2024

    @jorenham
    Member

    data_arr = np.array(data) is inferred as NDArray[Any], which is technically correct, but far from ideal.

    To give a bit more context, this is how ndarray.tolist is annotated:

        @overload
        def tolist(self: _HasShapeAndSupportsItem[tuple[()], _T], /) -> _T: ...
        @overload
        def tolist(self: _HasShapeAndSupportsItem[tuple[int], _T], /) -> list[_T]: ...
        @overload
        def tolist(self: _HasShapeAndSupportsItem[tuple[int, int], _T], /) -> list[list[_T]]: ...
        @overload
        def tolist(self: _HasShapeAndSupportsItem[tuple[int, int, int], _T], /) -> list[list[list[_T]]]: ...
        @overload
        def tolist(self: _HasShapeAndSupportsItem[Any, _T], /) -> _T | list[_T] | list[list[_T]] | list[list[list[Any]]]: ...

    So without a know shape-type, the last overload should be used. And without a known dtype (bound to _T here), the correct return type should be

    Any | list[Any] | list[list[Any]] | list[list[list[Any]]]

    And pyright actually infers this correctly, and accepts your example, as it should.
    But you're using mypy, and mypy is very broken, which this case is an example of, and causes it to infer _T as str. That makes no sense at all, which can only mean that it's one of the >1200 mypy bugs that aren't in pyright.

    So I'm afraid that I can't help you here, and that this is a bug within mypy.

  4. added
    57 - Close?Issues which may be closable unless discussion continued
    on Dec 10, 2024
  5. neutrinoceros commented on Dec 10, 2024

    @neutrinoceros
    ContributorAuthor

    Understood,. I may try to migrate to pyright, but for now I'll just put a lid on it. Thanks for your swift reply !

  6. removed
    57 - Close?Issues which may be closable unless discussion continued
    on Dec 10, 2024
  7. ishaan-mehta commented on Apr 2, 2025

    @ishaan-mehta

    Is anyone else seeing this with pyright too? I am.

  8. jorenham commented on Apr 2, 2025

    @jorenham
    Member

    Is anyone else seeing this with pyright too? I am.

    Could you share the exact code?

  9. ishaan-mehta commented on Apr 3, 2025

    @ishaan-mehta

    Is anyone else seeing this with pyright too? I am.

    Could you share the exact code?

    Sure, here's an MRE:

    import pandas as pd
    
    df = pd.DataFrame(
        ["2024-01-01", "2022-11-12", "2019-08-27", "2024-05-11", "2022-02-28"],
        columns=["date"],
    )
    
    df["date"] = pd.to_datetime(df["date"])
    unique_years = df["date"].dt.year.unique()
    print(f"unique_years type: {type(unique_years)}")
    unique_years_list: list[int] = unique_years.tolist()
    print(f"unique_years_list type: {type(unique_years_list)}")
    print(f"unique_years_list[0] type: {type(unique_years_list[0])}")

    This outputs:

    unique_years type: <class 'numpy.ndarray'>
    unique_years_list type: <class 'list'>
    unique_years_list[0] type: <class 'int'>
    

    But reveal_type(unique_years.tolist()) with Pylance says that it is a str. Same with raw pyright instead of Pylance:

    error: Type "str" is not assignable to declared type "list[int]"
        "str" is not assignable to "list[int]" (reportAssignmentType)
    

    Versions:

    • numpy==2.2.4
    • pyright==1.1.398
    • Pylance: 2025.4.1
  10. removed this from the 2.2.1 release milestone on Apr 4, 2025
  11. jorenham commented on Apr 4, 2025

    @jorenham
    Member

    In your example @ishaan-mehta, unique_years is inferred as ndarray[Unknown, Unknown]. This is effectively the same as ndarray[Any, Any]. In this case, both pyright and mypy will indeed infer str as the return type of .tolist(). Here's a minimal .pyi example that shows this:

    from typing import Any, reveal_type
    import numpy as np
    
    a: np.ndarray[Any, Any]
    reveal_type(a.tolist())

    mypy: Revealed type is "builtins.str"
    pyright: Type of "a.tolist()" is "str"


    The original post was about NDArray[Any], which reduces to ndarray[Any, dtype[Any]]. It's a subtle difference, but it seems to leads to different behavior in pyright. Mypy's behaviour is the same on both situations. So technically, this is a different issue.

    But that being said, solving one issue will probably solve the other. And to be honest, I was too quick in dismissing this as a "mypy-only" issue. So I'm reopening this, and I'll try my best to find a solution in the coming days.

  12. self-assigned this
    on Apr 4, 2025
  13. added this to the 2.2.5 release milestone on Apr 4, 2025
  14. ishaan-mehta commented on Apr 4, 2025

    @ishaan-mehta

    Thanks for isolating the issue and looking into it @jorenham! (And I appreciate you accommodating the semi-related topic in the same issue.)

  15. jorenham commented on Apr 20, 2025

    @jorenham
    Member

    Thanks @neutrinoceros and @ishaan-mehta for reporting this! The fix is included in the latest 2.2.5 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions