Skip to content

Type hinting / annotation (PEP 484) for ndarray, dtype, and ufunc #7370

Closed

Description

@InonS

Feature request: Organic support for PEP 484 with Numpy data structures.

Has anyone implemented type hinting for the specific numpy.ndarray class?

Right now, I'm using typing.Any, but it would be nice to have something more specific.

For instance if the numpy people added a type alias for their array_like object class. Better yet, implement support at the dtype level, so that other objects would be supported, as well as ufunc.

original SO question

Activity

  1. njsmith commented on Mar 2, 2016

    @njsmith
    Member

    I don't think anyone's thought about it. Perhaps you would like to? :-)

    I'm also going to suggest that if you want to followup on this that we close the gh issue and move the discussion to the mailing list, since it's better suited to open-ended design discussions.

  2. InonS commented on Mar 2, 2016

    @InonS
    Author

    After getting this answer on SO, I've decided to close the issue.

  3. njsmith commented on Mar 3, 2016

    @njsmith
    Member

    To be clear, we don't actually have any objection to supporting cool new python features or anything (rather the opposite); it's just that we're a volunteer run project without many resources, so stuff only happens if someone who's interested steps up to do it.

    The mailing list is usually the best place if you're trying to start working on something or hoping to recruit some other interested folks to help.

  4. InonS commented on Mar 3, 2016

    @InonS
    Author

    Thanks, @njsmith. I decided to start here because of the more orderly issue-tracking, as opposed to an unstructured mailing list (I was looking for a 'feature request' tag, among other features...)

    Since the guy who answered me on SO got back to me with a viable solution, I decided to leave the matter.
    Maybe the Numpy documentation should be updated to include his answer (please make sure to give him credit if you do).

    Thanks, again!

  5. JulesGM commented on Apr 27, 2017

    @JulesGM

    hello guys! I was just kindly wondering if there had been any progress on this issue. Thanks.

  6. eric-wieser commented on Apr 27, 2017

    @eric-wieser
    Member

    There is some discussion about it on the mailing list here.

  7. shoyer commented on May 8, 2017

    @shoyer
    Member

    I'm reopening this issue for those who are interested in discussing it further.

    I think this would certainly be desirable for NumPy, but there are indeed a few tricky aspects of the NumPy API for typing to sort through, such as how NumPy currently accepts arbitrary objects in the np.array constructor (though we want to clean this up, see #5353).

  8. reopened this on May 8, 2017
  9. engnadeau commented on Jun 16, 2017

    @engnadeau

    Some good work is being done here: https://github.com/machinalis/mypy-data

    There's discussion about whether to push the work upstream to numpy or typeshed: machinalis/mypy-data#16

  10. njsmith commented on Jun 16, 2017

    @njsmith
    Member
  11. henryJack commented on Sep 1, 2017

    @henryJack

    This really would be a great addition to NumPy. What would be the next steps to push this up to typeshed or NumPy? Even an incomplete stub would be useful and I'm happy to help with a bit of direction?

  12. shoyer commented on Sep 1, 2017

    @shoyer
    Member

    @henryJack The best place to start would probably be tooling: figure out how we can integrate basic type annotations into the NumPy repository (and ideally test them) in a way that works with mypy and supports adding them incrementally.

    Then, start with extremely minimal annotations and we can go from there. In particular, I would skip dtype annotations for now since we don't have a good way to specify them (i.e., only do ndarray, not ndarray[int]).

    If it's helpful, I have an alternative version of annotations that I've written for use at Google and could open source. But we have our own unique build system and do type checking with pytype, so there would likely be quirks porting it to upstream.

  13. jwkvam commented on Sep 1, 2017

    @jwkvam

    I suppose the only way to test annotations to actually run mypy on sample code snippets and check the output?

    Would it be better to have the annotations integrated with the code or as separate stubs?

    I suppose we should also learn from dropbox and pandas that we should start with the leaves of the codebase versus core data structures?

  14. 63 remaining items

  15. mrahtz commented on May 4, 2021

    @mrahtz

    Another quick update: Pradeep Kumar Srinivasan and I will be giving a talk on the approach we've been experimenting with over the past 6 months at the PyCon 2021 Typing Summit next week, Catching Tensor Shape Errors Using the Type Checker: https://us.pycon.org/2021/summits/typing/ We'll be discussing how it works, what it looks like in practice, and a few of its current limitations. Hope to see you there!

  16. BvB93 commented on May 7, 2021

    @BvB93
    Member

    Hi all,

    Some time ago, back in #17719, dtype support was introduced for np.ndarray (including a placeholder slot for shapes). As a followup, earlier today a new PR was submitted (#18935) adding a runtime-subscriptable alias for np.ndarray[Any, np.dtype[~Scalar]], the latter providing a convenient (and compact) alias for annotating arrays with a given dtype and unspecified shape:

    >>> import numpy as np
    >>> import numpy.typing as npt
    
    >>> print(npt.NDArray)
    numpy.ndarray[typing.Any, numpy.dtype[~ScalarType]]
    
    >>> print(npt.NDArray[np.float64])
    numpy.ndarray[typing.Any, numpy.dtype[numpy.float64]]
    
    >>> NDArrayInt = npt.NDArray[np.int_]
    >>> a: NDArrayInt = np.arange(10)
    
    >>> def func(a: npt.ArrayLike) -> npt.NDArray[Any]:
    ...     return np.array(a)
  17. NeilGirdhar commented on Jul 7, 2021

    @NeilGirdhar
    Contributor

    @BvB93 This is awesome. I've been using your change since numpy 1.21 came out. Are you planning on adding runtime subscripting like np.ndarray[Any, Any], np.integer[Any], np.floating[Any], np.dtype[Any]? MyPy complains if I don't put Any since I have the flag for no implicit generics, and I end up having to work around numpy's missing __class_getitem__.

  18. rggjan commented on Jul 7, 2021

    @rggjan

    @NeilGirdhar excellent idea. As a workaround in the meantime, you could try using from __future__ import annotations, which should make it work at runtime (as usually annotations are not evaluated at runtime anymore).

  19. NeilGirdhar commented on Jul 7, 2021

    @NeilGirdhar
    Contributor

    @rggjan Yup, thanks! That works in many cases except when you want to do set aliases like this. Also, pylint rightly complains that these type objects are not subscriptable.

  20. gsakkis commented on Aug 28, 2021

    @gsakkis

    Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).

  21. Jasha10 commented on Aug 28, 2021

    @Jasha10

    Is it possible to annotate structured arrays and if so how?

    I believe that as of Numpy 1.21 it is not yet possible.

  22. BvB93 commented on Aug 30, 2021

    @BvB93
    Member

    Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).

    Unfortunately not, and I very much doubt that list-of-tuples-syntax will ever be something that mypy will understand (not without some serious plugin magic, at least).

    As for structured arrays in general, there two main challenges here:

    1. How to type the necasary structure into the np.void dtype.

      Ideally we'd make it generic w.r.t. to something like TypedDict, so field dtypes can be assigned to each key (i.e. field names). Making np.void generic is however complicated by its flexibility, as it can be used for representing opaque bytes sequences ("V10"), dtypes with a field size ((np.float64, 8)) and structured dtypes with a set of keys and matching field dtypes ([("a", np.float64)] or [("a", np.float64, 8)]).

      Only the last category can reasonably be express via a TypedDict, so this raises the question what do about the other two? Make it generic w.r.t. a trinary Union? Treat all three categories as type-check-only subclasses? This is very much an open question.

    2. How to let ndarray.__getitem__ and __setitem__ access the named fields.

      Letting ndarray access and use the fields encoded within the dtype will be a challenge of its own. Namelly, the only two types that currently deal with arbitrary named fields (NamedTuple and TypedDict) have, in my experience, proven to be less than cooperative when dispatching with the help of protocols. This is very much a mypy bug, but in this context it's probably going to be a detrimental one.

      For example:

      from typing import TypedDict, Protocol, TypeVar, TYPE_CHECKING
      
      KT = TypeVar("KT")
      VT = TypeVar("VT")
      KT_contra = TypeVar("KT_contra", contravariant=True)
      VT_co = TypeVar("VT_co", covariant=True)
      
      class SupportsGetItem(Protocol[KT_contra, VT_co]):
          def __getitem__(self, key: KT_contra, /) -> VT_co: ...
      
      class TestDict(TypedDict):
          a: int
          b: str
      
      def getitem(dct: SupportsGetItem[KT, VT], key: KT) -> VT: ...
      
      test_dict: TestDict
      if TYPE_CHECKING:
          reveal_type(getitem(test_dict, "a"))  # Revealed type is "builtins.object*"
          reveal_type(getitem(test_dict, "b"))  # Revealed type is "builtins.object*"
          reveal_type(getitem(test_dict, "c"))  # Revealed type is "builtins.object*"
  23. BvB93 commented on Sep 16, 2021

    @BvB93
    Member

    @NeilGirdhar there is currently a PR up for making number, dtype and ndarray runtime-subscriptable (#19879),
    though note that this functionality does have a hard dependency on python >= 3.9.

    The hope is to wrap things up before the next 1.22 release.

  24. Jasha10 commented on Nov 21, 2021

    @Jasha10

    That works in many cases except when you want to do set aliases like this.

    @NeilGirdhar The workaround I've been using in that case is to put quotes around the subscripted numpy type:

        RealArray = npt.NDArray["np.floating[Any]"]
    
  25. Why-not-now commented on Dec 6, 2022

    @Why-not-now

    Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).

    Unfortunately not, and I very much doubt that list-of-tuples-syntax will ever be something that mypy will understand (not without some serious plugin magic, at least).

    As for structured arrays in general, there two main challenges here:

    1. How to type the necasary structure into the `np.void` dtype.
       Ideally we'd make it generic w.r.t. to something like `TypedDict`, so field dtypes can be assigned to each key (_i.e._ field names). Making `np.void` generic is however complicated by its flexibility, as it can be used for representing opaque bytes sequences (`"V10"`), dtypes with a field size (`(np.float64, 8)`) and structured dtypes with a set of keys and matching field dtypes (`[("a", np.float64)]` or `[("a", np.float64, 8)]`).
       Only the last category can reasonably be express via a `TypedDict`, so this raises the question what do about the other two? Make it generic w.r.t. a trinary `Union`? Treat all three categories as type-check-only subclasses? This is very much an open question.
    
    2. How to let `ndarray.__getitem__` and `__setitem__` access the named fields.
       Letting `ndarray` access and use the fields encoded within the dtype will be a challenge of its own. Namelly, the only two types that currently deal with arbitrary named fields (`NamedTuple` and `TypedDict`) have, in my experience, proven to be less than cooperative when dispatching with the help of protocols. This is very much a mypy bug, but in this context it's probably going to be a detrimental one.
       For example:
       ```python
       from typing import TypedDict, Protocol, TypeVar, TYPE_CHECKING
       
       KT = TypeVar("KT")
       VT = TypeVar("VT")
       KT_contra = TypeVar("KT_contra", contravariant=True)
       VT_co = TypeVar("VT_co", covariant=True)
       
       class SupportsGetItem(Protocol[KT_contra, VT_co]):
           def __getitem__(self, key: KT_contra, /) -> VT_co: ...
       
       class TestDict(TypedDict):
           a: int
           b: str
       
       def getitem(dct: SupportsGetItem[KT, VT], key: KT) -> VT: ...
       
       test_dict: TestDict
       if TYPE_CHECKING:
           reveal_type(getitem(test_dict, "a"))  # Revealed type is "builtins.object*"
           reveal_type(getitem(test_dict, "b"))  # Revealed type is "builtins.object*"
           reveal_type(getitem(test_dict, "c"))  # Revealed type is "builtins.object*"
       ```
    

    Sorry for the random reply but is this now supported now that we are beyond1.22? If so, the numpy typing documents have not made it clear, in the dev or in the stable release

  26. marcospgp commented on Mar 17, 2024

    @marcospgp

    Is there an update on this? This issue is referenced on Stack Overflow about ndarray typing still being a work in progress, but I suspect advances have been made as some python libraries have ok typing for ndarray returning functions.

  27. jorenham commented on Aug 17, 2024

    @jorenham
    Member
  28. Jasha10 commented on Aug 30, 2024

    @Jasha10

    For those who were following this issue and want to know when more type hint features will be available, here are some related issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions