You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Type hinting / annotation (PEP 484) for ndarray, dtype, and ufunc #7370
Feature request: Organic support for PEP 484 with Numpy data structures.
Has anyone implemented type hinting for the specific numpy.ndarray class?
Right now, I'm using typing.Any, but it would be nice to have something more specific.
For instance if the numpy people added a type alias for their array_like object class. Better yet, implement support at the dtype level, so that other objects would be supported, as well as ufunc.
Reacted by Christopher Barber, Yury Parfenov, chase, Jacques Kvam, Mitar, Michał Krassowski, Charlie, Julien Nicoulaud, Ivan Levkivskyi, Alexey Strokach and 234 more
Reacted by Adam J. Stewart, 塔奇巧克力, Janosh Riebesell and junsoo-ha-unknot
Reacted by Simon Biggs, Fritz Lekschas, Manuel, Muhammad N ElNokrashy, Eric Arellano, Ryan Soklaski, Karbovnichy Vasiliy, Carlos Ramos Carreño, Samuel Ainsworth, Thomas Jahoda and 30 more
Reacted by Serg, Eugene Marushchenko, Ilya Kalimulin, yasir-din-cko, Genaro Camele, Arseniy Terekhin, Pushkar Nimkar, Radoslaw Ejsmont, Li Hongzhang, David Assefa Tofu and 5 more
Reacted by Daniel Gomez, Tommy D. Rossi, Eugene Marushchenko, Ilya Kalimulin, yasir-din-cko, jolaem, Radoslaw Ejsmont, Li Hongzhang, David Assefa Tofu, Milosz Bednarzak and 4 more
I don't think anyone's thought about it. Perhaps you would like to? :-)
I'm also going to suggest that if you want to followup on this that we close the gh issue and move the discussion to the mailing list, since it's better suited to open-ended design discussions.
To be clear, we don't actually have any objection to supporting cool new python features or anything (rather the opposite); it's just that we're a volunteer run project without many resources, so stuff only happens if someone who's interested steps up to do it.
The mailing list is usually the best place if you're trying to start working on something or hoping to recruit some other interested folks to help.
Thanks, @njsmith. I decided to start here because of the more orderly issue-tracking, as opposed to an unstructured mailing list (I was looking for a 'feature request' tag, among other features...)
Since the guy who answered me on SO got back to me with a viable solution, I decided to leave the matter.
Maybe the Numpy documentation should be updated to include his answer (please make sure to give him credit if you do).
I'm reopening this issue for those who are interested in discussing it further.
I think this would certainly be desirable for NumPy, but there are indeed a few tricky aspects of the NumPy API for typing to sort through, such as how NumPy currently accepts arbitrary objects in the np.array constructor (though we want to clean this up, see #5353).
Reacted by Ryan Soklaski, Frank Niemeyer, Tautvydas Misiunas, Simon Biggs, Jules Gagnon-Marchand, Juha Jeronen, Giorgio Balestrieri, Inon S. and Janosh Riebesell
This really would be a great addition to NumPy. What would be the next steps to push this up to typeshed or NumPy? Even an incomplete stub would be useful and I'm happy to help with a bit of direction?
@henryJack The best place to start would probably be tooling: figure out how we can integrate basic type annotations into the NumPy repository (and ideally test them) in a way that works with mypy and supports adding them incrementally.
Then, start with extremely minimal annotations and we can go from there. In particular, I would skip dtype annotations for now since we don't have a good way to specify them (i.e., only do ndarray, not ndarray[int]).
If it's helpful, I have an alternative version of annotations that I've written for use at Google and could open source. But we have our own unique build system and do type checking with pytype, so there would likely be quirks porting it to upstream.
Another quick update: Pradeep Kumar Srinivasan and I will be giving a talk on the approach we've been experimenting with over the past 6 months at the PyCon 2021 Typing Summit next week, Catching Tensor Shape Errors Using the Type Checker: https://us.pycon.org/2021/summits/typing/ We'll be discussing how it works, what it looks like in practice, and a few of its current limitations. Hope to see you there!
Reacted by Alexey Strokach, Maximilian Roos, c-bata, codethief, Jonathan Striebel, Eastsun and Bas van Beek
Some time ago, back in #17719, dtype support was introduced for np.ndarray (including a placeholder slot for shapes). As a followup, earlier today a new PR was submitted (#18935) adding a runtime-subscriptable alias for np.ndarray[Any, np.dtype[~Scalar]], the latter providing a convenient (and compact) alias for annotating arrays with a given dtype and unspecified shape:
@BvB93 This is awesome. I've been using your change since numpy 1.21 came out. Are you planning on adding runtime subscripting like np.ndarray[Any, Any], np.integer[Any], np.floating[Any], np.dtype[Any]? MyPy complains if I don't put Any since I have the flag for no implicit generics, and I end up having to work around numpy's missing __class_getitem__.
@NeilGirdhar excellent idea. As a workaround in the meantime, you could try using from __future__ import annotations, which should make it work at runtime (as usually annotations are not evaluated at runtime anymore).
@rggjan Yup, thanks! That works in many cases except when you want to do set aliases like this. Also, pylint rightly complains that these type objects are not subscriptable.
Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).
Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).
Unfortunately not, and I very much doubt that list-of-tuples-syntax will ever be something that mypy will understand (not without some serious plugin magic, at least).
As for structured arrays in general, there two main challenges here:
How to type the necasary structure into the np.void dtype.
Ideally we'd make it generic w.r.t. to something like TypedDict, so field dtypes can be assigned to each key (i.e. field names). Making np.void generic is however complicated by its flexibility, as it can be used for representing opaque bytes sequences ("V10"), dtypes with a field size ((np.float64, 8)) and structured dtypes with a set of keys and matching field dtypes ([("a", np.float64)] or [("a", np.float64, 8)]).
Only the last category can reasonably be express via a TypedDict, so this raises the question what do about the other two? Make it generic w.r.t. a trinary Union? Treat all three categories as type-check-only subclasses? This is very much an open question.
How to let ndarray.__getitem__ and __setitem__ access the named fields.
Letting ndarray access and use the fields encoded within the dtype will be a challenge of its own. Namelly, the only two types that currently deal with arbitrary named fields (NamedTuple and TypedDict) have, in my experience, proven to be less than cooperative when dispatching with the help of protocols. This is very much a mypy bug, but in this context it's probably going to be a detrimental one.
For example:
fromtypingimportTypedDict, Protocol, TypeVar, TYPE_CHECKINGKT=TypeVar("KT")
VT=TypeVar("VT")
KT_contra=TypeVar("KT_contra", contravariant=True)
VT_co=TypeVar("VT_co", covariant=True)
classSupportsGetItem(Protocol[KT_contra, VT_co]):
def__getitem__(self, key: KT_contra, /) ->VT_co: ...
classTestDict(TypedDict):
a: intb: strdefgetitem(dct: SupportsGetItem[KT, VT], key: KT) ->VT: ...
test_dict: TestDictifTYPE_CHECKING:
reveal_type(getitem(test_dict, "a")) # Revealed type is "builtins.object*"reveal_type(getitem(test_dict, "b")) # Revealed type is "builtins.object*"reveal_type(getitem(test_dict, "c")) # Revealed type is "builtins.object*"
@NeilGirdhar there is currently a PR up for making number, dtype and ndarray runtime-subscriptable (#19879),
though note that this functionality does have a hard dependency on python >= 3.9.
The hope is to wrap things up before the next 1.22 release.
Reacted by Neil Girdhar, John Hagen, Carl Thomé, Jasha Sommer-Simpson and Why-not-now
Is it possible to annotate structured arrays and if so how? I tried ndarray[Any, [('i', np.int16), ('q', np.uint16)]] but got Bracketed expression "[...]" is not valid as a type (likewise for other failed attempts).
Unfortunately not, and I very much doubt that list-of-tuples-syntax will ever be something that mypy will understand (not without some serious plugin magic, at least).
As for structured arrays in general, there two main challenges here:
1. How to type the necasary structure into the `np.void` dtype.
Ideally we'd make it generic w.r.t. to something like `TypedDict`, so field dtypes can be assigned to each key (_i.e._ field names). Making `np.void` generic is however complicated by its flexibility, as it can be used for representing opaque bytes sequences (`"V10"`), dtypes with a field size (`(np.float64, 8)`) and structured dtypes with a set of keys and matching field dtypes (`[("a", np.float64)]` or `[("a", np.float64, 8)]`).
Only the last category can reasonably be express via a `TypedDict`, so this raises the question what do about the other two? Make it generic w.r.t. a trinary `Union`? Treat all three categories as type-check-only subclasses? This is very much an open question.
2. How to let `ndarray.__getitem__` and `__setitem__` access the named fields.
Letting `ndarray` access and use the fields encoded within the dtype will be a challenge of its own. Namelly, the only two types that currently deal with arbitrary named fields (`NamedTuple` and `TypedDict`) have, in my experience, proven to be less than cooperative when dispatching with the help of protocols. This is very much a mypy bug, but in this context it's probably going to be a detrimental one.
For example:
```python
from typing import TypedDict, Protocol, TypeVar, TYPE_CHECKING
KT = TypeVar("KT")
VT = TypeVar("VT")
KT_contra = TypeVar("KT_contra", contravariant=True)
VT_co = TypeVar("VT_co", covariant=True)
class SupportsGetItem(Protocol[KT_contra, VT_co]):
def __getitem__(self, key: KT_contra, /) -> VT_co: ...
class TestDict(TypedDict):
a: int
b: str
def getitem(dct: SupportsGetItem[KT, VT], key: KT) -> VT: ...
test_dict: TestDict
if TYPE_CHECKING:
reveal_type(getitem(test_dict, "a")) # Revealed type is "builtins.object*"
reveal_type(getitem(test_dict, "b")) # Revealed type is "builtins.object*"
reveal_type(getitem(test_dict, "c")) # Revealed type is "builtins.object*"
```
Sorry for the random reply but is this now supported now that we are beyond1.22? If so, the numpy typing documents have not made it clear, in the dev or in the stable release
Reacted by Jacob Stevens-Haas, Marcos Pereira, Andrii Oriekhov, Joren Hammudoglu and Felix F Zimmermann
Is there an update on this? This issue is referenced on Stack Overflow about ndarray typing still being a work in progress, but I suspect advances have been made as some python libraries have ok typing for ndarray returning functions.
Activity
njsmith commentedon Mar 2, 2016
I don't think anyone's thought about it. Perhaps you would like to? :-)
I'm also going to suggest that if you want to followup on this that we close the gh issue and move the discussion to the mailing list, since it's better suited to open-ended design discussions.
InonS commentedon Mar 2, 2016
After getting this answer on SO, I've decided to close the issue.
njsmith commentedon Mar 3, 2016
To be clear, we don't actually have any objection to supporting cool new python features or anything (rather the opposite); it's just that we're a volunteer run project without many resources, so stuff only happens if someone who's interested steps up to do it.
The mailing list is usually the best place if you're trying to start working on something or hoping to recruit some other interested folks to help.
InonS commentedon Mar 3, 2016
Thanks, @njsmith. I decided to start here because of the more orderly issue-tracking, as opposed to an unstructured mailing list (I was looking for a 'feature request' tag, among other features...)
Since the guy who answered me on SO got back to me with a viable solution, I decided to leave the matter.
Maybe the Numpy documentation should be updated to include his answer (please make sure to give him credit if you do).
Thanks, again!
JulesGM commentedon Apr 27, 2017
hello guys! I was just kindly wondering if there had been any progress on this issue. Thanks.
eric-wieser commentedon Apr 27, 2017
There is some discussion about it on the mailing list here.
shoyer commentedon May 8, 2017
I'm reopening this issue for those who are interested in discussing it further.
I think this would certainly be desirable for NumPy, but there are indeed a few tricky aspects of the NumPy API for typing to sort through, such as how NumPy currently accepts arbitrary objects in the
np.arrayconstructor (though we want to clean this up, see #5353).engnadeau commentedon Jun 16, 2017
Some good work is being done here: https://github.com/machinalis/mypy-data
There's discussion about whether to push the work upstream to numpy or typeshed: machinalis/mypy-data#16
njsmith commentedon Jun 16, 2017
CC @mrocklin
henryJack commentedon Sep 1, 2017
This really would be a great addition to NumPy. What would be the next steps to push this up to typeshed or NumPy? Even an incomplete stub would be useful and I'm happy to help with a bit of direction?
shoyer commentedon Sep 1, 2017
@henryJack The best place to start would probably be tooling: figure out how we can integrate basic type annotations into the NumPy repository (and ideally test them) in a way that works with mypy and supports adding them incrementally.
Then, start with extremely minimal annotations and we can go from there. In particular, I would skip dtype annotations for now since we don't have a good way to specify them (i.e., only do
ndarray, notndarray[int]).If it's helpful, I have an alternative version of annotations that I've written for use at Google and could open source. But we have our own unique build system and do type checking with pytype, so there would likely be quirks porting it to upstream.
jwkvam commentedon Sep 1, 2017
I suppose the only way to test annotations to actually run mypy on sample code snippets and check the output?
Would it be better to have the annotations integrated with the code or as separate stubs?
I suppose we should also learn from dropbox and pandas that we should start with the leaves of the codebase versus core data structures?
63 remaining items
mrahtz commentedon May 4, 2021
Another quick update: Pradeep Kumar Srinivasan and I will be giving a talk on the approach we've been experimenting with over the past 6 months at the PyCon 2021 Typing Summit next week, Catching Tensor Shape Errors Using the Type Checker: https://us.pycon.org/2021/summits/typing/ We'll be discussing how it works, what it looks like in practice, and a few of its current limitations. Hope to see you there!
BvB93 commentedon May 7, 2021
Hi all,
Some time ago, back in #17719, dtype support was introduced for
np.ndarray(including a placeholder slot for shapes). As a followup, earlier today a new PR was submitted (#18935) adding a runtime-subscriptable alias fornp.ndarray[Any, np.dtype[~Scalar]], the latter providing a convenient (and compact) alias for annotating arrays with a given dtype and unspecified shape:NeilGirdhar commentedon Jul 7, 2021
@BvB93 This is awesome. I've been using your change since numpy 1.21 came out. Are you planning on adding runtime subscripting like
np.ndarray[Any, Any],np.integer[Any],np.floating[Any],np.dtype[Any]? MyPy complains if I don't putAnysince I have the flag for no implicit generics, and I end up having to work around numpy's missing__class_getitem__.rggjan commentedon Jul 7, 2021
@NeilGirdhar excellent idea. As a workaround in the meantime, you could try using
from __future__ import annotations, which should make it work at runtime (as usually annotations are not evaluated at runtime anymore).NeilGirdhar commentedon Jul 7, 2021
@rggjan Yup, thanks! That works in many cases except when you want to do set aliases like this. Also, pylint rightly complains that these type objects are not subscriptable.
gsakkis commentedon Aug 28, 2021
Is it possible to annotate structured arrays and if so how? I tried
ndarray[Any, [('i', np.int16), ('q', np.uint16)]]but gotBracketed expression "[...]" is not valid as a type(likewise for other failed attempts).Jasha10 commentedon Aug 28, 2021
I believe that as of Numpy 1.21 it is not yet possible.
BvB93 commentedon Aug 30, 2021
Unfortunately not, and I very much doubt that list-of-tuples-syntax will ever be something that mypy will understand (not without some serious plugin magic, at least).
As for structured arrays in general, there two main challenges here:
How to type the necasary structure into the
np.voiddtype.Ideally we'd make it generic w.r.t. to something like
TypedDict, so field dtypes can be assigned to each key (i.e. field names). Makingnp.voidgeneric is however complicated by its flexibility, as it can be used for representing opaque bytes sequences ("V10"), dtypes with a field size ((np.float64, 8)) and structured dtypes with a set of keys and matching field dtypes ([("a", np.float64)]or[("a", np.float64, 8)]).Only the last category can reasonably be express via a
TypedDict, so this raises the question what do about the other two? Make it generic w.r.t. a trinaryUnion? Treat all three categories as type-check-only subclasses? This is very much an open question.How to let
ndarray.__getitem__and__setitem__access the named fields.Letting
ndarrayaccess and use the fields encoded within the dtype will be a challenge of its own. Namelly, the only two types that currently deal with arbitrary named fields (NamedTupleandTypedDict) have, in my experience, proven to be less than cooperative when dispatching with the help of protocols. This is very much a mypy bug, but in this context it's probably going to be a detrimental one.For example:
BvB93 commentedon Sep 16, 2021
@NeilGirdhar there is currently a PR up for making
number,dtypeandndarrayruntime-subscriptable (#19879),though note that this functionality does have a hard dependency on python >= 3.9.
The hope is to wrap things up before the next 1.22 release.
Jasha10 commentedon Nov 21, 2021
@NeilGirdhar The workaround I've been using in that case is to put quotes around the subscripted numpy type:
Why-not-now commentedon Dec 6, 2022
Sorry for the random reply but is this now supported now that we are beyond1.22? If so, the numpy typing documents have not made it clear, in the dev or in the stable release
marcospgp commentedon Mar 17, 2024
Is there an update on this? This issue is referenced on Stack Overflow about
ndarraytyping still being a work in progress, but I suspect advances have been made as some python libraries have ok typing forndarrayreturning functions.jorenham commentedon Aug 17, 2024
Almost all of NumPy has type annotations that are compatible with PEP 484:
https://numpy.org/doc/stable/reference/typing.html
Jasha10 commentedon Aug 30, 2024
For those who were following this issue and want to know when more type hint features will be available, here are some related issues: