You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I agree with @shoyer. type hints should be implemented as early as possible. I've taken a couple days off work, so I can get started on annotations for py2 tomorrow. Do you have any preference on how large pull requests should be? if it were up to me, given the relative safety of this sort of modification, I would probably do a PR per sub-module. Let me know if you'd like me to do it differently. After that i can move on to stubs
essentially you are taking a file, say pandas/core/series.py and stripping out the impl, leaving the signatures. Then we validate that this works with mypy. I really really want to be sure this stays in sync with the actual signature / impl files. So we need tests that compares these directly (in py3 this is actually not that hard, you can iterate over the definitions and simply compare the signatures). I think if we only compare this in py3 would be ok actually as our definitions should be py2/3 compat anyways. (or do we want / need differing signatures, e.g. a 2 and 3 impl)?
@jreback, to be honest, i've never used stubs as a core part of a project I was working on. I've only used them to supplement underlying dependencies or to workaround bugs in my IDE, so i've never thought too deeply about what consequences they might have
ok, I'm about to get started. To be clear, I'm going to only concentrate on method signatures and return types (for now). I'm not going to annotate local variables unless I have some justifiable reason.
Regarding the relevant parts of the contributing docs:
style: for short signatures i'll do the annotations on a single line, for longer ones, i'll annotate each parameter individually
committing: I should use the "CLN" prefix since this issue has the clean label, correct?
docs: Does this change warrant an entry in whatsnew/v0.20.0.txt? if so, where? under Other Enhancements?
I've never contributed to pandas so please let me know if there's anything I'm overlooking
importpandasfrompandas.api.typesimportCategoricalDtypefrompathlibimportPathfrompy._path.localimportLocalPathfromtypingimportUnion, Textfromtyping_extensionsimportProtocolclassSupportsRead(Protocol):
defread(self) ->Union[Text, bytes]:
...
RegionDtype=CategoricalDtype('RegionDtype', ['north', 'south', 'east', 'west'])
classMyDataFrameType(pandas.TypedDataFrame):
name=strid=intaddress=strregion=RegionDtypePandasReadable=Union[Text, Path, LocalPath, SupportsRead]
# or better yet, having a PandasReadable class accessible in the Pandas hierarchy somewheredefload_my_data(filepath_or_buffer: PandasReadable=None) ->MyDataFrameType:
dtypes=OrderedDict([
('name', str),
('id', int),
('address', str),
('region', RegionDtype)
])
# or ideally something more elgant like dtypes = MyDataFrameType.dtypesreturnpandas.read_csv(filepath_or_buffer, usecols=list(dtypes.keys()), dtypes=dtypes)
Reacted by Travis LaGrone, Luca Palmieri, Janis, wolfgang kuehn, Alexander Gil Casas, Duncan Mortimer, Austin Ray, fhiyo, Joël Bourgault, ali-naq-v and 9 more
Is there any way people can contribute to move this forward? I'm planning on start type checking a project and I'd definitely like having support for pandas types as well.
Currently considering mypy and pyre-check
Yep (and I have been in places, probably incorrectly). We'd just need to
convert those to annotations, though mypy is probably capable of using a
mixture?
I think the main benefit is that the `typing` module will be available.
@josh-theorem if you are referring to static analysis then no, we don't currently support generic parametrization of Series / Index / DataFrame objects. Not opposed to it in the long run but we have a large part of the code base that just needs to be annotated first.
If you would like to help out there is #26766 which is a pre-cursor to what you are asking for. Could definitely use community support to push that along if you'd like to submit PRs
Activity
max-sixty commentedon Oct 21, 2016
#12412
jorisvandenbossche commentedon Oct 22, 2016
See also wesm/pandas2#18 (a advantage for targetting this for 2.0 is we can directly use the python 3 syntax)
brmc commentedon Mar 12, 2017
continuing the conversation from wesm/pandas2#18
I agree with @shoyer. type hints should be implemented as early as possible. I've taken a couple days off work, so I can get started on annotations for py2 tomorrow. Do you have any preference on how large pull requests should be? if it were up to me, given the relative safety of this sort of modification, I would probably do a PR per sub-module. Let me know if you'd like me to do it differently. After that i can move on to stubs
jreback commentedon Mar 12, 2017
@brmc sounds great!
I think main things to add are support for
Series,DataFrame,Indexmethods (Timestamp, Timedelta, Periodwould be nice to).you are wanting to add
.pyifiles where the module live? (.e.g.pandas/core/)will need to add a build with
mypyas well for validation.are there other projects that do this (just for seeing examples), if so how are they adding it?
jreback commentedon Mar 12, 2017
A concern of mine is this:
essentially you are taking a file, say
pandas/core/series.pyand stripping out the impl, leaving the signatures. Then we validate that this works with mypy. I really really want to be sure this stays in sync with the actual signature / impl files. So we need tests that compares these directly (in py3 this is actually not that hard, you can iterate over the definitions and simply compare the signatures). I think if we only compare this in py3 would be ok actually as our definitions should be py2/3 compat anyways. (or do we want / need differing signatures, e.g. a 2 and 3 impl)?Have others done this?
TomAugspurger commentedon Mar 12, 2017
We'll want to push these upstream to https://github.com/python/typeshed/ once we have reasonable coverage.
TomAugspurger commentedon Mar 12, 2017
Oh, and I hadn't heard of this, but https://github.com/python/mypy/blob/master/mypy/stubgen.py may be a good starting place? I guess it operates one module at a time so
brmc commentedon Mar 13, 2017
@jreback, to be honest, i've never used stubs as a core part of a project I was working on. I've only used them to supplement underlying dependencies or to workaround bugs in my IDE, so i've never thought too deeply about what consequences they might have
brmc commentedon Mar 13, 2017
ok, I'm about to get started. To be clear, I'm going to only concentrate on method signatures and return types (for now). I'm not going to annotate local variables unless I have some justifiable reason.
Regarding the relevant parts of the contributing docs:
whatsnew/v0.20.0.txt? if so, where? underOther Enhancements?I've never contributed to pandas so please let me know if there's anything I'm overlooking
jreback commentedon Mar 13, 2017
contributing docs are: http://pandas-docs.github.io/pandas-docs-travis/contributing.html
sounds good
that's fine
yes
brmc commentedon Mar 13, 2017
right. I had already read through it. just double-checking I didn't miss anything major
16 remaining items
shoyer commentedon Apr 30, 2018
@gwerbin sorry, can you clarify exact what I wrote that you'd want out of a pandas type system? :)
gwerbin commentedon Apr 30, 2018
@shoyer the ability to do this:
westurner commentedon Aug 7, 2018
PEP-484 Type Annotations tools:
https://github.com/dropbox/pyannotate
https://github.com/Instagram/MonkeyType
EDIT: https://mypy.readthedocs.io/en/latest/existing_code.html
mfcabrera commentedon Aug 16, 2018
@teh do you have your stubs somewhere? and do you plan to continue this route?. I would interested in helping.
nicoabie commentedon Nov 28, 2018
Is there any way people can contribute to move this forward? I'm planning on start type checking a project and I'd definitely like having support for pandas types as well.
Currently considering mypy and pyre-check
TomAugspurger commentedon Nov 28, 2018
nicoabie commentedon Nov 28, 2018
Thanks @TomAugspurger, I'd do the same. If help can be provided for the migration I'm sure the community will be there me included.
jorisvandenbossche commentedon Nov 28, 2018
But in principle, some work could already be started using type comments?
TomAugspurger commentedon Nov 28, 2018
josh-theorem commentedon Aug 5, 2019
So, this is marked done, is it possible to have dataframes with specified column types now?
WillAyd commentedon Aug 5, 2019
@josh-theorem if you are referring to static analysis then no, we don't currently support generic parametrization of Series / Index / DataFrame objects. Not opposed to it in the long run but we have a large part of the code base that just needs to be annotated first.
If you would like to help out there is #26766 which is a pre-cursor to what you are asking for. Could definitely use community support to push that along if you'd like to submit PRs