Skip to content

adding static type checking with mypy #14468

Closed

Description

@jreback

Activity

  1. added this to the milestone on Oct 21, 2016
  2. max-sixty commented on Oct 21, 2016

    @max-sixty
    Contributor
  3. jorisvandenbossche commented on Oct 22, 2016

    @jorisvandenbossche
    Member

    See also wesm/pandas2#18 (a advantage for targetting this for 2.0 is we can directly use the python 3 syntax)

  4. brmc commented on Mar 12, 2017

    @brmc

    continuing the conversation from wesm/pandas2#18

    I agree with @shoyer. type hints should be implemented as early as possible. I've taken a couple days off work, so I can get started on annotations for py2 tomorrow. Do you have any preference on how large pull requests should be? if it were up to me, given the relative safety of this sort of modification, I would probably do a PR per sub-module. Let me know if you'd like me to do it differently. After that i can move on to stubs

  5. jreback commented on Mar 12, 2017

    @jreback
    ContributorAuthor

    @brmc sounds great!

    I think main things to add are support for Series, DataFrame, Index methods (Timestamp, Timedelta, Period would be nice to).

    you are wanting to add .pyi files where the module live? (.e.g. pandas/core/)

    will need to add a build with mypy as well for validation.

    are there other projects that do this (just for seeing examples), if so how are they adding it?

  6. jreback commented on Mar 12, 2017

    @jreback
    ContributorAuthor

    A concern of mine is this:

    essentially you are taking a file, say pandas/core/series.py and stripping out the impl, leaving the signatures. Then we validate that this works with mypy. I really really want to be sure this stays in sync with the actual signature / impl files. So we need tests that compares these directly (in py3 this is actually not that hard, you can iterate over the definitions and simply compare the signatures). I think if we only compare this in py3 would be ok actually as our definitions should be py2/3 compat anyways. (or do we want / need differing signatures, e.g. a 2 and 3 impl)?

    Have others done this?

  7. TomAugspurger commented on Mar 12, 2017

    @TomAugspurger
    Contributor

    We'll want to push these upstream to https://github.com/python/typeshed/ once we have reasonable coverage.

  8. TomAugspurger commented on Mar 12, 2017

    @TomAugspurger
    Contributor

    Oh, and I hadn't heard of this, but https://github.com/python/mypy/blob/master/mypy/stubgen.py may be a good starting place? I guess it operates one module at a time so

    mkdir out
    stubgen pandas.core.series
    stubgen pandas.core.frame
    stubgen pandas.core.index
  9. brmc commented on Mar 13, 2017

    @brmc

    @jreback, to be honest, i've never used stubs as a core part of a project I was working on. I've only used them to supplement underlying dependencies or to workaround bugs in my IDE, so i've never thought too deeply about what consequences they might have

  10. brmc commented on Mar 13, 2017

    @brmc

    ok, I'm about to get started. To be clear, I'm going to only concentrate on method signatures and return types (for now). I'm not going to annotate local variables unless I have some justifiable reason.

    Regarding the relevant parts of the contributing docs:

    • style: for short signatures i'll do the annotations on a single line, for longer ones, i'll annotate each parameter individually
    • committing: I should use the "CLN" prefix since this issue has the clean label, correct?
    • docs: Does this change warrant an entry in whatsnew/v0.20.0.txt? if so, where? under Other Enhancements?

    I've never contributed to pandas so please let me know if there's anything I'm overlooking

  11. jreback commented on Mar 13, 2017

    @jreback
    ContributorAuthor

    contributing docs are: http://pandas-docs.github.io/pandas-docs-travis/contributing.html

    style: for short signatures i'll do the annotations on a single line, for longer ones, i'll annotate each parameter individually

    sounds good

    committing: I should use the "CLN" prefix since this issue has the clean label, correct?

    that's fine

    docs: Does this change warrant an entry in whatsnew/v0.20.0.txt? if so, where? under Other Enhancements?

    yes

  12. brmc commented on Mar 13, 2017

    @brmc
  13. 16 remaining items

  14. shoyer commented on Apr 30, 2018

    @shoyer
    Member

    @gwerbin sorry, can you clarify exact what I wrote that you'd want out of a pandas type system? :)

  15. gwerbin commented on Apr 30, 2018

    @gwerbin

    @shoyer the ability to do this:

    import pandas
    from pandas.api.types import CategoricalDtype
    from pathlib import Path
    from py._path.local import LocalPath
    from typing import Union, Text
    from typing_extensions import Protocol
    
    class SupportsRead(Protocol):
        def read(self) -> Union[Text, bytes]:
           ...
    
    RegionDtype = CategoricalDtype('RegionDtype', ['north', 'south', 'east', 'west'])
    
    class MyDataFrameType(pandas.TypedDataFrame):
        name = str
        id = int
        address = str
        region = RegionDtype
    
    PandasReadable = Union[Text, Path, LocalPath, SupportsRead]
    # or better yet, having a PandasReadable class accessible in the Pandas hierarchy somewhere
    
    def load_my_data(filepath_or_buffer: PandasReadable=None) -> MyDataFrameType:
        dtypes = OrderedDict([
            ('name', str),
            ('id', int),
            ('address', str),
            ('region', RegionDtype)
        ])
        # or ideally something more elgant like dtypes = MyDataFrameType.dtypes
    
        return pandas.read_csv(filepath_or_buffer, usecols=list(dtypes.keys()), dtypes=dtypes)
  16. mfcabrera commented on Aug 16, 2018

    @mfcabrera

    @teh do you have your stubs somewhere? and do you plan to continue this route?. I would interested in helping.

  17. nicoabie commented on Nov 28, 2018

    @nicoabie

    Is there any way people can contribute to move this forward? I'm planning on start type checking a project and I'd definitely like having support for pandas types as well.
    Currently considering mypy and pyre-check

  18. TomAugspurger commented on Nov 28, 2018

    @TomAugspurger
    Contributor
  19. nicoabie commented on Nov 28, 2018

    @nicoabie

    Thanks @TomAugspurger, I'd do the same. If help can be provided for the migration I'm sure the community will be there me included.

  20. jorisvandenbossche commented on Nov 28, 2018

    @jorisvandenbossche
    Member

    But in principle, some work could already be started using type comments?

  21. TomAugspurger commented on Nov 28, 2018

    @TomAugspurger
    Contributor
  22. modified the milestones: , 0.25.0 on Mar 24, 2019
  23. josh-theorem commented on Aug 5, 2019

    @josh-theorem

    So, this is marked done, is it possible to have dataframes with specified column types now?

  24. WillAyd commented on Aug 5, 2019

    @WillAyd
    Member

    @josh-theorem if you are referring to static analysis then no, we don't currently support generic parametrization of Series / Index / DataFrame objects. Not opposed to it in the long run but we have a large part of the code base that just needs to be annotated first.

    If you would like to help out there is #26766 which is a pre-cursor to what you are asking for. Could definitely use community support to push that along if you'd like to submit PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIContinuous IntegrationCleanTypingtype annotations, mypy/pyright type checking

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions