Skip to content

Add mypy static type checking to our Travis CI setup #635

Closed

Description

@timabbott

It'd be great to add mypy static type checking to Travis CI so that we can avoid regressions making their way into the codebase (we have a couple right now, see #631 (comment)).

The current command line we're using is:

mypy --silent --py2 --check-untyped-defs zerver/ zproject/ bots/ zilencer/ analytics/ corporate/ confirmation/ tools/ scripts/setup/ scripts/lib/ *.py api/ | grep -v test | grep -v fenced_code | grep -v settings

It should be fairly easy to make a tool/run-mypy type script using @sharmaeklavya2's new lister.py script, replacing those grep -v lines with exclude rules. In order to make this all work, though, we'll need to arrange to have mypy installed as part of the Zulip development environment setup. May want to build on top of #634 to avoid a bunch of refactoring after that lands.

For actually integrating this with Travis CI, basically we'll want to add a new TEST_SUITE=mypy line in .travis.yml and then add scripts to do the setup (in tools/travis/setup-mypy) and actually run it (tools/travis/mypy). Mypy runs under python 3, so the setup process will likely involve installing Python 3, setting up pip3, and then using pip3 to install a current sha1 from mypy git (we'll want the sha1 pinned, so that if mypy breaks their master branch, it doesn't affect us).

@ashishk1994 as FYI; @sharmaeklavya2 are you interested in taking a look at adding this?

It could be useful to configure Travis CI to run builds against branches in your fork of zulip.git so that you can run Travis CI tests without opening a pull request (it's free to sign up since Zulip is open source).

Activity

  1. ashishk1994 commented on Apr 7, 2016

    @ashishk1994
    Contributor

    @timabbott
    I think to use the lister.py, we need to either add some condition at https://github.com/zulip/zulip/blob/master/tools/lister.py#L73 or give the path to every test or settings (excluding) file as we are excluding files only if it starts with the given name or exactly matches. I think we need something of regex search i.e. what grep does
    @timabbott @sharmaeklavya2 Also, I have tried adding a simple condition in lister.py and created a tool here:
    master...ashishk1994:travis-ci
    But it is giving me error

    zerver/views/webhooks.py:1: error: Duplicate module named 'zerver.views.webhooks'
    Thanks

  2. sharmaeklavya2 commented on Apr 7, 2016

    @sharmaeklavya2
    Member

    Here is my progress till now:

    I realized a few days ago that mypy doesn't run recursively on directories specified to it. It only runs on files immediately inside the given directories.

    For eg, mypy --py2 --silent --check-untyped-defs api/integrations/ doesn't show any errors, but mypy --py2 --silent --check-untyped-defs api/integrations/asana/ does. I thought putting an __init__.py in api/integrations/ might make it work, but it didn't.

    So we are actually not testing all files in the directories given in the command line above. This means that there are untested files which might fail when mypy is run on them (and a lot of them actually do fail!). Hence, we'll need to rethink which files we should exclude.

    Here is the command line which I initially used to test all python files:
    tools/lister.py -f py | xargs mypy --silent --py2 --check-untyped-defs

    It takes around 20s to run on my laptop. But this gave some errors which stopped execution.

    ...
    tools/webpack: note: At top level:
    tools/webpack:1: error: Duplicate module named '__main__'
    

    I tried to find out the cause of this error, but I couldn't.

    Here is the complete output:
    mypy_report_2.txt

    So I decided to (for now) check each file in a separate mypy instance.
    tools/lister.py -f py | xargs -L 1 mypy --silent --py2 --check-untyped-defs

    This took around 7.5 minutes on my laptop. The above errors on duplicate module went away when I ran it like this, but I encountered an internal error on tools/print-all/print-all. I deleted that file temporarily to proceed with completing the mypy check.

    This is the complete output:
    mypy_report.txt

    Unfortunately, mypy gives different errors on the same files when run like this (maybe because of inferior type inference because of not giving it all dependencies of a file. For eg in zerver/lib/bugdown/, running mypy on only __init__.py gives 4 errors in __init__.py, while running mypy on both __init__.py and codehilite.py gives 3 errors in __init__.py. The error which went away said zerver/lib/bugdown/__init__.py:34: error: Module has no attribute 'codehilite').

  3. sharmaeklavya2 commented on Apr 7, 2016

    @sharmaeklavya2
    Member

    @ashishk1994, #637 fixes the error you are getting.

  4. sharmaeklavya2 commented on Apr 7, 2016

    @sharmaeklavya2
    Member

    @ashishk1994, I looked at the modification you made to lister.py. If we exclude files like that, a user using that script might accidentally exclude files he/she did not mean to (for example, the pattern 'zerver' will also exclude files in templates/zerver). We should probably add an option to choose what kind of exclusion strategy to use.

    We can also move all tests in zerver/ to zerver/tests/. It doesn't seem like too much work, since those tests are probably called exclusively by tools/test-backend. What's your opinion on this @timabbott?

  5. timabbott commented on Apr 7, 2016

    @timabbott
    SponsorMemberAuthor

    Thanks for the great investigations!

    @sharmaeklavya2 Moving the tests to live under a zerver/tests/ directory is totally reasonable. There are references to some of those filenames in the documentation, so we should be careful about doing it right, but overall it'll probably net cleanup the zerver/ directory.

    I think we definitely want to be running it in the way that takes 20s. FYI there's work being done on optimizing mypy's performance that will make mypy faster in the long term and also support incrementally running on just what's changed. This might even get to the point where we could run it as part of our lint pass.

    Looking at the errors that showed up via running the tool on everything, there's a few classes of errors.

    • Some, like the values initialized to None in zulip_*_config.py may be things we just want to ignore, since type annotations might be net confusing for users of those integrations.
    • Others, like the error: IO[str] not callable errors in the codebase integrations, I think are actual bugs where we have code that does sys.stderr("string") rather than sys.stderr.write("string")
    • The check-templates stuff, is because we're using a questionable coding pattern we should probably change.
    • For large third-party code like git_p4.py, it's possible we want to exlude rather than annotate because realistically we're just going to update to new versions from upstream rather than incrementally improving it ourselves.
    • Probably there will be some where the issue is actually caused by buggy stubs in the typeshed repository (or even bugs in the mypy tool), in which case we'll need to contribute there to address the issue (and can use a type: ignore to unblock things in the meantime).

    I think it probably makes sense to work through those handling them individually. From a process perspective, I think we could work on running mypy in CI (with a longer than ideal exclude list) and in parallel work on fixing the errors here.

  6. ashishk1994 commented on Apr 7, 2016

    @ashishk1994
    Contributor

    @sharmaeklavya2 Thanks for the thorough investigation. Yes you are right we should pass an argument to lister.py to choose the type of exclusion strategy, It would help us in writing tools for travis-ci. What do you say @timabbott.

    Also, I have tried to integrate mypy into travis ci (PR #639 ). Currently, mypy in travis is not throwing any error, once exclude list is decided i.e. all the errors from the mypy commands are gone then we can use it.

  7. sharmaeklavya2 commented on Apr 18, 2016

    @sharmaeklavya2
    Member

    I tried to see how many files do we need to exclude from type checking to get no errors. I'm not checking files which do not have the .py extension because of #1380 in python/mypy. Out of the 254 .py files that we have, we'll only be checking 33 files (pretty sad).

    Here is my exclude list:

    api/integrations/
    api/zulip/__init__.py
    bots/jabber_mirror_backend.py
    bots/zephyr_mirror_backend.py
    docs/conf.py
    puppet/zulip_internal/files/postgresql/pg_backup_and_purge.py
    tools/deprecated/
    tools/lister.py
    zproject/
    zerver/lib/actions.py
    zerver/lib/bugdown/__init__.py
    zerver/lib/bugdown/fenced_code.py
    zerver/lib/push_notifications.py
    zerver/migrations/
    zerver/models.py
    zerver/tests/
    zerver/views/__init__.py
    zerver/views/messages.py
    zulip_tools.py
    

    There were also instances where excluding one file caused errors of the form could not find module ..., so I had to exclude those files as well. I could have further refined this list by including only those files which were erring instead of including whole directories. But that would have made this list very long.

  8. sharmaeklavya2 commented on Apr 18, 2016

    @sharmaeklavya2
    Member

    Sorry I miscalculated. We're checking 194 files not 33. I had used wc incorrectly.

  9. added 2 commits that reference this issue on Apr 18, 2016
    2f7c28d
    28d7d43
  10. timabbott commented on Apr 19, 2016

    @timabbott
    SponsorMemberAuthor

    OK 194/254 is enough to be worth turning it on in CI; we can do the project of shrinking that list as a follow-on effort once we have this running in Travis CI.

  11. added 8 commits that reference this issue on Apr 24, 2016
    2ce0863
    332dc88
    2544c69
    683aab2
    f4ea433
    bf8a1b5
    69a9eb2
    2236624
  12. added a commit that references this issue on Apr 28, 2016
    c80f699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions