You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add mypy static type checking to our Travis CI setup #635
It'd be great to add mypy static type checking to Travis CI so that we can avoid regressions making their way into the codebase (we have a couple right now, see #631 (comment)).
It should be fairly easy to make a tool/run-mypy type script using @sharmaeklavya2's new lister.py script, replacing those grep -v lines with exclude rules. In order to make this all work, though, we'll need to arrange to have mypy installed as part of the Zulip development environment setup. May want to build on top of #634 to avoid a bunch of refactoring after that lands.
For actually integrating this with Travis CI, basically we'll want to add a new TEST_SUITE=mypy line in .travis.yml and then add scripts to do the setup (in tools/travis/setup-mypy) and actually run it (tools/travis/mypy). Mypy runs under python 3, so the setup process will likely involve installing Python 3, setting up pip3, and then using pip3 to install a current sha1 from mypy git (we'll want the sha1 pinned, so that if mypy breaks their master branch, it doesn't affect us).
It could be useful to configure Travis CI to run builds against branches in your fork of zulip.git so that you can run Travis CI tests without opening a pull request (it's free to sign up since Zulip is open source).
I realized a few days ago that mypy doesn't run recursively on directories specified to it. It only runs on files immediately inside the given directories.
For eg, mypy --py2 --silent --check-untyped-defs api/integrations/ doesn't show any errors, but mypy --py2 --silent --check-untyped-defs api/integrations/asana/ does. I thought putting an __init__.py in api/integrations/ might make it work, but it didn't.
So we are actually not testing all files in the directories given in the command line above. This means that there are untested files which might fail when mypy is run on them (and a lot of them actually do fail!). Hence, we'll need to rethink which files we should exclude.
Here is the command line which I initially used to test all python files: tools/lister.py -f py | xargs mypy --silent --py2 --check-untyped-defs
It takes around 20s to run on my laptop. But this gave some errors which stopped execution.
...
tools/webpack: note: At top level:
tools/webpack:1: error: Duplicate module named '__main__'
I tried to find out the cause of this error, but I couldn't.
So I decided to (for now) check each file in a separate mypy instance. tools/lister.py -f py | xargs -L 1 mypy --silent --py2 --check-untyped-defs
This took around 7.5 minutes on my laptop. The above errors on duplicate module went away when I ran it like this, but I encountered an internal error on tools/print-all/print-all. I deleted that file temporarily to proceed with completing the mypy check.
Unfortunately, mypy gives different errors on the same files when run like this (maybe because of inferior type inference because of not giving it all dependencies of a file. For eg in zerver/lib/bugdown/, running mypy on only __init__.py gives 4 errors in __init__.py, while running mypy on both __init__.py and codehilite.py gives 3 errors in __init__.py. The error which went away said zerver/lib/bugdown/__init__.py:34: error: Module has no attribute 'codehilite').
@ashishk1994, I looked at the modification you made to lister.py. If we exclude files like that, a user using that script might accidentally exclude files he/she did not mean to (for example, the pattern 'zerver' will also exclude files in templates/zerver). We should probably add an option to choose what kind of exclusion strategy to use.
We can also move all tests in zerver/ to zerver/tests/. It doesn't seem like too much work, since those tests are probably called exclusively by tools/test-backend. What's your opinion on this @timabbott?
@sharmaeklavya2 Moving the tests to live under a zerver/tests/ directory is totally reasonable. There are references to some of those filenames in the documentation, so we should be careful about doing it right, but overall it'll probably net cleanup the zerver/ directory.
I think we definitely want to be running it in the way that takes 20s. FYI there's work being done on optimizing mypy's performance that will make mypy faster in the long term and also support incrementally running on just what's changed. This might even get to the point where we could run it as part of our lint pass.
Looking at the errors that showed up via running the tool on everything, there's a few classes of errors.
Some, like the values initialized to None in zulip_*_config.py may be things we just want to ignore, since type annotations might be net confusing for users of those integrations.
Others, like the error: IO[str] not callable errors in the codebase integrations, I think are actual bugs where we have code that does sys.stderr("string") rather than sys.stderr.write("string")
The check-templates stuff, is because we're using a questionable coding pattern we should probably change.
For large third-party code like git_p4.py, it's possible we want to exlude rather than annotate because realistically we're just going to update to new versions from upstream rather than incrementally improving it ourselves.
Probably there will be some where the issue is actually caused by buggy stubs in the typeshed repository (or even bugs in the mypy tool), in which case we'll need to contribute there to address the issue (and can use a type: ignore to unblock things in the meantime).
I think it probably makes sense to work through those handling them individually. From a process perspective, I think we could work on running mypy in CI (with a longer than ideal exclude list) and in parallel work on fixing the errors here.
@sharmaeklavya2 Thanks for the thorough investigation. Yes you are right we should pass an argument to lister.py to choose the type of exclusion strategy, It would help us in writing tools for travis-ci. What do you say @timabbott.
Also, I have tried to integrate mypy into travis ci (PR #639 ). Currently, mypy in travis is not throwing any error, once exclude list is decided i.e. all the errors from the mypy commands are gone then we can use it.
I tried to see how many files do we need to exclude from type checking to get no errors. I'm not checking files which do not have the .py extension because of #1380 in python/mypy. Out of the 254 .py files that we have, we'll only be checking 33 files (pretty sad).
There were also instances where excluding one file caused errors of the form could not find module ..., so I had to exclude those files as well. I could have further refined this list by including only those files which were erring instead of including whole directories. But that would have made this list very long.
OK 194/254 is enough to be worth turning it on in CI; we can do the project of shrinking that list as a follow-on effort once we have this running in Travis CI.
Activity
ashishk1994 commentedon Apr 7, 2016
@timabbott
I think to use the lister.py, we need to either add some condition at https://github.com/zulip/zulip/blob/master/tools/lister.py#L73 or give the path to every test or settings (excluding) file as we are excluding files only if it starts with the given name or exactly matches. I think we need something of regex search i.e. what grep does
@timabbott @sharmaeklavya2 Also, I have tried adding a simple condition in lister.py and created a tool here:
master...ashishk1994:travis-ci
But it is giving me error
zerver/views/webhooks.py:1: error: Duplicate module named 'zerver.views.webhooks'
Thanks
sharmaeklavya2 commentedon Apr 7, 2016
Here is my progress till now:
I realized a few days ago that mypy doesn't run recursively on directories specified to it. It only runs on files immediately inside the given directories.
For eg,
mypy --py2 --silent --check-untyped-defs api/integrations/doesn't show any errors, butmypy --py2 --silent --check-untyped-defs api/integrations/asana/does. I thought putting an__init__.pyin api/integrations/ might make it work, but it didn't.So we are actually not testing all files in the directories given in the command line above. This means that there are untested files which might fail when mypy is run on them (and a lot of them actually do fail!). Hence, we'll need to rethink which files we should exclude.
Here is the command line which I initially used to test all python files:
tools/lister.py -f py | xargs mypy --silent --py2 --check-untyped-defsIt takes around 20s to run on my laptop. But this gave some errors which stopped execution.
I tried to find out the cause of this error, but I couldn't.
Here is the complete output:
mypy_report_2.txt
So I decided to (for now) check each file in a separate mypy instance.
tools/lister.py -f py | xargs -L 1 mypy --silent --py2 --check-untyped-defsThis took around 7.5 minutes on my laptop. The above errors on duplicate module went away when I ran it like this, but I encountered an internal error on tools/print-all/print-all. I deleted that file temporarily to proceed with completing the mypy check.
This is the complete output:
mypy_report.txt
Unfortunately, mypy gives different errors on the same files when run like this (maybe because of inferior type inference because of not giving it all dependencies of a file. For eg in zerver/lib/bugdown/, running mypy on only
__init__.pygives 4 errors in__init__.py, while running mypy on both__init__.pyandcodehilite.pygives 3 errors in__init__.py. The error which went away saidzerver/lib/bugdown/__init__.py:34: error: Module has no attribute 'codehilite').sharmaeklavya2 commentedon Apr 7, 2016
@ashishk1994, #637 fixes the error you are getting.
sharmaeklavya2 commentedon Apr 7, 2016
@ashishk1994, I looked at the modification you made to lister.py. If we exclude files like that, a user using that script might accidentally exclude files he/she did not mean to (for example, the pattern 'zerver' will also exclude files in templates/zerver). We should probably add an option to choose what kind of exclusion strategy to use.
We can also move all tests in zerver/ to zerver/tests/. It doesn't seem like too much work, since those tests are probably called exclusively by tools/test-backend. What's your opinion on this @timabbott?
timabbott commentedon Apr 7, 2016
Thanks for the great investigations!
@sharmaeklavya2 Moving the tests to live under a
zerver/tests/directory is totally reasonable. There are references to some of those filenames in the documentation, so we should be careful about doing it right, but overall it'll probably net cleanup the zerver/ directory.I think we definitely want to be running it in the way that takes 20s. FYI there's work being done on optimizing mypy's performance that will make mypy faster in the long term and also support incrementally running on just what's changed. This might even get to the point where we could run it as part of our lint pass.
Looking at the errors that showed up via running the tool on everything, there's a few classes of errors.
zulip_*_config.pymay be things we just want to ignore, since type annotations might be net confusing for users of those integrations.error: IO[str] not callableerrors in the codebase integrations, I think are actual bugs where we have code that doessys.stderr("string")rather than sys.stderr.write("string")check-templatesstuff, is because we're using a questionable coding pattern we should probably change.I think it probably makes sense to work through those handling them individually. From a process perspective, I think we could work on running mypy in CI (with a longer than ideal exclude list) and in parallel work on fixing the errors here.
ashishk1994 commentedon Apr 7, 2016
@sharmaeklavya2 Thanks for the thorough investigation. Yes you are right we should pass an argument to lister.py to choose the type of exclusion strategy, It would help us in writing tools for travis-ci. What do you say @timabbott.
Also, I have tried to integrate mypy into travis ci (PR #639 ). Currently, mypy in travis is not throwing any error, once exclude list is decided i.e. all the errors from the mypy commands are gone then we can use it.
sharmaeklavya2 commentedon Apr 18, 2016
I tried to see how many files do we need to exclude from type checking to get no errors. I'm not checking files which do not have the .py extension because of #1380 in python/mypy. Out of the 254 .py files that we have, we'll only be checking 33 files (pretty sad).
Here is my exclude list:
There were also instances where excluding one file caused errors of the form
could not find module ..., so I had to exclude those files as well. I could have further refined this list by including only those files which were erring instead of including whole directories. But that would have made this list very long.sharmaeklavya2 commentedon Apr 18, 2016
Sorry I miscalculated. We're checking 194 files not 33. I had used
wcincorrectly.Modify .travis.yml to run mypyer using python 3.4.
Modify .travis.yml to run mypyer using python 3.4.
timabbott commentedon Apr 19, 2016
OK 194/254 is enough to be worth turning it on in CI; we can do the project of shrinking that list as a follow-on effort once we have this running in Travis CI.
Modify .travis.yml to run tools/travis/mypy using python 3.4.
Modify .travis.yml to run tools/travis/mypy using python 3.4.
Modify .travis.yml to run tools/travis/mypy using python 3.4.
Modify .travis.yml to run tools/travis/mypy using python 3.4.
Add tools/run-mypy to Travis checks.
Add tools/run-mypy to Travis checks.
Add tools/run-mypy to Travis checks.
Add tools/run-mypy to Travis checks.
Add tools/run-mypy to Travis checks.