Skip to content

Commit 7af3191

Browse files
Michael0x2aMichael0x2a
authored
Make ModuleFinder try identifying non-PEP 561 packages (#8238)
This pull request is an attempt at mitigating #4542 by making mypy report a custom error when it detects installed packages that are not PEP 561 compliant. (I don't think it resolves it though -- I've come to the conclusion that import handling is just inherently complex/spooky. So if you were in a cynical mode, you could perhaps argue the issue is just fundamentally unresolvable...) But anyways, this PR: 1. Removes the hard-coded list of "popular third party libraries" from `moduleinfo.py` and replaces it with a heuristic that tries to find when an import "plausibly matches" some directory or Python file while we search for packages containing ``py.typed``. If we do find a plausible match, we generate an error that looks something like this: ``` test.py:1: error: Skipping analyzing 'scipy': found module but no type hints or library stubs test.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports ``` The heuristic I'm using obviously isn't foolproof since we don't have any obvious signifiers like the ``py.typed`` file we can look for, but it seemed to work well enough when I tried testing in on some of the libraries in the old list. Hopefully this will result in less confusion when users use a mix of "popular" and "unpopular" libraries. 2. Gives us a way to add more fine-grained "module not found" error messages and heuristics in the future: we can add more entries to the ModuleNotFoundReason enum. 3. Updates the docs about missing imports to use these new errors. I added a new subsection per each error type to try and make things a little less unwieldy. 4. Adds what I think are common points of confusion to the doc -- e.g. that missing imports are inferred to be of type Any, what exactly it means to add a `# type: ignore`, and the whole virtualenv confusion thing. 5. Modifies the docs to more strongly discourage the use of MYPYPATH. Unless I'm wrong, it's not a feature most people will find useful. One limitation of this PR is that I added tests for just ModuleFinder. I didn't want to dive into modifying our testcases framework to support adding custom site-packages/some moral equivalent -- and my PR only changes the behavior of ModuleFinder when it would have originally reported something was not found, anyways.
1 parent 14ac8af commit 7af3191

30 files changed

Lines changed: 408 additions & 339 deletions

File tree

‎docs/source/running_mypy.rst‎

Lines changed: 135 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -126,72 +126,102 @@ sections will discuss what to do in the other two cases.
126126
.. _ignore-missing-imports:
127127

128128
Missing imports
129-
---------------
129+
***************
130130

131131
When you import a module, mypy may report that it is unable to
132132
follow the import.
133133

134-
This can cause a lot of errors that look like the following::
134+
This can cause errors that look like the following::
135135

136136
main.py:1: error: No library stub file for standard library module 'antigravity'
137-
main.py:2: error: No library stub file for module 'flask'
137+
main.py:2: error: Skipping analyzing 'django': found module but no type hints or library stubs
138138
main.py:3: error: Cannot find implementation or library stub for module named 'this_module_does_not_exist'
139139

140-
There are several different things you can try doing, depending on the exact
141-
nature of the module.
140+
If you get any of these errors on an import, mypy will assume the type of that
141+
module is ``Any``, the dynamic type. This means attempting to access any
142+
attribute of the module will automatically succeed:
142143

143-
If the module is a part of your own codebase, try:
144+
.. code-block:: python
144145
145-
1. Making sure your import does not contain a typo.
146-
2. Reading the :ref:`finding-imports` section below to make sure you
147-
understand how exactly mypy searches for and finds modules and modify
148-
how you're invoking mypy accordingly.
149-
3. Adding the directory containing that module to either the ``MYPYPATH``
150-
environment variable or the ``mypy_path``
151-
:ref:`config file option <config-file-import-discovery>`.
146+
# Error: Cannot find implementation or library stub for module named 'does_not_exist'
147+
import does_not_exist
152148
153-
Note: if the module you are trying to import is actually a *submodule* of
154-
some package, you should add the directory containing the *entire* package
155-
to ``MYPYPATH``. For example, suppose you are trying to add the module
156-
``foo.bar.baz``, which is located at ``~/foo-project/src/foo/bar/baz.py``.
157-
In this case, you should add ``~/foo-project/src`` to ``MYPYPATH``.
158-
159-
If the module is a third party library, you must make sure that there are
160-
type hints available for that library. Mypy by default will not attempt to
161-
infer the types of any 3rd party libraries you may have installed
149+
# But this type checks, and x will have type 'Any'
150+
x = does_not_exist.foobar()
151+
152+
The next three sections describe what each error means and recommended next steps.
153+
154+
Missing type hints for standard library module
155+
----------------------------------------------
156+
157+
If you are getting a "No library stub file for standard library module" error,
158+
this means that you are attempting to import something from the standard library
159+
which has not yet been annotated with type hints. In this case, try:
160+
161+
1. Updating mypy and re-running it. It's possible type hints for that corner
162+
of the standard library were added in a newer version of mypy.
163+
164+
2. Filing a bug report or submitting a pull request to
165+
`typeshed <https://github.com/python/typeshed>`_, the repository of type hints
166+
for the standard library that comes bundled with mypy.
167+
168+
Changes to typeshed will come bundled with mypy the next time it's released.
169+
In the meantime, you can add a ``# type: ignore`` to the import to suppress
170+
the errors generated on that line. After upgrading, run mypy with the
171+
:option:`--warn-unused-ignores <mypy --warn-unused-ignores>` flag to help you
172+
find any ``# type: ignore`` annotations you no longer need.
173+
174+
.. _missing-type-hints-for-third-party-library:
175+
176+
Missing type hints for third party library
177+
------------------------------------------
178+
179+
If you are getting a "Skipping analyzing X: found module but no type hints or library stubs",
180+
error, this means mypy was able to find the module you were importing, but no
181+
corresponding type hints.
182+
183+
Mypy will not try inferring the types of any 3rd party libraries you have installed
162184
unless they either have declared themselves to be
163185
:ref:`PEP 561 compliant stub package <installed-packages>` or have registered
164-
themselves on `typeshed <https://github.com/python/typeshed>`_,
165-
the repository of types for the standard library and some 3rd party libraries.
186+
themselves on `typeshed <https://github.com/python/typeshed>`_, the repository
187+
of types for the standard library and some 3rd party libraries.
166188

167-
If you are getting an import-related error, this means the library you
168-
are trying to use has done neither of these things. In that case, you can try:
189+
If you are getting this error, try:
169190

170-
1. Searching to see if there is a :ref:`PEP 561 compliant stub package <installed-packages>`.
191+
1. Upgrading the version of the library you're using, in case a newer version
192+
has started to include type hints.
193+
194+
2. Searching to see if there is a :ref:`PEP 561 compliant stub package <installed-packages>`.
171195
corresponding to your third party library. Stub packages let you install
172196
type hints independently from the library itself.
173197

174-
2. :ref:`Writing your own stub files <stub-files>` containing type hints for
198+
For example, if you want type hints for the ``django`` library, you can
199+
install the `django-stubs <https://pypi.org/project/django-stubs/>`_ package.
200+
201+
3. :ref:`Writing your own stub files <stub-files>` containing type hints for
175202
the library. You can point mypy at your type hints either by passing
176-
them in via the command line, by adding the location to the
177-
``MYPYPATH`` environment variable, or by using the ``mypy_path``
178-
:ref:`config file option <config-file-import-discovery>`.
203+
them in via the command line, by using the ``files`` or ``mypy_path``
204+
:ref:`config file options <config-file-import-discovery>`, or by
205+
adding the location to the ``MYPYPATH`` environment variable.
179206

180-
Note that if you decide to write your own stub files, they don't need
181-
to be complete! A good strategy is to add stubs for just the parts
182-
of the library you need and iterate on them over time.
207+
These stub files do not need to be complete! A good strategy is to use
208+
stubgen, a program that comes bundled with mypy, to generate a first
209+
rough draft of the stubs. You can then iterate on just the parts of the
210+
library you need.
183211

184212
If you want to share your work, you can try contributing your stubs back
185213
to the library -- see our documentation on creating
186214
:ref:`PEP 561 compliant packages <installed-packages>`.
187215

188-
If the module is a third party library, but you cannot find any existing
189-
type hints nor have time to write your own, you can *silence* the errors:
216+
If you are unable to find any existing type hints nor have time to write your
217+
own, you can instead *suppress* the errors. All this will do is make mypy stop
218+
reporting an error on the line containing the import: the imported module
219+
will continue to be of type ``Any``.
190220

191-
1. To silence a *single* missing import error, add a ``# type: ignore`` at the end of the
221+
1. To suppress a *single* missing import error, add a ``# type: ignore`` at the end of the
192222
line containing the import.
193223

194-
2. To silence *all* missing import imports errors from a single library, add
224+
2. To suppress *all* missing import imports errors from a single library, add
195225
a section to your :ref:`mypy config file <config-file>` for that library setting
196226
``ignore_missing_imports`` to True. For example, suppose your codebase
197227
makes heavy use of an (untyped) library named ``foobar``. You can silence
@@ -206,7 +236,7 @@ type hints nor have time to write your own, you can *silence* the errors:
206236
documentation about configuring
207237
:ref:`import discovery <config-file-import-discovery>` in config files.
208238

209-
3. To silence *all* missing import errors for *all* libraries in your codebase,
239+
3. To suppress *all* missing import errors for *all* libraries in your codebase,
210240
invoke mypy with the :option:`--ignore-missing-imports <mypy --ignore-missing-imports>` command line flag or set
211241
the ``ignore_missing_imports``
212242
:ref:`config file option <config-file-import-discovery>` to True
@@ -218,26 +248,59 @@ type hints nor have time to write your own, you can *silence* the errors:
218248
We recommend using this approach only as a last resort: it's equivalent
219249
to adding a ``# type: ignore`` to all unresolved imports in your codebase.
220250

221-
If the module is a part of the standard library, try:
251+
Unable to find module
252+
---------------------
222253

223-
1. Updating mypy and re-running it. It's possible type hints for that corner
224-
of the standard library were added in a later version of mypy.
254+
If you are getting a "Cannot find implementation or library stub for module"
255+
error, this means mypy was not able to find the module you are trying to
256+
import, whether it comes bundled with type hints or not. If you are getting
257+
this error, try:
225258

226-
2. Filing a bug report on `typeshed <https://github.com/python/typeshed>`_,
227-
the repository of type hints for the standard library that comes bundled
228-
with mypy. You can expedite this process by also submitting a pull request
229-
fixing the bug.
259+
1. Making sure your import does not contain a typo.
230260

231-
Changes to typeshed will come bundled with mypy the next time it's released.
232-
In the meantime, you can add a ``# type: ignore`` to silence any relevant
233-
errors. After upgrading, we recommend running mypy using the
234-
:option:`--warn-unused-ignores <mypy --warn-unused-ignores>` flag to help you find any ``# type: ignore``
235-
annotations you no longer need.
261+
2. If the module is a third party library, making sure that mypy is able
262+
to find the interpreter containing the installed library.
263+
264+
For example, if you are running your code in a virtualenv, make sure
265+
to install and use mypy within the virtualenv. Alternatively, if you
266+
want to use a globally installed mypy, set the
267+
:option:`--python-executable <mypy --python-executable>` command
268+
line flag to point the Python interpreter containing your installed
269+
third party packages.
270+
271+
2. Reading the :ref:`finding-imports` section below to make sure you
272+
understand how exactly mypy searches for and finds modules and modify
273+
how you're invoking mypy accordingly.
274+
275+
3. Directly specifying the directory containing the module you want to
276+
type check from the command line, by using the ``files`` or
277+
``mypy_path`` :ref:`config file options <config-file-import-discovery>`,
278+
or by using the ``MYPYPATH`` environment variable.
279+
280+
Note: if the module you are trying to import is actually a *submodule* of
281+
some package, you should specific the directory containing the *entire* package.
282+
For example, suppose you are trying to add the module ``foo.bar.baz``
283+
which is located at ``~/foo-project/src/foo/bar/baz.py``. In this case,
284+
you must run ``mypy ~/foo-project/src`` (or set the ``MYPYPATH`` to
285+
``~/foo-project/src``.
286+
287+
4. If you are using namespace packages -- packages which do not contain
288+
``__init__.py`` files within each subfolder -- using the
289+
:option:`--namespace-packages <mypy --namespace-packages>` command
290+
line flag.
291+
292+
In some rare cases, you may get the "Cannot find implementation or library
293+
stub for module" error even when the module is installed in your system.
294+
This can happen when the module is both missing type hints and is installed
295+
on your system in a unconventional way.
296+
297+
In this case, follow the steps above on how to handle
298+
:ref:`missing type hints in third party libraries <missing-type-hints-for-third-party-library>`.
236299

237300
.. _follow-imports:
238301

239302
Following imports
240-
-----------------
303+
*****************
241304

242305
Mypy is designed to :ref:`doggedly follow all imports <finding-imports>`,
243306
even if the imported module is not a file you explicitly wanted mypy to check.
@@ -401,3 +464,23 @@ same directory on the search path, only the stub file is used.
401464
(However, if the files are in different directories, the one found
402465
in the earlier directory is used.)
403466

467+
Other advice and best practices
468+
*******************************
469+
470+
There are multiple ways of telling mypy what files to type check, ranging
471+
from passing in command line arguments to using the ``files`` or ``mypy_path``
472+
:ref:`config file options <config-file-import-discovery>` to setting the
473+
``MYPYPATH`` environment variable.
474+
475+
However, in practice, it is usually sufficient to just use either
476+
command line arguments or the ``files`` config file option (the two
477+
are largely interchangeable).
478+
479+
Setting ``mypy_path``/``MYPYPATH`` is mostly useful in the case
480+
where you want to try running mypy against multiple distinct
481+
sets of files that happen to share some common dependencies.
482+
483+
For example, if you have multiple projects that happen to be
484+
using the same set of work-in-progress stubs, it could be
485+
convenient to just have your ``MYPYPATH`` point to a single
486+
directory containing the stubs.

‎mypy/build.py‎

Lines changed: 32 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,10 @@
4242
from mypy.report import Reports # Avoid unconditional slow import
4343
from mypy import moduleinfo
4444
from mypy.fixup import fixup_module
45-
from mypy.modulefinder import BuildSource, compute_search_paths, FindModuleCache, SearchPaths
45+
from mypy.modulefinder import (
46+
BuildSource, compute_search_paths, FindModuleCache, SearchPaths, ModuleSearchResult,
47+
ModuleNotFoundReason
48+
)
4649
from mypy.nodes import Expression
4750
from mypy.options import Options
4851
from mypy.parse import parse
@@ -2369,48 +2372,48 @@ def find_module_and_diagnose(manager: BuildManager,
23692372
# difference and just assume 'builtins' everywhere,
23702373
# which simplifies code.
23712374
file_id = '__builtin__'
2372-
path = find_module_simple(file_id, manager)
2373-
if path:
2375+
result = find_module_with_reason(file_id, manager)
2376+
if isinstance(result, str):
23742377
# For non-stubs, look at options.follow_imports:
23752378
# - normal (default) -> fully analyze
23762379
# - silent -> analyze but silence errors
23772380
# - skip -> don't analyze, make the type Any
23782381
follow_imports = options.follow_imports
23792382
if (root_source # Honor top-level modules
2380-
or (not path.endswith('.py') # Stubs are always normal
2383+
or (not result.endswith('.py') # Stubs are always normal
23812384
and not options.follow_imports_for_stubs) # except when they aren't
23822385
or id in mypy.semanal_main.core_modules): # core is always normal
23832386
follow_imports = 'normal'
23842387
if skip_diagnose:
23852388
pass
23862389
elif follow_imports == 'silent':
23872390
# Still import it, but silence non-blocker errors.
2388-
manager.log("Silencing %s (%s)" % (path, id))
2391+
manager.log("Silencing %s (%s)" % (result, id))
23892392
elif follow_imports == 'skip' or follow_imports == 'error':
23902393
# In 'error' mode, produce special error messages.
23912394
if id not in manager.missing_modules:
2392-
manager.log("Skipping %s (%s)" % (path, id))
2395+
manager.log("Skipping %s (%s)" % (result, id))
23932396
if follow_imports == 'error':
23942397
if ancestor_for:
2395-
skipping_ancestor(manager, id, path, ancestor_for)
2398+
skipping_ancestor(manager, id, result, ancestor_for)
23962399
else:
23972400
skipping_module(manager, caller_line, caller_state,
2398-
id, path)
2401+
id, result)
23992402
raise ModuleNotFound
24002403
if not manager.options.no_silence_site_packages:
24012404
for dir in manager.search_paths.package_path + manager.search_paths.typeshed_path:
2402-
if is_sub_path(path, dir):
2405+
if is_sub_path(result, dir):
24032406
# Silence errors in site-package dirs and typeshed
24042407
follow_imports = 'silent'
24052408
if (id in CORE_BUILTIN_MODULES
2406-
and not is_typeshed_file(path)
2409+
and not is_typeshed_file(result)
24072410
and not options.use_builtins_fixtures
24082411
and not options.custom_typeshed_dir):
24092412
raise CompileError([
2410-
'mypy: "%s" shadows library module "%s"' % (path, id),
2413+
'mypy: "%s" shadows library module "%s"' % (result, id),
24112414
'note: A user-defined top-level module with name "%s" is not supported' % id
24122415
])
2413-
return (path, follow_imports)
2416+
return (result, follow_imports)
24142417
else:
24152418
# Could not find a module. Typically the reason is a
24162419
# misspelled module name, missing stub, module not in
@@ -2419,7 +2422,7 @@ def find_module_and_diagnose(manager: BuildManager,
24192422
raise ModuleNotFound
24202423
if caller_state:
24212424
if not (options.ignore_missing_imports or in_partial_package(id, manager)):
2422-
module_not_found(manager, caller_line, caller_state, id)
2425+
module_not_found(manager, caller_line, caller_state, id, result)
24232426
raise ModuleNotFound
24242427
elif root_source:
24252428
# If we can't find a root source it's always fatal.
@@ -2456,10 +2459,17 @@ def exist_added_packages(suppressed: List[str],
24562459

24572460
def find_module_simple(id: str, manager: BuildManager) -> Optional[str]:
24582461
"""Find a filesystem path for module `id` or `None` if not found."""
2462+
x = find_module_with_reason(id, manager)
2463+
if isinstance(x, ModuleNotFoundReason):
2464+
return None
2465+
return x
2466+
2467+
2468+
def find_module_with_reason(id: str, manager: BuildManager) -> ModuleSearchResult:
2469+
"""Find a filesystem path for module `id` or the reason it can't be found."""
24592470
t0 = time.time()
24602471
x = manager.find_module_cache.find_module(id)
24612472
manager.add_stats(find_module_time=time.time() - t0, find_module_calls=1)
2462-
24632473
return x
24642474

24652475

@@ -2493,35 +2503,23 @@ def in_partial_package(id: str, manager: BuildManager) -> bool:
24932503

24942504

24952505
def module_not_found(manager: BuildManager, line: int, caller_state: State,
2496-
target: str) -> None:
2506+
target: str, reason: ModuleNotFoundReason) -> None:
24972507
errors = manager.errors
24982508
save_import_context = errors.import_context()
24992509
errors.set_import_context(caller_state.import_context)
25002510
errors.set_file(caller_state.xpath, caller_state.id)
2501-
stub_msg = "(Stub files are from https://github.com/python/typeshed)"
25022511
if target == 'builtins':
25032512
errors.report(line, 0, "Cannot find 'builtins' module. Typeshed appears broken!",
25042513
blocker=True)
25052514
errors.raise_error()
2506-
elif ((manager.options.python_version[0] == 2 and moduleinfo.is_py2_std_lib_module(target))
2507-
or (manager.options.python_version[0] >= 3
2508-
and moduleinfo.is_py3_std_lib_module(target))):
2509-
errors.report(
2510-
line, 0, "No library stub file for standard library module '{}'".format(target),
2511-
code=codes.IMPORT)
2512-
errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT)
2513-
elif moduleinfo.is_third_party_module(target):
2514-
errors.report(line, 0, "No library stub file for module '{}'".format(target),
2515-
code=codes.IMPORT)
2516-
errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT)
2515+
elif moduleinfo.is_std_lib_module(manager.options.python_version, target):
2516+
msg = "No library stub file for standard library module '{}'".format(target)
2517+
note = "(Stub files are from https://github.com/python/typeshed)"
2518+
errors.report(line, 0, msg, code=codes.IMPORT)
2519+
errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT)
25172520
else:
2518-
note = "See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports"
2519-
errors.report(
2520-
line,
2521-
0,
2522-
"Cannot find implementation or library stub for module named '{}'".format(target),
2523-
code=codes.IMPORT
2524-
)
2521+
msg, note = reason.error_message_templates()
2522+
errors.report(line, 0, msg.format(target), code=codes.IMPORT)
25252523
errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT)
25262524
errors.set_import_context(save_import_context)
25272525

0 commit comments

Comments
 (0)