bpo-32303: Consistency fixes for namespace loaders by warsaw · Pull Request #5481 · python/cpython

warsaw · 2018-02-01T20:14:47Z

This branch actually fixes two inconsistencies for namespace package loaders.

Namespace packages have inconsistent __loader__ and __spec__.loader. This inconsistency leads to namespace packages having a valid __loader__ but None for __spec__.loader. This branch fixes it by ensuring that the two attributes start out the same. (As always, a user could change one or the other, and no attempt is made by Python to keep them in sync.)
Namespace packages have inconsistent __file__ and __spec__.origin. Here, namespace packages have a __spec__.origin that is the literal string "namespace", and they do not have a __file__ attribute. Not only is this inconsistent, I believe it violates the documentation. The fix here is to set both __spec__.origin and __file__ to None for namespace packages.

I suppose it's possible that this will break existing code, but I'd argue that because current behavior runs counter to the documentation and makes no sense given the inconsistencies, it is better to fix them. I propose this change be applied to 3.7 and 3.6, although if you, my friendly reviewer, disagrees about 3.6, we can talk about it!

This isn't a security issue to back porting to 3.5 is off the table.

https://bugs.python.org/issue32303

…ckages

Also, repair a couple of tests.

brettcannon · 2018-02-02T18:22:54Z

Lib/test/test_importlib/util.py

        is_package=is_package)
    module.__spec__ = spec
-    module.__loader__ = loader
+    module.__loader__ = module.__loader__ = loader


Any specific reason to double-assign the same attribute?

Nope, good catch! I thought I needed that in a previous commit, but I don't, so removed!

warsaw · 2018-02-02T18:24:17Z

Doc/library/importlib.rst

   built-in modules and the filename for modules loaded from source.
   Normally "origin" should be set, but it may be ``None`` (the default)
-   which indicates it is unspecified.
+   which indicates it is unspecified (e.g. for namespace pacakges).


s/pacakges/packages/

bedevere-bot · 2018-02-02T20:16:01Z

@warsaw: Please replace # with GH- in the commit message next time. Thanks!

miss-islington · 2018-02-02T20:16:01Z

Thanks @warsaw for the PR 🌮🎉.. I'm working now to backport this PR to: 3.6, 3.7.
🐍🍒⛏🤖

bedevere-bot · 2018-02-02T20:16:16Z

GH-5498 is a backport of this pull request to the 3.7 branch.

* Make sure ``__spec__.loader`` matches ``__loader__`` for namespace packages. * Make sure ``__spec__.origin` matches ``__file__`` for namespace packages. https://bugs.python.org/issue32303 https://bugs.python.org/issue32305 (cherry picked from commit bbbcf86) Co-authored-by: Barry Warsaw <[email protected]>

miss-islington · 2018-02-02T20:17:05Z

Sorry, @warsaw, I could not cleanly backport this to 3.6 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker bbbcf8693b876daae4469765aa62f8924f39a7d2 3.6

* Make sure ``__spec__.loader`` matches ``__loader__`` for namespace packages. * Make sure ``__spec__.origin` matches ``__file__`` for namespace packages. https://bugs.python.org/issue32303 https://bugs.python.org/issue32305 (cherry picked from commit bbbcf86) Co-authored-by: Barry Warsaw <[email protected]>

* Make sure ``__spec__.loader`` matches ``__loader__`` for namespace packages. * Make sure ``__spec__.origin` matches ``__file__`` for namespace packages. https://bugs.python.org/issue32303 https://bugs.python.org/issue32305. (cherry picked from commit bbbcf86) Co-authored-by: Barry Warsaw <[email protected]>

bedevere-bot · 2018-02-03T01:04:31Z

GH-5504 is a backport of this pull request to the 3.6 branch.

…5504) * Make sure ``__spec__.loader`` matches ``__loader__`` for namespace packages. * Make sure ``__spec__.origin` matches ``__file__`` for namespace packages. https://bugs.python.org/issue32303 https://bugs.python.org/issue32305. (cherry picked from commit bbbcf86) Co-authored-by: Barry Warsaw <[email protected]>

…thonGH-5481) (python#5504)" This reverts commit a71397f.

…-5481) (#5504)" (#5911) This reverts commit a71397f.

…pace package ModuleSpec.origin was changed in python/cpython#5481 to be None for namespace packages.

Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. Now it's discovering the package and tests inside it correctly. If used in conjunction with =consider_namespace_packages= in config, the test modules get correct `__package__` and `__name__` attributes. Fixes: pytest-dev#478 Relevant: pytest-dev#10569 In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See python/cpython#5481 and https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations .

Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. Now it's discovering the package and tests inside it correctly. If used in conjunction with `consider_namespace_packages` in config, test modules get correct `__package__` and `__name__` attributes as well. Fixes: pytest-dev#478 Relevant: - pytest-dev#2371 - pytest-dev#10569 In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See python/cpython#5481 and https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations .

Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. Now it's discovering the package and tests inside it correctly. If used in conjunction with `consider_namespace_packages` in config, test modules get correct `__package__` and `__name__` attributes as well. Fixes: pytest-dev#478 Other relevant issues: - pytest-dev#2371 - pytest-dev#10569 In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See: - python/cpython#5481 - https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations .

Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. Now it's discovering the package and tests inside it correctly. If used in conjunction with `consider_namespace_packages` in config, test modules get correct `__package__` and `__name__` attributes as well. Fixes: pytest-dev#478 Other relevant issues: - pytest-dev#2371 - pytest-dev#10569 In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See: - python/cpython#5481 - https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations

…13426) Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. If used in conjunction with `consider_namespace_packages` in config, pytest discovers the package and tests inside it correctly. If used in conjunction with `consider_namespace_packages` in config, test modules get correct `__package__` and `__name__` attributes as well. In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See: - python/cpython#5481 - https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations Fixes #478 Fixes #2371 Fixes #10569

…ytest-dev#13426) Previously, when running `--pyargs pkg`, if you didn't have `pkg/__init__.py`, pytest would fail with `ERROR: module or package not found: pkg (missing __init__.py?)`. If used in conjunction with `consider_namespace_packages` in config, pytest discovers the package and tests inside it correctly. If used in conjunction with `consider_namespace_packages` in config, test modules get correct `__package__` and `__name__` attributes as well. In addition, remove `"namespace"` origin handling -- this value isn't used since python 3.8. See: - python/cpython#5481 - https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.submodule_search_locations Fixes pytest-dev#478 Fixes pytest-dev#2371 Fixes pytest-dev#10569

warsaw added 3 commits February 1, 2018 10:16

Make sure __spec__.loader matches __loader__ for namespace pa…

1c5a7b0

…ckages

Update the documentation

24ad3ef

Factor out the check for origin and has_location.

acaba0d

Also, repair a couple of tests.

warsaw added needs backport to 3.6 labels Feb 1, 2018

warsaw self-assigned this Feb 1, 2018

the-knights-who-say-ni added the CLA signed label Feb 1, 2018

bedevere-bot added the awaiting merge label Feb 1, 2018

brettcannon approved these changes Feb 2, 2018

View reviewed changes

warsaw commented Feb 2, 2018

View reviewed changes

warsaw added 2 commits February 2, 2018 14:46

Merge branch 'master' into nsloader

ad9a78d

Respond to review comments.

cd4c4ec

warsaw merged commit bbbcf86 into python:master Feb 2, 2018

bedevere-bot removed the awaiting merge label Feb 2, 2018

warsaw deleted the nsloader branch February 2, 2018 20:16

bedevere-bot removed the needs backport to 3.7 label Feb 2, 2018

bedevere-bot removed the needs backport to 3.6 label Feb 3, 2018

serhiy-storchaka changed the title ~~bpo-32303 - Consistency fixes for namespace loaders~~ bpo-32303: Consistency fixes for namespace loaders Feb 19, 2018

warsaw added a commit to warsaw/cpython that referenced this pull request Feb 26, 2018

Revert "[3.6] bpo-32303 - Consistency fixes for namespace loaders (py…

484f61c

…thonGH-5481) (python#5504)" This reverts commit a71397f.

warsaw mentioned this pull request Feb 26, 2018

bpo-32872 - Revert "[3.6] bpo-32303 - Consistency fixes for namespace loaders... #5911

Merged

warsaw added a commit that referenced this pull request Feb 26, 2018

Revert "[3.6] bpo-32303 - Consistency fixes for namespace loaders (GH…

86ea851

…-5481) (#5504)" (#5911) This reverts commit a71397f.

PCManticore added a commit to pylint-dev/astroid that referenced this pull request Jun 4, 2018

Check if the ModuleSpec origin is None to figure out if it is a names…

a4996b4

…pace package ModuleSpec.origin was changed in python/cpython#5481 to be None for namespace packages.

geryogam mentioned this pull request Jun 20, 2019

bpo-37800: Clean up importlib documentation for some module attributes #10016

Merged

jacobtylerwalls mentioned this pull request May 4, 2022

Require Python 3.7.2 pylint-dev/astroid#1542

Merged

karlicoss mentioned this pull request May 16, 2025

Support PEP420 (implicit namespace packages) as --pyargs target. pytest-dev/pytest#13426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comments

bpo-32303: Consistency fixes for namespace loaders#5481

bpo-32303: Consistency fixes for namespace loaders#5481
warsaw merged 5 commits intopython:masterfrom
warsaw:nsloader

warsaw commented Feb 1, 2018 •

edited

Loading

Uh oh!

brettcannon Feb 2, 2018

Uh oh!

warsaw Feb 2, 2018

Uh oh!

warsaw Feb 2, 2018

Uh oh!

bedevere-bot commented Feb 2, 2018

Uh oh!

miss-islington commented Feb 2, 2018

Uh oh!

bedevere-bot commented Feb 2, 2018

Uh oh!

miss-islington commented Feb 2, 2018

Uh oh!

bedevere-bot commented Feb 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Comments

Conversation

warsaw commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brettcannon Feb 2, 2018

Choose a reason for hiding this comment

Uh oh!

warsaw Feb 2, 2018

Choose a reason for hiding this comment

Uh oh!

warsaw Feb 2, 2018

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Feb 2, 2018

Uh oh!

miss-islington commented Feb 2, 2018

Uh oh!

bedevere-bot commented Feb 2, 2018

Uh oh!

miss-islington commented Feb 2, 2018

Uh oh!

bedevere-bot commented Feb 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

warsaw commented Feb 1, 2018 •

edited

Loading