Skip to content

Fix code to restore sys.modules after running a handler#26111

Open
jgraham wants to merge 2 commits intomasterfrom
restore_sys_modules
Open

Fix code to restore sys.modules after running a handler#26111
jgraham wants to merge 2 commits intomasterfrom
restore_sys_modules

Conversation

@jgraham
Copy link
Contributor

@jgraham jgraham commented Oct 14, 2020

Replacing the whole dict doesn't work as expected, but we can remove
entries that weren't in the dict before the call. This fixes the code
to work in simple cases, but doesn't address possible race conditions
between handlers.

Replacing the whole dict doesn't work as expected, but we can remove
entries that weren't in the dict before the call. This fixes the code
to work in simple cases, but doesn't address possible race conditions
between handlers.
@Hexcles
Copy link
Member

Hexcles commented Oct 14, 2020

This seems to at least fix the original issue in Python 3:

./wpt --py3 run --log-mach=- --channel=beta chrome html/infrastructure/urls/resolving-urls/query-encoding/utf-8.html?include=css

now works without errors

@Hexcles
Copy link
Member

Hexcles commented Oct 14, 2020

@stephenmcgruer informed me that the issue doesn't reproduce reliably, so I tried a few more times, and sure enough after ~5 tries the issue reappeared... In other words, this doesn't really work :(

@Hexcles
Copy link
Member

Hexcles commented Oct 14, 2020

FWIW, this does fix the other issue that earlier relative imports shadow later ones with the same name (the new infra tests would fail without this fix), so it's probably still nice to have.

Tangent: the new infra tests fail consistently in Python 3 both with and without the change when doing from .utils import return_hello: KeyError: "'__name__' not in globals".

According to https://docs.python.org/3/reference/import.html#import-related-module-attributes

The name attribute must be set to the fully-qualified name of the module.

Might as well fix this by the way.

# https://docs.python.org/3.3/whatsnew/3.3.html#a-finer-grained-import-lock
if PY3:
for mod_name in list(sys.modules.keys()):
with importlib._bootstrap._ModuleLockManager(mod_name):
Copy link
Member

@Hexcles Hexcles Oct 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW https://github.com/python/cpython/blob/8b4642d3288e187faad24283c949ecf53fecad5b/Lib/importlib/_bootstrap.py#L3-L6

This module is NOT meant to be directly imported! It has been designed such
that it can be bootstrapped into Python as the implementation of import. As
such it requires the injection of specific modules and attributes in order to
work. One should use importlib as the public-facing version of this module.

@Hexcles
Copy link
Member

Hexcles commented Oct 29, 2020

FWIW, b7fa472 seems to fix the race condition when importing encoding. I can fairly confidently reason about it from the code, and I also tried many times to verify.

@stephenmcgruer stephenmcgruer removed their assignment Jan 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants