bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython

asvetlov · 2019-06-24T15:30:15Z

The resurrection of #13630
Now without dangling threads in tests :)

https://bugs.python.org/issue35621

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

asvetlov · 2019-06-24T15:38:22Z

I tested it with manual adding 1-second timeout in spawned waiter threads to emulate slow test machine.
After making sure that ./python -Werror -m test -v --fail-env-changed test_asyncio doesn't produce an error I removed the sleep.

asvetlov · 2019-06-24T15:43:13Z

Lib/asyncio/unix_events.py

    def add_child_handler(self, pid, callback, *args):
        assert self._forks, "Must use the context manager"

-        if self._loop is None:


The check is replaced by watcher.is_active() call in subprocess transport

1st1

LGTM overall. Thanks for pushing this through, Andrew.

Doc/library/asyncio-policy.rst

Doc/library/asyncio-subprocess.rst

Lib/asyncio/unix_events.py

Co-Authored-By: Yury Selivanov <[email protected]>

asvetlov · 2019-06-30T09:00:21Z

Thanks for review!

miss-islington · 2019-06-30T09:55:02Z

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

bedevere-bot · 2019-06-30T09:55:16Z

GH-14484 is a backport of this pull request to the 3.8 branch.

…ted in non-main thread (pythonGH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

…ted in non-main thread (pythonGH-14344)

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <[email protected]>

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <[email protected]> (cherry picked from commit 7080a8b)

With Python 3.7 asynchronous tests randomly fail with "RuntimeError: Event loop is closed" during shutdown. Backtrace of the error includes `SafeChildWatcher` calls. Python 3.8 has replaced `SafeChildWatcher` with a new `ThreadedChildWatcher` by default [1] as a bugfix for "asyncio.create_subprocess_exec() only works with main event loop" bug [2]. Python 3.7 scheduled end of life is 2023-06-27 according to <https://devguide.python.org/versions/>. [1] python/cpython#14344 [2] https://bugs.python.org/issue35621

asvetlov added 7 commits June 4, 2019 02:32

Close pid waiters on watcher.close()

3695efd

Revert "Revert "bpo-35621: Support running subprocesses in asyncio wh…

3559581

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

Merge branch 'dangling-threads-in-watcher' into threaded-waiter

b18e6b9

Merge remote-tracking branch 'upstream/master' into threaded-waiter

1bc1d8e

Work on

ef1e21c

Merge branch 'master' into threaded-waiter

c982e4a

Fix dangling thread warnings in tests

6912ec7

asvetlov requested review from 1st1 and gpshead as code owners June 24, 2019 15:30

the-knights-who-say-ni added the CLA signed label Jun 24, 2019

bedevere-bot added the awaiting core review label Jun 24, 2019

asvetlov commented Jun 24, 2019

View reviewed changes

Update documentation

efb19c0

asvetlov added topic-asyncio needs backport to 3.8 labels Jun 24, 2019

Add docstrings

eb07403

1st1 approved these changes Jun 30, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jun 30, 2019

asvetlov and others added 8 commits June 30, 2019 11:50

Update Doc/library/asyncio-policy.rst

6aa3357

Co-Authored-By: Yury Selivanov <[email protected]>

Fix doc text

25d9ac2

Update Doc/library/asyncio-policy.rst

ae3a706

Co-Authored-By: Yury Selivanov <[email protected]>

Update Doc/library/asyncio-policy.rst

8baf48c

Co-Authored-By: Yury Selivanov <[email protected]>

Update Doc/library/asyncio-policy.rst

6e6a783

Co-Authored-By: Yury Selivanov <[email protected]>

Update Lib/asyncio/unix_events.py

1e1281a

Co-Authored-By: Yury Selivanov <[email protected]>

Update Lib/asyncio/unix_events.py

936aeb0

Co-Authored-By: Yury Selivanov <[email protected]>

Update docs

cc5a752

asvetlov merged commit 0d671c0 into python:master Jun 30, 2019

bedevere-bot removed the awaiting merge label Jun 30, 2019

asvetlov deleted the threaded-waiter branch June 30, 2019 09:55

bedevere-bot removed the needs backport to 3.8 label Jun 30, 2019

miss-islington added a commit that referenced this pull request Jun 30, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

bf8cb31

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <[email protected]>

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

af4d7c9

…ted in non-main thread (pythonGH-14344)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-35621: Support running subprocesses in asyncio when loop is execu…

b340eb2

…ted in non-main thread (pythonGH-14344)

webknjaz mentioned this pull request Jan 25, 2021

pytest-xdist breaks asyncio-based code expecting to be run in the main thread pytest-dev/pytest-xdist#620

Closed

webknjaz mentioned this pull request Jan 25, 2021

Re-introduce pytest-xdist in supported envs aio-libs/aiohttp#5431

Merged

5 tasks

link2xt mentioned this pull request May 19, 2023

"Event loop is closed" error in _provide_clean_event_loop() pytest-dev/pytest-asyncio#548

Closed

link2xt mentioned this pull request May 19, 2023

ci: require Python 3.8 for deltachat-rpc-client chatmail/core#4410

Merged

gvanrossum mentioned this pull request Oct 15, 2023

test_asyncio.test_subprocess leaked dangling threads on PPC64LE Fedora Stable 3.x: #110205

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comments

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344
asvetlov merged 17 commits intopython:masterfrom
asvetlov:threaded-waiter

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot

Loading

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlov Jun 24, 2019

Uh oh!

1st1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Comments

Conversation

asvetlov commented Jun 24, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlov Jun 24, 2019

Choose a reason for hiding this comment

Uh oh!

1st1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot

Loading