bpo-42183: Fix a stack overflow error for asyncio Task or Future repr()#23020
bpo-42183: Fix a stack overflow error for asyncio Task or Future repr()#23020asvetlov merged 5 commits intopython:masterfrom
Conversation
The overflow occures under some circumstances when a task or future recursively returns itself.
There was a problem hiding this comment.
Other than a few minor comments, LGTM.
For comparison, I primarily referred to the implementation of reprlib.recursive_repr(). Based on my local testing, it addresses the issue of recursive reprs and fully prevents the stack overflow.
| return f'cb=[{cb}]' | ||
|
|
||
|
|
||
| _repr_running = set() |
There was a problem hiding this comment.
Might be worth adding a brief comment here to clarify the purpose; e.g. "Used for guarding against recursive reprs from futures" or something along those lines. The name _repr_running isn't overly clear by itself without knowing the context of the internal set in reprlib.recursive_repr().
There was a problem hiding this comment.
Sure, I'll add. Plus, I want to describe why reprlib.recursive_repr decorator is not applicable (mine first approach was just applying it).
There was a problem hiding this comment.
I would say that adding _asyncio.Task.__module__ allows us to use reprlib.recursive_repr but it is a more invasive change.
| async def func(): | ||
| return asyncio.all_tasks() | ||
|
|
||
| self.assertIn('...', repr(await asyncio.wait_for(func(), timeout=10))) |
There was a problem hiding this comment.
Should this not more strictly test result=...? I think they would be effectively the same in most cases, but during local testing of the PR branch, I noticed some issues with regards to the ellipses colliding/cutting off parts of the repr when using asyncio.gather():
>>> await asyncio.gather(
... asyncio.wait_for(func(), timeout=10),
... asyncio.wait_for(func(), timeout=10))
[{<Task finished name='Task-18' coro=<wait_for() done, defined at /home/aeros/repos/cpython/Lib/asyncio/tasks.py:419> result={<Task finishe...9> result=...>, <Task finishe...res.py:391]>}>, <Task finishe...res.py:391]>}>, <Task pending...tures.py:391]>, <Task finishe... result=...>}>}>, <Task finished name='Task-21' coro=<func() done, defined at <console>:1> result={<Task finishe...res.py:391]>}>, <Task finishe...esult=...>}>}>, <Task finishe...1> result=...>, <Task pending...tures.py:391]>}>, <Task finished name='Task-19' coro=<wait_for() done, defined at /home/aeros/repos/cpython/Lib/asyncio/tasks.py:419> result={<Task finishe...9> result=...>, <Task finishe...esult=...>}>}>, <Task finishe...res.py:391]>}>, <Task pending...tures.py:391]>}>, <Task pending name='Task-17' coro=<<module>() running at <console>:1> cb=[_chain_future.<locals>._call_set_state() at /home/aeros/repos/cpython/Lib/asyncio/futures.py:391]>, <Task finished name='Task-20' coro=<func() done, defined at <console>:1> result={<Task finishe... result=...>}>, <Task finishe...res.py:391]>}>, <Task finishe...res.py:391]>}>, <Task pending...tures.py:391]>, <Task finishe...1> result=...>}>}, {<Task finished name='Task-19' coro=<wait_for() done, defined at /home/aeros/repos/cpython/Lib/asyncio/tasks.py:419> result={<Task finishe...9> result=...>, <Task finishe...esult=...>}>}>, <Task finishe...res.py:391]>}>, <Task pending...tures.py:391]>}>, <Task finished name='Task-18' coro=<wait_for() done, defined at /home/aeros/repos/cpython/Lib/asyncio/tasks.py:419> result={<Task finishe...9> result=...>, <Task finishe...res.py:391]>}>, <Task finishe...res.py:391]>}>, <Task pending...tures.py:391]>, <Task finishe... result=...>}>}>, <Task finished name='Task-21' coro=<func() done, defined at <console>:1> result={<Task finishe...res.py:391]>}>, <Task finishe...esult=...>}>}>, <Task finishe...1> result=...>, <Task pending...tures.py:391]>}>, <Task pending name='Task-17' coro=<<module>() running at <console>:1> cb=[_chain_future.<locals>._call_set_state() at /home/aeros/repos/cpython/Lib/asyncio/futures.py:391]>}]If it's a bit hard to read, this was the problematic line:
<Task finished name='Task-21' coro=<func() done, defined at <console>:1> result={<Task finishe...res.py:391]>}>, <Task finishe...esult=...>
That might just be an issue with asyncio.gather() reprs though, so I'm not sure exactly what the fix would be.
There was a problem hiding this comment.
The main thing that the test check is the code doesn't raise RecursionError.
Probably I should mention it in a comment for the call.
I'm open to suggestions but not I don't see a better test check.
A comparison of the whole repr() string is weak.
It will be broken on any repr() change, e.g. providing info about an additional task's internal state.
I agree that the formatting is not always tidy. From my understanding, it is the result of two strategies: preventing recursion calls and limiting the repr string size.
Maybe we can improve this but I consider it as a separate issue.
There was a problem hiding this comment.
My idea was to use self.assertIn('result=...', repr(await asyncio.wait_for(func(), timeout=10))) rather than matchiing the whole repr() string, but I agree that the repr formatting is more of a separate issue (and lower priority than what the PR mainly addresses).
| @@ -0,0 +1,15 @@ | |||
| # IsolatedAsyncioTestCase based tests | |||
There was a problem hiding this comment.
Is the motivation here to start a new set of tests using IsolatedAsyncioTestCase for simplification? If so, is there going to be some effort to gradually migrate the existing test_futures or is it just for new ones?
There was a problem hiding this comment.
Yes, the gradual migration of the asyncio test suite would be nice.
I believe IsolatedAsyncioTestCase based tests are much more readable.
On another hand, don't fix if not broken :) Plus tests rewriting takes time with very little effort.
For a new test, I prefer the new approach, definitely.
Misc/NEWS.d/next/Library/2020-10-29-11-17-35.bpo-42183.50ZcIi.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Kyle Stanley <[email protected]>
|
With the 3.7 branch now being security only, I believe the |
|
I'm not sure if |
|
Comments are added |
It doesn't sound like something that a third-party could use to cause a DoS but, If you and @1st1 think it should be fixed, I would likely not object to backporting to 3.7. |
|
Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9. |
…() (pythonGH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]> (cherry picked from commit 42d873c) Co-authored-by: Andrew Svetlov <[email protected]>
|
GH-23221 is a backport of this pull request to the 3.9 branch. |
…() (pythonGH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]> (cherry picked from commit 42d873c) Co-authored-by: Andrew Svetlov <[email protected]>
|
GH-23222 is a backport of this pull request to the 3.8 branch. |
|
GH-23223 is a backport of this pull request to the 3.7 branch. |
…() (pythonGH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]> (cherry picked from commit 42d873c) Co-authored-by: Andrew Svetlov <[email protected]>
…() (GH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]> (cherry picked from commit 42d873c) Co-authored-by: Andrew Svetlov <[email protected]>
…() (GH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]> (cherry picked from commit 42d873c) Co-authored-by: Andrew Svetlov <[email protected]>
…() (pythonGH-23020) The overflow occurs under some circumstances when a task or future recursively returns itself. Co-authored-by: Kyle Stanley <[email protected]>
The overflow occurs under some circumstances when a task or future
recursively returns itself.
https://bugs.python.org/issue42183