Repository navigation
Async fixtures may break current event loop #868
Description
It seems pytest-asyncio (0.23.7) is currently badly broken with code relying on aiohttp.
This simple example currently breaks. The example is heavily simplified from the actual fixture setup. This used to work fine before 0.22.
import aiohttp
import pytest
# No longer working for getting a single event loop for aiohttp
# @pytest.fixture(scope="session")
# def event_loop():
# loop = asyncio.get_event_loop_policy().new_event_loop()
# try:
# yield loop
# finally:
# loop.close()
@pytest.fixture(scope="session")
async def client(
# event_loop <- accessing this no longer working. But neither this works without it
):
async with aiohttp.ClientSession(
# loop=event_loop <- aiohttp requires using a single event_loop that can not be closed between the tests...
) as session:
yield session
async def test_the_client(client: aiohttp.ClientSession):
await client.get("http://localhost:8080/foobar") # RuntimeError: Timeout context manager should be used inside a taskUsing asyncio_mode = "auto":
[tool.poetry]
name = "foobar"
package-mode = false
[tool.poetry.dependencies]
python = "~3.12"
aiohttp = "==3.9.5"
[tool.poetry.dev-dependencies]
pytest = "==8.2.2"
pytest-asyncio = "==0.23.7"
[tool.pytest.ini_options]
asyncio_mode = "auto"Reacted by Ed Ayers and Artem Ilin
Activity
MarkusSintonen commentedon Jul 2, 2024
Using this https://pytest-asyncio.readthedocs.io/en/latest/how-to-guides/run_session_tests_in_same_loop.html doesnt have any effect. Seems there are multiple issues now in
pytest-asyncio. There is no longer access to a single event loop used for tests. Alsopytest-asynciois now closing the event loop between the tests. How do I get back the previous working behaviour which is compatible eg withaiohttp? It is neither possible to change the session scoped client fixture to a function fixture. (Also it doesnt make sense, we want a single client per test session)MarkusSintonen commentedon Jul 2, 2024
There seems to be multiple open issues related to latest version of
pytest-asyncio. Couldn't find anything directly related to aiohttp usage.seifertm commentedon Jul 4, 2024
Thanks for the code example.
In your specific code, the issue is that the test and the fixture run in different asyncio event loops. Each pytest scope has its own event loop. When specifying a
@pytest.fixture(scope="session"), the fixture code runs in a session-scoped loop, whereastest_the_clientuses the default function-scoped loop.Adding
@pytest.mark.asyncio(scope="session")totest_the_clientmake the test code to run in the same session-scoped loop, thus eliminating the issue.However, there's a tightly linked know issue in pytest-asyncio, which makes it impossible to separate the scope of the event loop from the scope of the pytest fixture. Sometimes, it's necessary to have code that runs in a session-wide event loop, but that code should be re-executed for every function. This is not possible with v0.23 at the moment. If you require this functionality, you should pin pytest-asyncio to v0.21 until the next major release. This issue is tracked in #706.
As for why the code mentioned in the docs doesn't work, that's an entirely different topic that needs investigating.
seifertm commentedon Jul 4, 2024
When I add the following code snippet to
conftest.py(instead of@pytest.mark.asyncio(scope="session"as mentioned in my previous comment), your provided example runs as expected.I referred to the exact same documentation page as you did (see How to run all tests in the session in the same event loop).
Do you think you could provide another code example where the code snippet in the docs doesn't solve the issues you're experiencing?
MarkusSintonen commentedon Jul 8, 2024
@seifertm Thanks
It seems I was missing the
append=Falseparameter 🤔 Strange... It partially works with that. Btw could we instead have some config parameter to not have to add this boilerplate to every test suite (like withasyncio_modetoml param)?But even with this, I still have problems getting it working with more complex sets of fixtures. I was able to narrow the problems into a next weird case. It again breaks with
RuntimeError: Timeout context manager should be used inside a taskwhen adding an async-autouse fixture (to either to the test file or to the conftest):Full failing example:
The
pytest.mark.asyncio(scope="session")-thing in conftest doesnt help here... In real usage Im also seeingEvent loop is closederrors. But I wasnt yet able to narrow what exactly causes that other error yet. Lets first investigate the above issue with the autouse fixture...seifertm commentedon Jul 8, 2024
Thanks for the reproducer!
The autouse fixture seems to pull in the function-scoped
event_loopfixture as a dependency oftest_the_clientin addition to the session-scoped loop. I can verify this using pytest's--setup-showoutput:Since the
event_loopfixture is evaluated after the setup of the session-scopedclientfixture, the test ends up running in theevent_looploop.This should actually result in a
MultipleEventLoopsRequestedError, but it seems to be circumvented by the autouse mechanism. This is clearly a bug in pytest-asyncio.Until this is fixed, you can change the scope of
some_function_scope_autouse_async_fixturetoscope="session", if your test suite allows that.[-]pytest-asyncio unusable with code using `aiohttp`[/-][+]Async autouse fixture may break current event loop[/+]seifertm commentedon Jul 8, 2024
This feature has been requested a number of times and it's very reasonable. There are plans to add this kind of "default event loop scope" in #871
MarkusSintonen commentedon Jul 8, 2024
No that's not possible and won't make sense in the real usage. 😕 The example was greatly simplified.
11 remaining items
seifertm commentedon Jul 12, 2024
Just by looking at the diff, reverting to 0.23 seems to be quite the effort:
v0.21.x...main
It also means that we need to provide a migration path for users who already upgraded to v0.23. They would then upgrade to v0.21 behavior, only to have breaking changes again when going to the 1.0(-beta).
I'm not saying we should entirely rule out moving back to the v0.21 behavior and reverting the event loop fixture deprecation. (You're not the only proponent of that approach.) I'm just saying, we should rather patch the current main branch to get the old behavior, instead of backporting changes. If we decide to revert the event loop fixture deprecation, that is.
For now, I'm betting on #871 to solve the major issue with pytest-asyncio v0.23. If it turns out this is a bad idea as well, then pytest-asyncio should probably go back to v0.21 behavior.
If I have a pre-release with the PR merged, do you mind if I ping you as a reviewer/tester? I believe your input would be valuable.
MarkusSintonen commentedon Jul 12, 2024
Yes you can put me there as reviewer/tester!
I tried your branch there but still I'm getting the same errors 🤔 Used the new param also:
I dont really understand what the new param means. Why are fixtures somehow run in varying event loop scopes and also tests are somehow run in different event loops, or something. 🤔 (Tried to look from homepage docs but they dont explain the new behaviour either very clearly).
seifertm commentedon Jul 30, 2024
@MarkusSintonen As part of the pytest-asyncio 0.24.0a0 pre-release I updated the docs with how-to guides and a section explaining the new configuration setting. Do you think this is enough to make sense of it?
MarkusSintonen commentedon Aug 1, 2024
Not sure I understand what is the purpose of running fixtures and tests in different event loops. The docs dont explain it really. I can not either grasp how it changed from the previous working state. So why the new parameters are needed. Some migration guide would be probably needed also mentioning what is currently not working.
Lawouach commentedon Feb 26, 2025
I personally never had the need for this either. I'm assuming good faith from @seifertm and I suppose it was requested :)
That said, it's been a bad surprise when I did upgrade today and I had to revert back to 0.21.1 as well. I have tried all sorts of combination and none of them work. I always end up either with "Event loop was closed" or "another event loop with session scope is provided". I can't seem to find the right way to have a session level fixture and tests using that loop throughout the entire test session. I'm also failing to fully understand the documentation rationale for the change so I'm struggling to update my tests accordingly to match the new behavior. I'll pin to 0.21.1 for now :)
jefflieu commentedon Jul 22, 2025
I'm also trying to understand the mechanics behind this too
These 2 changes work for me, to get the old pytest-asyncio's behavior:
#1171
If you have answer please help point me to that.
Thank you
MarkusSintonen commentedon Aug 1, 2025
We were able to upgrade to
pytest-asyncio1.1.0by using the session-modes:So Im fine at closing the issue. Although the docs should be improved to explain what is the use on different event loops in tests.
gt-fm-4562 commentedon Aug 5, 2025
This seems to be some kind of a bug. Both
asyncio_default_fixture_loop_scopeandasyncio_default_test_loop_scopeseems to be ignored.Similar to @jefflieu the discussion #1171 does provide partial solution:
However, this applies only to tests not fixtures. In my case I also had multiple fixtures that relied on event_loop. Following solution worked for me:
brianredbeard commentedon Aug 13, 2025
After some review it seems that the issue was introduced in 78e93b8.
This patch introduced a significant refactoring that moved from manually
managing event loops to using the asyncio.Runner class. This change is the
source of the issue.
The previous implementation created scoped event loops and explicitly set the
correct loop as the current one for the thread using
_set_event_loop(). Thisensured that libraries like
aiohttp, which depend onasyncio.get_running_loop(), would always find the correct loop.The new implementation in this patch creates a scoped Runner instance. The
runner.run(coro)method executes the coroutine on the runner's internalloop, but it does not set that loop as the current one for the thread.
This creates a discrepancy when a test involves fixtures with different loop
scopes, as described above. The
autousefixture can cause one loopto be active, but when the test runs using its own session-scoped Runner, the
thread's "current loop" is not updated. Consequently, aiohttp's client,
which was created in the session-scoped loop, fails during its operations
because it cannot find its task on what it perceives as the current event loop,
triggering the RuntimeError.
seifertm commentedon Sep 17, 2025
Thanks for sharing @gt-fm-4562 . Pytest markers are only applicable to test items, not fixture.
You're saying that the configuration settings are ignored. If so, this is a separate bug. I'd appreciate if you could open a dedicated issue for this and include a minimal reproducer.
@brianredbeard I have to disagree with your assessment. The OP opened the issue in July 2024. The commit you're referencing as the root cause hasn't been introduced until June 2025. If you look at the CPython implementation of
asyncio.Runner, you will find thatasyncio.Runner.runcalls _lazy_init, which creates a new event loop and sets the event loop in the same way asasyncio.set_event_loop.That statement that you cannot mix different loop scopes in fixtures and/or tests is absolutely correct, though. We should improve the detection for this and give a better error message.
seifertm commentedon Jul 21, 2026
The actionable thing that remains from this issue is that users should get warnings when requesting fixtures with different loop scopes. I'll close this, because #1514 represents a dedicated issue for this topic.