Repository navigation
Avoid sync deadlocks when the default IO thread has stopped - #2210
Conversation
| if sys.is_finalizing(): | ||
| return |
There was a problem hiding this comment.
The intent is to make sync() a no-op during shutdown here; so what happens at the call site - a normal exception?
There was a problem hiding this comment.
Good catch - the quiet return was wrong. A caller would get None and treat it as a real result: one wrapper hands None back to its caller, another would loop forever, and the HTTP code would crash on it later. The only place that needs quiet at shutdown is close_session. So now sync() always raises a clear RuntimeError when the IO thread is gone, close_session catches it and shuts down quietly, and the tests expect the raise in both cases.
| sync(loop, session.close, timeout=0.1) | ||
| return | ||
| except (TimeoutError, FSTimeoutError, NotImplementedError): | ||
| except (TimeoutError, FSTimeoutError, NotImplementedError, RuntimeError): |
There was a problem hiding this comment.
HTTP may be where we exhaustively test AsyncFileSystem, but there are other implementations in other repos (s3fs, gcsfs. adlfs and maybe other). Would they need a change?
There was a problem hiding this comment.
Good question - I only changed the HTTP one here, so I checked the others by reading their main branch. s3fs and gcsfs call sync() from close_session only when the loop reports running, but they catch just FSTimeoutError - so if the IO thread is gone but the loop still looks alive, they'd now see this RuntimeError instead of falling back to closing the connector. Worth them catching RuntimeError the same way. adlfs calls sync() directly from its weakref finalizers with no try/except, so at shutdown it would now raise from the finalizer instead of silently returning - a matching guard is a one-line follow-up in adlfs, not this PR. I haven't run any of them against this branch, so that's a reading of the code, not a test result.
| if loop is globals()["loop"][0] and iothread[0] is not None: | ||
| if not iothread[0].is_alive(): | ||
| raise RuntimeError("fsspec IO thread has already stopped.") |
There was a problem hiding this comment.
How about (loop._thread_id or 0) == iothread[0].ident? That would account for any supplied loop.
There was a problem hiding this comment.
Good direction - a supplied loop that lives on the fsspec IO thread should get the same guard. Applied in 9f79f44 with two adjustments, both verified locally:
- Kept the identity check as a disjunct. After a clean
loop.stop(),run_forever'sfinallyclearsloop._thread_idback to None, so a thread-id-only check misses exactly the stopped-default-loop case this PR is for (verified here:_thread_idequals the thread's ident while the loop runs, None after stop). getattr(loop, "_thread_id", 0)because uvloop loops don't carry_thread_idat all, so the attribute access would raise for them.
The check is now:
if iothread[0] is not None and (
loop is globals()["loop"][0]
or (getattr(loop, "_thread_id", 0) or 0) == iothread[0].ident
):Tests: added a case for a supplied loop whose _thread_id matches the stopped IO thread's ident (raises), and gave the custom-loop mock an ident. The dead-thread tests pass; the rest of test_async.py shows the same 5 environment failures as unpatched master (no aiohttp on this box), nothing new. Ruff format clean.
…heck Signed-off-by: Charan Rathore <[email protected]>
If the default fsspec IO thread has stopped, sync() can schedule a coroutine on its loop and wait forever. The loop can still be marked as running during interpreter shutdown.
Check the cached IO thread before creating or scheduling a coroutine. Raise RuntimeError during normal execution; return quietly during interpreter finalization so cleanup does not produce a traceback. The check only applies to fsspec's cached default loop. Custom loops and healthy IO threads still run work during finalization.
Tests cover the stopped thread, both finalization states, healthy-thread cleanup and custom loops. The new dead-thread tests fail on the default branch and pass with the fix. The async, spec, core and generic suites pass: 372 passed, 16 skipped, 1 xfailed. Ruff checks pass with the repository's pinned version.
As suggested by @martindurant in the issue, this guards on
sys.is_finalizing()rather than blocking all sync calls at shutdown, so cleanup that needs a healthy loop still runs.Fixes #1723.