Skip to content

Avoid sync deadlocks when the default IO thread has stopped - #2210

Merged
martindurant merged 4 commits into
fsspec:masterfrom
charan-rathore:fix-sync-dead-io-thread
Oct 7, 2026
Merged

martindurant merged 4 commits into
fsspec:masterfrom
charan-rathore:fix-sync-dead-io-thread

Conversation

@charan-rathore

Copy link
Copy Markdown
Contributor

If the default fsspec IO thread has stopped, sync() can schedule a coroutine on its loop and wait forever. The loop can still be marked as running during interpreter shutdown.

Check the cached IO thread before creating or scheduling a coroutine. Raise RuntimeError during normal execution; return quietly during interpreter finalization so cleanup does not produce a traceback. The check only applies to fsspec's cached default loop. Custom loops and healthy IO threads still run work during finalization.

Tests cover the stopped thread, both finalization states, healthy-thread cleanup and custom loops. The new dead-thread tests fail on the default branch and pass with the fix. The async, spec, core and generic suites pass: 372 passed, 16 skipped, 1 xfailed. Ruff checks pass with the repository's pinned version.

As suggested by @martindurant in the issue, this guards on sys.is_finalizing() rather than blocking all sync calls at shutdown, so cleanup that needs a healthy loop still runs.

Fixes #1723.

Comment thread fsspec/asyn.py Outdated
Comment on lines +72 to +73
if sys.is_finalizing():
return

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent is to make sync() a no-op during shutdown here; so what happens at the call site - a normal exception?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - the quiet return was wrong. A caller would get None and treat it as a real result: one wrapper hands None back to its caller, another would loop forever, and the HTTP code would crash on it later. The only place that needs quiet at shutdown is close_session. So now sync() always raises a clear RuntimeError when the IO thread is gone, close_session catches it and shuts down quietly, and the tests expect the raise in both cases.

sync(loop, session.close, timeout=0.1)
return
except (TimeoutError, FSTimeoutError, NotImplementedError):
except (TimeoutError, FSTimeoutError, NotImplementedError, RuntimeError):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HTTP may be where we exhaustively test AsyncFileSystem, but there are other implementations in other repos (s3fs, gcsfs. adlfs and maybe other). Would they need a change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question - I only changed the HTTP one here, so I checked the others by reading their main branch. s3fs and gcsfs call sync() from close_session only when the loop reports running, but they catch just FSTimeoutError - so if the IO thread is gone but the loop still looks alive, they'd now see this RuntimeError instead of falling back to closing the connector. Worth them catching RuntimeError the same way. adlfs calls sync() directly from its weakref finalizers with no try/except, so at shutdown it would now raise from the finalizer instead of silently returning - a matching guard is a one-line follow-up in adlfs, not this PR. I haven't run any of them against this branch, so that's a reading of the code, not a test result.

Comment thread fsspec/asyn.py Outdated
Comment on lines +70 to +72
if loop is globals()["loop"][0] and iothread[0] is not None:
if not iothread[0].is_alive():
raise RuntimeError("fsspec IO thread has already stopped.")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about (loop._thread_id or 0) == iothread[0].ident? That would account for any supplied loop.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good direction - a supplied loop that lives on the fsspec IO thread should get the same guard. Applied in 9f79f44 with two adjustments, both verified locally:

  1. Kept the identity check as a disjunct. After a clean loop.stop(), run_forever's finally clears loop._thread_id back to None, so a thread-id-only check misses exactly the stopped-default-loop case this PR is for (verified here: _thread_id equals the thread's ident while the loop runs, None after stop).
  2. getattr(loop, "_thread_id", 0) because uvloop loops don't carry _thread_id at all, so the attribute access would raise for them.

The check is now:

if iothread[0] is not None and (
    loop is globals()["loop"][0]
    or (getattr(loop, "_thread_id", 0) or 0) == iothread[0].ident
):

Tests: added a case for a supplied loop whose _thread_id matches the stopped IO thread's ident (raises), and gave the custom-loop mock an ident. The dead-thread tests pass; the rest of test_async.py shows the same 5 environment failures as unpatched master (no aiohttp on this box), nothing new. Ruff format clean.

@martindurant
martindurant merged commit 29e62a5 into fsspec:master Oct 7, 2026
11 checks passed
@charan-rathore
charan-rathore deleted the fix-sync-dead-io-thread branch October 7, 2026 19:14
charan-rathore added a commit to charan-rathore/charan-rathore that referenced this pull request Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fsspec.asyn.sync shoud check the liveness of IO thread to avoid deadlocks

2 participants