Skip to content

fix(sandbox): keep UnixLocal filesystem and tar work off the event loop - #4678

Closed
rajarshidattapy wants to merge 2 commits into
openai:mainfrom
rajarshidattapy:fix/unix-local-sandbox-blocking-io
Closed

rajarshidattapy wants to merge 2 commits into
openai:mainfrom
rajarshidattapy:fix/unix-local-sandbox-blocking-io

Conversation

@rajarshidattapy

@rajarshidattapy rajarshidattapy commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

This pull request fixes persist_workspace and hydrate_workspace holding the asyncio event loop while they archive or extract the entire workspace tree.

The bug

Both methods are async def but do their work with synchronous tarfile and pathlib calls, and both sit on the snapshot lifecycle — persist_snapshot() awaits persist_workspace(), and restore_snapshot_into_workspace_on_resume() awaits hydrate_workspace():

async def persist_workspace(self) -> io.IOBase:
    ...
    with tarfile.open(fileobj=buf, mode="w") as tar:
        tar.add(root, arcname=".", filter=...)   # walks and reads the whole tree

A workspace holding a node_modules, a build directory, or a model checkpoint takes seconds to minutes to archive, and for that whole window nothing else on the loop runs: concurrent agent runs, streamed responses, MCP sessions, and tracing exports all freeze.

The change

The blocking bodies move into two module-level helpers, _archive_workspace and _extract_workspace_archive, run on a worker thread. Session state is read on the loop — persist_workspace reads manifest.root and _persist_workspace_skip_relpaths() itself — and only plain values cross into the thread.

Cancellation ownership. asyncio.to_thread cannot interrupt the thread it starts, so awaiting it plainly would report the operation as stopped while the worker is still walking or writing the workspace. Callers act on that return immediately: the snapshot lifecycle closes the archive stream in a finally as soon as the await returns, and resume clears the workspace root. Either would land on a live worker.

_run_workspace_io therefore records the cancellation, lets the worker settle, and only then propagates it. Repeated cancellation is absorbed the same way, and a worker failure that arrives after cancellation is consumed rather than surfacing instead of the cancellation. This follows memory.sqlite_session._await_mutation, which owns cancellation for session writes; if you would rather hoist that helper into a shared util than have a second copy, say the word.

Scope

Per review, this is limited to the two operations with a demonstrated defect. The bounded mkdir, exists, and resolve calls in _prepare_backend_workspace, _resolved_exec_context, _exec_internal, pty_exec_start, and _write_stream_with_exec are unchanged, so ruff --select ASYNC still reports ASYNC240 at those five sites. Leaving them alone also keeps this change off the exec confinement path.

Behavior

  • Error types, messages, and context payloads are unchanged. The helpers keep the same guards and raise the same WorkspaceArchiveReadError and WorkspaceArchiveWriteError, including the workspace_root_not_found reason.
  • No public API change, and no change to any private signature outside these two methods.

Tests

tests/sandbox/test_unix_local.py covers both properties the review asked for.

Event-loop responsiveness — two tests hold the archive/extract helper open until a coroutine scheduled alongside it has run, so doing that work on the loop deadlocks instead of passing.

Cancellation — three tests hold the worker open, cancel the task, and assert it has neither completed nor let the worker settle, then release the worker and assert CancelledError surfaces only afterwards. One covers repeated cancellation. All three fail against a plain await asyncio.to_thread(...), on assert not task.done(). A fourth pins that a worker failure arriving after cancellation still reports as cancelled.

Two behavior guards round it out: a persist-then-hydrate round trip, and a missing workspace root still raising WorkspaceArchiveReadError.

Because this module refuses to import on Windows by design, verification ran on Linux: tests/sandbox is 1435 passed, 22 skipped, and mypy is clean on the module.

Resolves #4675.

`UnixLocalSandboxSession` called synchronous `pathlib` and `tarfile` APIs
directly from `async def` methods, so the loop was held for the full duration
of the I/O. `persist_workspace` and `hydrate_workspace` are the costly ones:
both sit on the snapshot path and tar or extract the entire workspace tree, so
a workspace holding a `node_modules` or a build directory stalled every other
task on the loop for seconds to minutes.

Move the blocking bodies into module-level helpers invoked through
`asyncio.to_thread`, matching what the rest of this module already does for
PTY reads and fd cleanup. Session state is read on the loop and only plain
values cross into the worker thread.

`_resolved_exec_context` now performs its existence check and resolves the
workspace root in a single thread hop and returns that root, so the three exec
paths no longer each repeat `Path(cwd).resolve()` for themselves. That removes
the remaining loop-blocking syscalls and a redundant path walk per exec.

Error types and messages are unchanged; the helpers keep the same guards and
raise the same `WorkspaceArchiveReadError` / `WorkspaceArchiveWriteError` /
`WorkspaceRootNotFoundError` as before.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17e8806a72

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

) from e
except (tarfile.TarError, OSError) as e:
raise WorkspaceArchiveWriteError(path=root, cause=e) from e
await asyncio.to_thread(_extract_workspace_archive, root, data)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Settle extraction before closing a cancelled snapshot

When cancellation interrupts a resumed LocalSnapshot while a large archive is restoring, cancelling this to_thread await does not stop _extract_workspace_archive; meanwhile restore_snapshot_into_workspace_on_resume() immediately closes workspace_archive in its finally. The worker then continues reading a closed stream or leaves partial files in the workspace after the caller has observed cancellation, so a retry or cleanup can race stale extraction. Keep the worker owned until it settles (and roll back/serialize the workspace mutation) before closing the archive or returning cancellation.

AGENTS.md reference: AGENTS.md:L149-L150

Useful? React with 👍 / 👎.

Review feedback: keep this change to the two operations with a demonstrated
defect, and make cancellation ownership explicit.

Revert the bounded sites. `_prepare_backend_workspace`, `_resolved_exec_context`
and the three `Path(cwd).resolve()` callers go back to their original form. An
ASYNC240 report alone does not establish a user-visible defect for them, and
reverting keeps this change off the exec confinement path.

Own the worker's lifetime. `asyncio.to_thread` cannot interrupt the thread it
starts, so a cancelled caller previously returned while the worker was still
walking or writing the workspace. Callers act on that return immediately: the
snapshot lifecycle closes the archive stream in a `finally`, and resume clears
the workspace root, so either would land on a live worker. `_run_workspace_io`
now records the cancellation, waits for the worker to settle, and only then
propagates it. Repeated cancellation is absorbed the same way, and a worker
failure arriving after cancellation is consumed rather than reported. This
follows `memory.sqlite_session._await_mutation`.

Cancellation coverage is deterministic: three tests hold the worker open, cancel
the task, and assert it has neither completed nor let the worker settle before
the worker is released. All three fail against the plain `to_thread` version.
@rajarshidattapy

Copy link
Copy Markdown
Contributor Author

Thanks — both points taken, and the PR is updated in place.

Scope narrowed. The bounded mkdir, exists, and resolve sites are reverted to their original form; only persist_workspace and hydrate_workspace change now. You are right that an ASYNC240 report alone did not establish a defect for them, and I have edited #4675 to say so rather than lump them in with the archive paths. Reverting also keeps this change off the exec confinement path, which is a better place for it not to be.

Cancellation ownership. This was a real hole in what I first pushed, not just a review preference — restore_snapshot_into_workspace_on_resume closes the archive stream in a finally as soon as the await returns, so a cancelled hydrate_workspace left a live worker reading a closed stream and writing into a workspace that resume may then clear.

_run_workspace_io now records the cancellation, waits for the worker to settle, and only then propagates it; repeated cancellation is absorbed the same way, and a worker failure arriving after cancellation is consumed so the caller still hears cancellation. It follows memory.sqlite_session._await_mutation. That does mean a second copy of that pattern in the tree — happy to hoist it into a shared util instead if you would prefer one home for it.

Coverage is deterministic for both properties. Responsiveness: two tests hold the helper open until a coroutine scheduled beside it has run. Cancellation: three tests hold the worker open, cancel, and assert the task has neither completed nor let the worker settle before it is released. All three fail against a plain await asyncio.to_thread(...) on assert not task.done(), so they pin the behavior rather than just exercising it.

Verified on Linux, since the module does not import on Windows: tests/sandbox is 1435 passed, 22 skipped, mypy clean.

@seratch

seratch commented Aug 26, 2026

Copy link
Copy Markdown
Member

Thanks for putting this together. The underlying #4675 issue is real, but I do not think this PR is a safe basis for the fix. asyncio.to_thread() does not stop its worker when the awaiting task is cancelled. In hydrate_workspace(), snapshot cleanup can close the archive and proceed into session shutdown or deletion while the worker is still extracting into the same workspace. The tests do not cover this cancellation and cleanup ordering.

The scope is also driven by ASYNC240 rather than the lifecycle boundary: it moves bounded mkdir and resolve calls but leaves other unbounded synchronous filesystem work, including recursive removal and streamed writes, on the event loop. Fixing this correctly requires a cancellation-safe blocking-I/O design rather than extending this patch. I am going to close #4678 while keeping #4675 open for that redesign.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnixLocalSandbox blocks the event loop with synchronous filesystem and tar I/O

2 participants