Repository navigation
fix(sandbox): keep UnixLocal filesystem and tar work off the event loop - #4678
rajarshidattapy wants to merge 2 commits into
Conversation
`UnixLocalSandboxSession` called synchronous `pathlib` and `tarfile` APIs directly from `async def` methods, so the loop was held for the full duration of the I/O. `persist_workspace` and `hydrate_workspace` are the costly ones: both sit on the snapshot path and tar or extract the entire workspace tree, so a workspace holding a `node_modules` or a build directory stalled every other task on the loop for seconds to minutes. Move the blocking bodies into module-level helpers invoked through `asyncio.to_thread`, matching what the rest of this module already does for PTY reads and fd cleanup. Session state is read on the loop and only plain values cross into the worker thread. `_resolved_exec_context` now performs its existence check and resolves the workspace root in a single thread hop and returns that root, so the three exec paths no longer each repeat `Path(cwd).resolve()` for themselves. That removes the remaining loop-blocking syscalls and a redundant path walk per exec. Error types and messages are unchanged; the helpers keep the same guards and raise the same `WorkspaceArchiveReadError` / `WorkspaceArchiveWriteError` / `WorkspaceRootNotFoundError` as before.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17e8806a72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) from e | ||
| except (tarfile.TarError, OSError) as e: | ||
| raise WorkspaceArchiveWriteError(path=root, cause=e) from e | ||
| await asyncio.to_thread(_extract_workspace_archive, root, data) |
There was a problem hiding this comment.
Settle extraction before closing a cancelled snapshot
When cancellation interrupts a resumed LocalSnapshot while a large archive is restoring, cancelling this to_thread await does not stop _extract_workspace_archive; meanwhile restore_snapshot_into_workspace_on_resume() immediately closes workspace_archive in its finally. The worker then continues reading a closed stream or leaves partial files in the workspace after the caller has observed cancellation, so a retry or cleanup can race stale extraction. Keep the worker owned until it settles (and roll back/serialize the workspace mutation) before closing the archive or returning cancellation.
AGENTS.md reference: AGENTS.md:L149-L150
Useful? React with 👍 / 👎.
Review feedback: keep this change to the two operations with a demonstrated defect, and make cancellation ownership explicit. Revert the bounded sites. `_prepare_backend_workspace`, `_resolved_exec_context` and the three `Path(cwd).resolve()` callers go back to their original form. An ASYNC240 report alone does not establish a user-visible defect for them, and reverting keeps this change off the exec confinement path. Own the worker's lifetime. `asyncio.to_thread` cannot interrupt the thread it starts, so a cancelled caller previously returned while the worker was still walking or writing the workspace. Callers act on that return immediately: the snapshot lifecycle closes the archive stream in a `finally`, and resume clears the workspace root, so either would land on a live worker. `_run_workspace_io` now records the cancellation, waits for the worker to settle, and only then propagates it. Repeated cancellation is absorbed the same way, and a worker failure arriving after cancellation is consumed rather than reported. This follows `memory.sqlite_session._await_mutation`. Cancellation coverage is deterministic: three tests hold the worker open, cancel the task, and assert it has neither completed nor let the worker settle before the worker is released. All three fail against the plain `to_thread` version.
|
Thanks — both points taken, and the PR is updated in place. Scope narrowed. The bounded Cancellation ownership. This was a real hole in what I first pushed, not just a review preference —
Coverage is deterministic for both properties. Responsiveness: two tests hold the helper open until a coroutine scheduled beside it has run. Cancellation: three tests hold the worker open, cancel, and assert the task has neither completed nor let the worker settle before it is released. All three fail against a plain Verified on Linux, since the module does not import on Windows: |
|
Thanks for putting this together. The underlying #4675 issue is real, but I do not think this PR is a safe basis for the fix. The scope is also driven by |
This pull request fixes
persist_workspaceandhydrate_workspaceholding the asyncio event loop while they archive or extract the entire workspace tree.The bug
Both methods are
async defbut do their work with synchronoustarfileandpathlibcalls, and both sit on the snapshot lifecycle —persist_snapshot()awaitspersist_workspace(), andrestore_snapshot_into_workspace_on_resume()awaitshydrate_workspace():A workspace holding a
node_modules, a build directory, or a model checkpoint takes seconds to minutes to archive, and for that whole window nothing else on the loop runs: concurrent agent runs, streamed responses, MCP sessions, and tracing exports all freeze.The change
The blocking bodies move into two module-level helpers,
_archive_workspaceand_extract_workspace_archive, run on a worker thread. Session state is read on the loop —persist_workspacereadsmanifest.rootand_persist_workspace_skip_relpaths()itself — and only plain values cross into the thread.Cancellation ownership.
asyncio.to_threadcannot interrupt the thread it starts, so awaiting it plainly would report the operation as stopped while the worker is still walking or writing the workspace. Callers act on that return immediately: the snapshot lifecycle closes the archive stream in afinallyas soon as the await returns, and resume clears the workspace root. Either would land on a live worker._run_workspace_iotherefore records the cancellation, lets the worker settle, and only then propagates it. Repeated cancellation is absorbed the same way, and a worker failure that arrives after cancellation is consumed rather than surfacing instead of the cancellation. This followsmemory.sqlite_session._await_mutation, which owns cancellation for session writes; if you would rather hoist that helper into a shared util than have a second copy, say the word.Scope
Per review, this is limited to the two operations with a demonstrated defect. The bounded
mkdir,exists, andresolvecalls in_prepare_backend_workspace,_resolved_exec_context,_exec_internal,pty_exec_start, and_write_stream_with_execare unchanged, soruff --select ASYNCstill reportsASYNC240at those five sites. Leaving them alone also keeps this change off the exec confinement path.Behavior
WorkspaceArchiveReadErrorandWorkspaceArchiveWriteError, including theworkspace_root_not_foundreason.Tests
tests/sandbox/test_unix_local.pycovers both properties the review asked for.Event-loop responsiveness — two tests hold the archive/extract helper open until a coroutine scheduled alongside it has run, so doing that work on the loop deadlocks instead of passing.
Cancellation — three tests hold the worker open, cancel the task, and assert it has neither completed nor let the worker settle, then release the worker and assert
CancelledErrorsurfaces only afterwards. One covers repeated cancellation. All three fail against a plainawait asyncio.to_thread(...), onassert not task.done(). A fourth pins that a worker failure arriving after cancellation still reports as cancelled.Two behavior guards round it out: a persist-then-hydrate round trip, and a missing workspace root still raising
WorkspaceArchiveReadError.Because this module refuses to import on Windows by design, verification ran on Linux:
tests/sandboxis 1435 passed, 22 skipped, andmypyis clean on the module.Resolves #4675.