Skip to content

Commit 0ad2e44

Browse files
Python: fix: make concurrent FileCheckpointStorage saves not race on a shared temp path (microsoft#7757)
* Python: fix: make concurrent FileCheckpointStorage saves not race on a shared temp path FileCheckpointStorage.save() wrote to and then renamed a fixed "<checkpoint-id>.json.tmp" path, so concurrent saves of the same checkpoint ID raced over the shared temp file. Whichever save renamed first removed the temp file still being written by another save, which then failed with FileNotFoundError / PermissionError in os.replace. Create a unique temp file per save in the destination directory (so os.replace remains atomic), serialize same-ID writes with a per-ID lock, and retry the atomic move briefly to absorb the transient Windows background-handle PermissionError that surfaces even for fully serialized replaces. Fixes microsoft#7748 * Python: fix: lock file writes at destination path, shield workers from cancellation Address two post-merge review concerns on FileCheckpointStorage.save(): - The per-event-loop, per-instance asyncio.Lock registry could not serialize two FileCheckpointStorage instances pointed at the same directory, nor a single instance driven from two event loops. Replace it with a process-wide, per-canonical-path threading.Lock registry (lazily populated, bounded by distinct destinations actually written). Because asyncio.to_thread runs the actual file I/O on a worker thread, the threading lock serializes correctly across coroutines, loops, and instances and spans the entire open + write + os.replace window. - Caller-side cancellation previously released the write lock early: a CancelledError delivered inside 'await asyncio.to_thread(...)' exited the 'async with' and ran _release_save_lock_ref while the OS thread kept writing. The next save for the same checkpoint_id could then reach os.replace concurrently, reintroducing the PermissionError race and possibly landing stale data over a newer write. Shield the worker so cancellation propagates only after the in-flight write completes. Also replace the refcount-aware registry test with: - registry-bounded test for the new invariant (one lock per destination) - cross-instance concurrent save test (two storages, same directory) - gated os.replace cancel-race test (deterministically parks A's worker mid-publish, confirms save B cannot reach os.replace until A completes) Fixes microsoft#7748 * Python: take checkpoint destination ownership before submitting the write Implements the revised A shape chosen in review: process-wide coordination keyed by canonical destination, asynchronous waiters, and registry entries released after the final queued or running operation. Three review findings shared one cause -- ownership was established inside the executor thread rather than before submission. Acquiring the destination lock inside the worker meant every same-path save occupied an `asyncio.to_thread` worker while merely waiting, so a burst could fill the default pool and stall unrelated work including checkpoint loads, and could deadlock once the write that had to finish first was queued behind those waiters. Ownership now comes from a queue keyed by the canonical path and is taken before the write is submitted, so only the active write holds a worker. Waiters suspend on `asyncio.wrap_future` over a `concurrent.futures.Future`, which is not bound to a loop, so one chain also orders writers enqueued from different event loops -- something a per-path lock could not do. The registry was a dict that never removed an entry. Because `WorkflowCheckpoint` generates a fresh UUID by default, every save retained one entry for the life of the process. Entries are now reference-counted by queued-or-running operations and dropped by the last release. Cancellation is handled explicitly, both ways the review allowed. A save cancelled while still waiting for the destination has nothing in the executor, so it is simply removed -- and it still hands ownership on, or every later save for that path would wait forever. A save cancelled mid-write drains its worker before propagating, because releasing first would let the next save start while this replace is still in flight. That drain absorbs re-delivered cancellations: once a task has a cancellation pending its next await raises immediately, so a single shielded await returns with the worker still running and the write can still land after ownership is released. Ownership is released as the write's last act on the worker thread, with the coroutine's `finally` as a backstop for the case where no write was submitted. Releasing only from the coroutine regressed on the design it replaced: a `threading.Lock` was released by the worker thread, which outlives the loop that submitted it, whereas a coroutine that never resumes -- its loop closed with the task still pending -- would leave the destination owned for the life of the process and hang every later save to it. The two releasers are idempotent, so exactly one of them accounts for each ticket. Tests are deterministic and cover each case: registry release for repeated, distinct-ID and concurrent saves; queued saves not occupying executor threads, gated by a two-worker executor with three saves in flight; serialization across two real event loops; cancellation before submission writing nothing and not stalling the follower; repeated cancellation still draining; a write failing while draining being reported rather than lost; a failed save releasing its destination and leaving the path usable; recovery of a destination whose loop was closed mid-write; and the defensive release path. * Python: tie checkpoint write ownership to the write, not to asyncio state Follow-up review on microsoft#7757 found two ways a later save could still overtake an earlier one. Both had the same cause: asyncio objects were standing in for the state of work that lives outside asyncio, and cancellation can mark those done while the work continues. `asyncio.wrap_future` chains cancellation into the future it wraps. Cancelling the middle of three queued saves therefore cancelled the hand-off signal the first save still had to resolve, and the unconditional release then let the third reach `os.replace` beside the first -- after which the first write could land last and overwrite the newer checkpoint. Waiting now goes through `_wait_for_signal`, which feeds a fresh per-wait future from a done-callback and leaves the shared signal untouched no matter what happens to the waiter. A save cancelled while queued no longer resolves its own signal either; it defers the hand-off until its predecessor has actually finished, so the queue stays ordered while still guaranteeing nothing waits behind it forever. Draining on the task wrapping `asyncio.to_thread` was the second: a cancelled task reports `done()` while its function is still inside `os.replace`, so loop shutdown released ownership with the write in flight. The drain now waits on the signal the worker thread resolves, which cancellation cannot mark done. Fixing those exposed two more failures of the same kind, both found by probing rather than by review. `ensure_future(asyncio.to_thread(...))` makes the write a Task, and shutdown cancels every task -- possibly before it reaches the executor. Nothing then released the destination, and the drain waited for a signal that could never be resolved, so the save never completed at all. The write is now submitted with `run_in_executor`, which submits synchronously and returns a plain Future that `asyncio.all_tasks()` does not include, plus a done-callback that releases on success, failure and cancellation alike. Deferring a cancelled ticket's hand-off onto its predecessor made release recursive: resolving one signal ran the next ticket's callback synchronously, so a run of cancelled queued saves longer than the recursion limit raised `RecursionError` on the worker thread partway through and left the destination owned for good. Releases are now collected per thread and drained in a loop, so the depth is flat however long the run. Two smaller points from the same review. A write that fails while draining is recorded on the ticket rather than read back from the worker future, because a cancelled task hides its own exception and the cancelled caller never sees it. And the drain registers one done-callback pointing at whichever waiter is current, rather than one per wait, so a caller cancelling in a loop no longer grows that list in step with it. Tests: cancelling the middle of three queued saves cannot let the third overtake; a cancelled worker task does not release while the thread writes; shutdown before the write starts neither hangs nor strands the destination, asserted structurally on the write not being a task so the regression fails cleanly instead of hanging CI; a deferred-release run of 2000 links does not recurse; submission against a dead executor releases; and resolving a signal after its loop closed does not raise. * Python: release a queued checkpoint save whose event loop is abandoned Follow-up review on microsoft#7757. A submitted write is released by its worker thread, which outlives the loop that submitted it, but a ticket still waiting for its predecessor has no worker to fall back on. If that waiter's loop closes, `call_soon_threadsafe` raises `RuntimeError`, the suppression swallowed it, and the coroutine never resumed -- so nothing resolved the ticket's hand-off signal and every later save for that destination waited forever. Reproduced: the entry stays at `pending=1` and a later save from a fresh loop times out. The predecessor's callback now owns that case. `_wait_for_signal` takes an `on_abandoned` hook, invoked when the waiter's loop has closed, and the queued wait passes its own release -- performed on the thread that resolved the predecessor rather than on a loop that will never run again. The drain keeps suppressing, and the two guards now differ deliberately: by the time anything drains, the write has been submitted, so the worker thread and the submitted future's callback both release without needing that loop. The comment says so, since the asymmetry is otherwise unexplained. Two windows this cannot close, both stated in the comment rather than implied: a loop that closes after `call_soon_threadsafe` succeeded but before the callback runs, and one abandoned without being closed at all. Reaching either needs a loop closed with tasks still pending, which asyncio already reports as an error. Graceful shutdown is unaffected -- `asyncio.run` cancels pending tasks first, which drives a queued save through its cancellation path and defers the hand-off onto its predecessor. That path now has its own test, since it is the guarantee callers actually depend on. * Python: cover the checkpoint write error paths this PR introduced Three branches added by this PR had no test. Coverage of `_checkpoint.py` was reporting 97% with the gaps sitting exactly on the error handling: * the bounded `os.replace` retry giving up after five attempts. The suite only ever reached the surrounding lines by accident -- the concurrency tests trip a real `PermissionError` on Windows now and then, which is environment-dependent and never happens on Linux CI. The new test pins both halves that matter: the retry is bounded, and giving up still releases the destination rather than wedging every later save to it. * the temp-file cleanup swallowing an `OSError`. It runs in a `finally` while a write exception is propagating, so letting one out would substitute a misleading "cannot remove temp file" for the real disk failure. Reverting the handler makes the new test fail with exactly that substitution. * the fast path in `_await_signal_through_cancellation` for a signal that has already resolved. Exercised directly, since arranging the race through `save()` is not deterministic. No source change. Every line this PR adds is now covered; the three that remain uncovered in the file all predate it. * Python: correct two test docstrings that still described the superseded design Copilot review on microsoft#7757. Both described machinery this PR replaced, which would point the next reader at the wrong design. The cross-instance test said the fix "switched to a process-wide, per-destination threading.Lock keyed by canonical path". Ownership comes from a process-wide queue keyed by the canonical destination path; the lock is what it replaced. The cross-loop test was worse: it said the hand-off signal is one "any loop can await through `asyncio.wrap_future`". That is the mechanism this PR deliberately does not use -- it chains a waiter's cancellation into the shared signal an earlier writer still has to resolve, which is the defect the review caught earlier on this PR. It now names `_wait_for_signal` and says why `wrap_future` is excluded. Checked the file's other references: the remaining `threading.Lock` and `wrap_future` mentions are either locks used by the tests themselves or past-tense descriptions of what was replaced, which are accurate as written. * Python: hold the abandoned-loop tests' pending tasks instead of letting GC decide `test_file_checkpoint_storage_abandonment_mid_chain_keeps_the_queue_ordered` crashed a Windows CI worker. Not an assertion failure -- the worker process died, with no Python traceback. The cause is an unreferenced pending task. The test did `abandoned_loop.create_task(storage.save(...))` without keeping the result, so the task was collectable the moment the call returned, and the loop it belongs to is deliberately closed while the task is still pending. That left the task's finalizer -- which touches the closed loop -- to run at whatever point the garbage collector chose, including during worker teardown. That also explains the shape of the failure. It is not a version boundary: 3.11, 3.12 and 3.14 failed on Windows while 3.10 and 3.13 passed, and every Linux job passed. Non-monotonic scatter like that is a timing flake, and GC timing is the thing that varies. It did not reproduce locally in ten runs on 3.12 -- the test alone, the file under xdist, the workflow directory, and the full core suite under xdist six times. Three sibling tests had the same unreferenced-task pattern, all of them in tests that abandon or shut down a loop on purpose, so all four now hold their tasks for the duration of the test. The sibling at `test_file_checkpoint_storage_abandoned_loop_while_queued_releases_the_destination` already did this correctly, which is what made the inconsistency visible. Holding the reference does not weaken what these tests prove: the abandonment being tested is of the loop, not of the task object, and the task stays pending either way. It only moves the finalizer to a deterministic point. * Python: stop the shutdown test racing the filesystem CI failed with `found 0 tasks` on Linux 3.13. Not the code -- the test's timing assumption. It created the save task, yielded once with `await asyncio.sleep(0)`, then asserted exactly one pending task to prove the write is not itself a task. But a single yield drains every ready callback, so on a fast tmpfs the executor write could finish and resolve the shield inside that same batch; the save completed and `asyncio.all_tasks()` came back empty. The write is now held at its first syscall, so the save is deterministically suspended when the count is taken. Three things about the gate matter: * It only gates this save's own temp file. `checkpoint_module.os` is the real `os` module, so the patch is process-wide, and blocking every `os.open` would stall pytest's own I/O and coverage writes on the same event and deadlock the run. * The assertions are split. An empty list now reports that the gate did not hold, which is a problem with the test; two tasks reports the write being a task, which is the regression. One message could not honestly explain both, and `0` was indistinguishable from the flake it replaced. * A failing assertion releases the worker before propagating, so the real error surfaces immediately instead of after the gate's timeout. Verified rather than assumed: reverting the source to `ensure_future(asyncio.to_thread(...))` still fails the test, now with `found 2 tasks` and the pending `to_thread()` task in the output. Stable over ten runs of the test and five shuffled xdist runs of the whole file. --------- Co-authored-by: Evan Mattson <[email protected]>
1 parent 85fdd7f commit 0ad2e44

2 files changed

Lines changed: 1882 additions & 6 deletions

File tree

0 commit comments

Comments
 (0)