Repository navigation
Commit 0ad2e44
Python: fix: make concurrent FileCheckpointStorage saves not race on a shared temp path (microsoft#7757)
* Python: fix: make concurrent FileCheckpointStorage saves not race on a shared temp path
FileCheckpointStorage.save() wrote to and then renamed a fixed
"<checkpoint-id>.json.tmp" path, so concurrent saves of the same
checkpoint ID raced over the shared temp file. Whichever save renamed
first removed the temp file still being written by another save, which
then failed with FileNotFoundError / PermissionError in os.replace.
Create a unique temp file per save in the destination directory (so
os.replace remains atomic), serialize same-ID writes with a per-ID
lock, and retry the atomic move briefly to absorb the transient
Windows background-handle PermissionError that surfaces even for fully
serialized replaces.
Fixes microsoft#7748
* Python: fix: lock file writes at destination path, shield workers from cancellation
Address two post-merge review concerns on FileCheckpointStorage.save():
- The per-event-loop, per-instance asyncio.Lock registry could not
serialize two FileCheckpointStorage instances pointed at the same
directory, nor a single instance driven from two event loops. Replace
it with a process-wide, per-canonical-path threading.Lock registry
(lazily populated, bounded by distinct destinations actually written).
Because asyncio.to_thread runs the actual file I/O on a worker thread,
the threading lock serializes correctly across coroutines, loops, and
instances and spans the entire open + write + os.replace window.
- Caller-side cancellation previously released the write lock early:
a CancelledError delivered inside 'await asyncio.to_thread(...)' exited
the 'async with' and ran _release_save_lock_ref while the OS thread
kept writing. The next save for the same checkpoint_id could then
reach os.replace concurrently, reintroducing the PermissionError race
and possibly landing stale data over a newer write. Shield the worker
so cancellation propagates only after the in-flight write completes.
Also replace the refcount-aware registry test with:
- registry-bounded test for the new invariant (one lock per destination)
- cross-instance concurrent save test (two storages, same directory)
- gated os.replace cancel-race test (deterministically parks A's worker
mid-publish, confirms save B cannot reach os.replace until A completes)
Fixes microsoft#7748
* Python: take checkpoint destination ownership before submitting the write
Implements the revised A shape chosen in review: process-wide coordination keyed
by canonical destination, asynchronous waiters, and registry entries released
after the final queued or running operation.
Three review findings shared one cause -- ownership was established inside the
executor thread rather than before submission.
Acquiring the destination lock inside the worker meant every same-path save
occupied an `asyncio.to_thread` worker while merely waiting, so a burst could
fill the default pool and stall unrelated work including checkpoint loads, and
could deadlock once the write that had to finish first was queued behind those
waiters. Ownership now comes from a queue keyed by the canonical path and is
taken before the write is submitted, so only the active write holds a worker.
Waiters suspend on `asyncio.wrap_future` over a `concurrent.futures.Future`,
which is not bound to a loop, so one chain also orders writers enqueued from
different event loops -- something a per-path lock could not do.
The registry was a dict that never removed an entry. Because
`WorkflowCheckpoint` generates a fresh UUID by default, every save retained one
entry for the life of the process. Entries are now reference-counted by
queued-or-running operations and dropped by the last release.
Cancellation is handled explicitly, both ways the review allowed. A save
cancelled while still waiting for the destination has nothing in the executor,
so it is simply removed -- and it still hands ownership on, or every later save
for that path would wait forever. A save cancelled mid-write drains its worker
before propagating, because releasing first would let the next save start while
this replace is still in flight. That drain absorbs re-delivered cancellations:
once a task has a cancellation pending its next await raises immediately, so a
single shielded await returns with the worker still running and the write can
still land after ownership is released.
Ownership is released as the write's last act on the worker thread, with the
coroutine's `finally` as a backstop for the case where no write was submitted.
Releasing only from the coroutine regressed on the design it replaced: a
`threading.Lock` was released by the worker thread, which outlives the loop that
submitted it, whereas a coroutine that never resumes -- its loop closed with the
task still pending -- would leave the destination owned for the life of the
process and hang every later save to it. The two releasers are idempotent, so
exactly one of them accounts for each ticket.
Tests are deterministic and cover each case: registry release for repeated,
distinct-ID and concurrent saves; queued saves not occupying executor threads,
gated by a two-worker executor with three saves in flight; serialization across
two real event loops; cancellation before submission writing nothing and not
stalling the follower; repeated cancellation still draining; a write failing
while draining being reported rather than lost; a failed save releasing its
destination and leaving the path usable; recovery of a destination whose loop was
closed mid-write; and the defensive release path.
* Python: tie checkpoint write ownership to the write, not to asyncio state
Follow-up review on microsoft#7757 found two ways a later save could still overtake an
earlier one. Both had the same cause: asyncio objects were standing in for the
state of work that lives outside asyncio, and cancellation can mark those done
while the work continues.
`asyncio.wrap_future` chains cancellation into the future it wraps. Cancelling
the middle of three queued saves therefore cancelled the hand-off signal the
first save still had to resolve, and the unconditional release then let the
third reach `os.replace` beside the first -- after which the first write could
land last and overwrite the newer checkpoint. Waiting now goes through
`_wait_for_signal`, which feeds a fresh per-wait future from a done-callback and
leaves the shared signal untouched no matter what happens to the waiter. A save
cancelled while queued no longer resolves its own signal either; it defers the
hand-off until its predecessor has actually finished, so the queue stays ordered
while still guaranteeing nothing waits behind it forever.
Draining on the task wrapping `asyncio.to_thread` was the second: a cancelled
task reports `done()` while its function is still inside `os.replace`, so loop
shutdown released ownership with the write in flight. The drain now waits on the
signal the worker thread resolves, which cancellation cannot mark done.
Fixing those exposed two more failures of the same kind, both found by probing
rather than by review.
`ensure_future(asyncio.to_thread(...))` makes the write a Task, and shutdown
cancels every task -- possibly before it reaches the executor. Nothing then
released the destination, and the drain waited for a signal that could never be
resolved, so the save never completed at all. The write is now submitted with
`run_in_executor`, which submits synchronously and returns a plain Future that
`asyncio.all_tasks()` does not include, plus a done-callback that releases on
success, failure and cancellation alike.
Deferring a cancelled ticket's hand-off onto its predecessor made release
recursive: resolving one signal ran the next ticket's callback synchronously, so
a run of cancelled queued saves longer than the recursion limit raised
`RecursionError` on the worker thread partway through and left the destination
owned for good. Releases are now collected per thread and drained in a loop, so
the depth is flat however long the run.
Two smaller points from the same review. A write that fails while draining is
recorded on the ticket rather than read back from the worker future, because a
cancelled task hides its own exception and the cancelled caller never sees it.
And the drain registers one done-callback pointing at whichever waiter is
current, rather than one per wait, so a caller cancelling in a loop no longer
grows that list in step with it.
Tests: cancelling the middle of three queued saves cannot let the third
overtake; a cancelled worker task does not release while the thread writes;
shutdown before the write starts neither hangs nor strands the destination,
asserted structurally on the write not being a task so the regression fails
cleanly instead of hanging CI; a deferred-release run of 2000 links does not
recurse; submission against a dead executor releases; and resolving a signal
after its loop closed does not raise.
* Python: release a queued checkpoint save whose event loop is abandoned
Follow-up review on microsoft#7757. A submitted write is released by its worker thread,
which outlives the loop that submitted it, but a ticket still waiting for its
predecessor has no worker to fall back on. If that waiter's loop closes,
`call_soon_threadsafe` raises `RuntimeError`, the suppression swallowed it, and
the coroutine never resumed -- so nothing resolved the ticket's hand-off signal
and every later save for that destination waited forever. Reproduced: the entry
stays at `pending=1` and a later save from a fresh loop times out.
The predecessor's callback now owns that case. `_wait_for_signal` takes an
`on_abandoned` hook, invoked when the waiter's loop has closed, and the queued
wait passes its own release -- performed on the thread that resolved the
predecessor rather than on a loop that will never run again.
The drain keeps suppressing, and the two guards now differ deliberately: by the
time anything drains, the write has been submitted, so the worker thread and the
submitted future's callback both release without needing that loop. The comment
says so, since the asymmetry is otherwise unexplained.
Two windows this cannot close, both stated in the comment rather than implied: a
loop that closes after `call_soon_threadsafe` succeeded but before the callback
runs, and one abandoned without being closed at all. Reaching either needs a loop
closed with tasks still pending, which asyncio already reports as an error.
Graceful shutdown is unaffected -- `asyncio.run` cancels pending tasks first,
which drives a queued save through its cancellation path and defers the hand-off
onto its predecessor. That path now has its own test, since it is the guarantee
callers actually depend on.
* Python: cover the checkpoint write error paths this PR introduced
Three branches added by this PR had no test. Coverage of `_checkpoint.py` was
reporting 97% with the gaps sitting exactly on the error handling:
* the bounded `os.replace` retry giving up after five attempts. The suite only
ever reached the surrounding lines by accident -- the concurrency tests trip a
real `PermissionError` on Windows now and then, which is environment-dependent
and never happens on Linux CI. The new test pins both halves that matter: the
retry is bounded, and giving up still releases the destination rather than
wedging every later save to it.
* the temp-file cleanup swallowing an `OSError`. It runs in a `finally` while a
write exception is propagating, so letting one out would substitute a
misleading "cannot remove temp file" for the real disk failure. Reverting the
handler makes the new test fail with exactly that substitution.
* the fast path in `_await_signal_through_cancellation` for a signal that has
already resolved. Exercised directly, since arranging the race through
`save()` is not deterministic.
No source change. Every line this PR adds is now covered; the three that remain
uncovered in the file all predate it.
* Python: correct two test docstrings that still described the superseded design
Copilot review on microsoft#7757. Both described machinery this PR replaced, which would
point the next reader at the wrong design.
The cross-instance test said the fix "switched to a process-wide, per-destination
threading.Lock keyed by canonical path". Ownership comes from a process-wide
queue keyed by the canonical destination path; the lock is what it replaced.
The cross-loop test was worse: it said the hand-off signal is one "any loop can
await through `asyncio.wrap_future`". That is the mechanism this PR deliberately
does not use -- it chains a waiter's cancellation into the shared signal an
earlier writer still has to resolve, which is the defect the review caught
earlier on this PR. It now names `_wait_for_signal` and says why `wrap_future`
is excluded.
Checked the file's other references: the remaining `threading.Lock` and
`wrap_future` mentions are either locks used by the tests themselves or
past-tense descriptions of what was replaced, which are accurate as written.
* Python: hold the abandoned-loop tests' pending tasks instead of letting GC decide
`test_file_checkpoint_storage_abandonment_mid_chain_keeps_the_queue_ordered`
crashed a Windows CI worker. Not an assertion failure -- the worker process died,
with no Python traceback.
The cause is an unreferenced pending task. The test did
`abandoned_loop.create_task(storage.save(...))` without keeping the result, so the
task was collectable the moment the call returned, and the loop it belongs to is
deliberately closed while the task is still pending. That left the task's
finalizer -- which touches the closed loop -- to run at whatever point the garbage
collector chose, including during worker teardown.
That also explains the shape of the failure. It is not a version boundary: 3.11,
3.12 and 3.14 failed on Windows while 3.10 and 3.13 passed, and every Linux job
passed. Non-monotonic scatter like that is a timing flake, and GC timing is the
thing that varies. It did not reproduce locally in ten runs on 3.12 -- the test
alone, the file under xdist, the workflow directory, and the full core suite under
xdist six times.
Three sibling tests had the same unreferenced-task pattern, all of them in tests
that abandon or shut down a loop on purpose, so all four now hold their tasks for
the duration of the test. The sibling at
`test_file_checkpoint_storage_abandoned_loop_while_queued_releases_the_destination`
already did this correctly, which is what made the inconsistency visible.
Holding the reference does not weaken what these tests prove: the abandonment
being tested is of the loop, not of the task object, and the task stays pending
either way. It only moves the finalizer to a deterministic point.
* Python: stop the shutdown test racing the filesystem
CI failed with `found 0 tasks` on Linux 3.13. Not the code -- the test's timing
assumption.
It created the save task, yielded once with `await asyncio.sleep(0)`, then
asserted exactly one pending task to prove the write is not itself a task. But a
single yield drains every ready callback, so on a fast tmpfs the executor write
could finish and resolve the shield inside that same batch; the save completed and
`asyncio.all_tasks()` came back empty.
The write is now held at its first syscall, so the save is deterministically
suspended when the count is taken. Three things about the gate matter:
* It only gates this save's own temp file. `checkpoint_module.os` is the real `os`
module, so the patch is process-wide, and blocking every `os.open` would stall
pytest's own I/O and coverage writes on the same event and deadlock the run.
* The assertions are split. An empty list now reports that the gate did not hold,
which is a problem with the test; two tasks reports the write being a task,
which is the regression. One message could not honestly explain both, and `0`
was indistinguishable from the flake it replaced.
* A failing assertion releases the worker before propagating, so the real error
surfaces immediately instead of after the gate's timeout.
Verified rather than assumed: reverting the source to
`ensure_future(asyncio.to_thread(...))` still fails the test, now with `found 2
tasks` and the pending `to_thread()` task in the output. Stable over ten runs of
the test and five shuffled xdist runs of the whole file.
---------
Co-authored-by: Evan Mattson <[email protected]>1 parent 85fdd7f commit 0ad2e44
2 files changed
Lines changed: 1882 additions & 6 deletions
File tree
- python/packages/core
- agent_framework/_workflows
- tests/workflow
0 commit comments