Repository navigation
Resuming a paused actor detaches it when its node has no room - #2598
Closed
Zoe Zhao (zoez7) wants to merge 2 commits into
Closed
Zoe Zhao (zoez7) wants to merge 2 commits into
Zoe Zhao (zoez7) wants to merge 2 commits into
Conversation
Zoe Zhao (zoez7)
force-pushed
the
restore-snapshot-api
branch
from
October 13, 2026 20:50
c40d955 to
220dd23
Compare
2 tasks
Zoe Zhao (zoez7)
force-pushed
the
restore-snapshot-api
branch
from
October 13, 2026 21:07
220dd23 to
04cf06e
Compare
…omoteSnapshot Checkpoint and Restore carried a CheckpointType plus a per-type configuration oneof, and UploadPausedCheckpoint spelled the same snapshot out as separate name, URI, and fidelity fields. Replace them with a shared Snapshot message (uuid, generation, fidelity, object storage URI) on Checkpoint, Restore, and PromoteSnapshot, an UploadConfig (UPLOAD_MODE_DEFERRED and UPLOAD_MODE_IMMEDIATE) on Checkpoint, and ActorSnapshotState responses. Update callers and handlers to use Snapshot, UploadConfig, and PromoteSnapshot in place of UploadPausedCheckpoint.
Zoe Zhao (zoez7)
requested review from
Dmitry Berkovich (dberkov) and
Michelle Au (msau42)
October 13, 2026 21:24
Collaborator
Author
|
Note: don't review until Review #2592 is merged. |
Zoe Zhao (zoez7)
removed request for
Dmitry Berkovich (dberkov) and
Michelle Au (msau42)
October 13, 2026 22:16
Zoe Zhao (zoez7)
force-pushed
the
restore-snapshot-api
branch
from
October 13, 2026 23:38
1235804 to
85c87a8
Compare
A PAUSED actor restores from the local snapshot on the node it paused on, so its worker assignment was limited to that node. When the node had no room the resume failed with ResourceExhausted until a worker there freed up, however much room the rest of the pool had. The resume now falls back when the scheduler finds no capacity on that node. It records the snapshot's durable copy as PENDING, asks the node's atelet to PromoteSnapshot it, then commits the durable copy, drops the local snapshot and the node pin, and marks the actor SUSPENDED in one update. The assignment then runs again with no node constraint, and the restore reads the durable copy. The fallback relies on PromoteSnapshot still deleting the actor's local snapshots after the upload. A resume interrupted mid-detach finds the PENDING record and promotes again; atelet answers from the uploaded manifest once the upload is done. A failed promote leaves the actor PAUSED instead of crashing it, since its local snapshot is still on the node. The no-capacity error now wraps a sentinel so the fallback can recognize it; its message is unchanged.
Zoe Zhao (zoez7)
force-pushed
the
restore-snapshot-api
branch
from
October 13, 2026 23:40
85c87a8 to
779c2b8
Compare
This was referenced Oct 14, 2026
Collaborator
Author
|
Replaced by #2881 |
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PAUSED actor's snapshot sits on the local disk of the node it paused on, so it could only resume there. If that node was full, the resume failed, even if other nodes had free workers. Now the resume runs these steps:
ResourceExhausted, so the caller can retry later.Steps 3 to 5 are the same steps
SuspendActorruns for a PAUSED actor, so the snapshot is uploaded with the template'spreferredFidelity, as a suspend would do.If the upload in step 4 fails:
Unavailable. Its local snapshot is still on the old node.SuspendActorfails.Resume only starts from PAUSED or SUSPENDED, and also accepts RESUMING, to finish a resume that was cut off. A RUNNING actor returns right away. Resume refuses a SUSPENDING actor with
FailedPrecondition, including one left behind by a failed step 4. To recover it, callSuspendActor, which finishes the upload to the location recorded in step 3, then resume the actor.sequenceDiagram participant C as client participant A as ate-api-server participant S as store participant L1 as atelet on node1 participant O as object storage participant L2 as atelet on node2 C->>A: ResumeActor A->>S: assign worker, required node = node1 S-->>A: no capacity (errNoWorkerRoom) A->>S: Schedule with no node constraint S-->>A: some node has room A->>S: UpdateActor: SUSPENDING, local snapshot durable = PENDING at snapshot URI A->>L1: PromoteSnapshot(snapshot uuid, fidelity, object URI) L1->>O: upload snapshot files, then manifest L1->>L1: delete the actor's local snapshots L1-->>A: ok A->>S: UpdateActor: durable COMPLETED, locality and assigned_node cleared, SUSPENDED A->>O: release the previous durable snapshot (best effort) A->>S: assign worker, any node S-->>A: worker on node2 A->>L2: Restore(same snapshot) L2->>O: fetch manifest, download files L2-->>A: ok A->>S: UpdateActor: RUNNING A-->>C: ResumeActorResponse