Skip to content

Resuming a paused actor detaches it when its node has no room - #2598

Closed
Zoe Zhao (zoez7) wants to merge 2 commits into
agent-substrate:mainfrom
zoez7:restore-snapshot-api
Closed

Zoe Zhao (zoez7) wants to merge 2 commits into
agent-substrate:mainfrom
zoez7:restore-snapshot-api

Conversation

@zoez7

@zoez7 Zoe Zhao (zoez7) commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

A PAUSED actor's snapshot sits on the local disk of the node it paused on, so it could only resume there. If that node was full, the resume failed, even if other nodes had free workers. Now the resume runs these steps:

  1. Try to get a free worker on the actor's own node. If there is one, restore there from the local snapshot, as before.
  2. If the node is full, check whether any other node has a free worker. If none does, stop: the actor stays PAUSED and the resume returns ResourceExhausted, so the caller can retry later.
  3. Mark the actor SUSPENDING, and record where in object storage its snapshot will be uploaded. That record is marked PENDING until the upload finishes.
  4. Ask atelet on the old node to promote the snapshot: upload it to object storage, then delete the local copy.
  5. Mark the upload COMPLETED, forget the old node, and mark the actor SUSPENDED, all in one update. Then delete the uploaded snapshot this one replaced, if any. Failing that delete does not fail the resume.
  6. Get a free worker on any node, restore the actor there from the uploaded snapshot, and mark it RUNNING.

Steps 3 to 5 are the same steps SuspendActor runs for a PAUSED actor, so the snapshot is uploaded with the template's preferredFidelity, as a suspend would do.

If the upload in step 4 fails:

  • If atelet is unreachable, the actor stays SUSPENDING and the resume returns Unavailable. Its local snapshot is still on the old node.
  • For any other error, the actor is crashed, the same as when SuspendActor fails.

Resume only starts from PAUSED or SUSPENDED, and also accepts RESUMING, to finish a resume that was cut off. A RUNNING actor returns right away. Resume refuses a SUSPENDING actor with FailedPrecondition, including one left behind by a failed step 4. To recover it, call SuspendActor, which finishes the upload to the location recorded in step 3, then resume the actor.

sequenceDiagram
    participant C as client
    participant A as ate-api-server
    participant S as store
    participant L1 as atelet on node1
    participant O as object storage
    participant L2 as atelet on node2

    C->>A: ResumeActor
    A->>S: assign worker, required node = node1
    S-->>A: no capacity (errNoWorkerRoom)
    A->>S: Schedule with no node constraint
    S-->>A: some node has room
    A->>S: UpdateActor: SUSPENDING, local snapshot durable = PENDING at snapshot URI
    A->>L1: PromoteSnapshot(snapshot uuid, fidelity, object URI)
    L1->>O: upload snapshot files, then manifest
    L1->>L1: delete the actor's local snapshots
    L1-->>A: ok
    A->>S: UpdateActor: durable COMPLETED, locality and assigned_node cleared, SUSPENDED
    A->>O: release the previous durable snapshot (best effort)
    A->>S: assign worker, any node
    S-->>A: worker on node2
    A->>L2: Restore(same snapshot)
    L2->>O: fetch manifest, download files
    L2-->>A: ok
    A->>S: UpdateActor: RUNNING
    A-->>C: ResumeActorResponse
Loading
  • Tests pass
  • Appropriate changes to documentation are included in the PR

@zoez7 Zoe Zhao (zoez7) changed the title Restore snapshot api atelet Restore and DetachActor name their Snapshot Oct 13, 2026
@zoez7 Zoe Zhao (zoez7) changed the title atelet Restore and DetachActor name their Snapshot Restore snapshot api Oct 13, 2026
@zoez7 Zoe Zhao (zoez7) changed the title Restore snapshot api Update atelet.RestoreRequest and atelet.DetachActor API Oct 13, 2026
…omoteSnapshot

Checkpoint and Restore carried a CheckpointType plus a per-type configuration oneof, and UploadPausedCheckpoint spelled the same snapshot out as separate name, URI, and fidelity fields. Replace them with a shared Snapshot message (uuid, generation, fidelity, object storage URI) on Checkpoint, Restore, and PromoteSnapshot, an UploadConfig (UPLOAD_MODE_DEFERRED and UPLOAD_MODE_IMMEDIATE) on Checkpoint, and ActorSnapshotState responses. Update callers and handlers to use Snapshot, UploadConfig, and PromoteSnapshot in place of UploadPausedCheckpoint.
@zoez7

Copy link
Copy Markdown
Collaborator Author

Note: don't review until Review #2592 is merged.

@zoez7 Zoe Zhao (zoez7) changed the title Update atelet.RestoreRequest and atelet.DetachActor API Resuming a paused actor detaches it when its node has no room Oct 13, 2026
A PAUSED actor restores from the local snapshot on the node it paused
on, so its worker assignment was limited to that node. When the node
had no room the resume failed with ResourceExhausted until a worker
there freed up, however much room the rest of the pool had.

The resume now falls back when the scheduler finds no capacity on that
node. It records the snapshot's durable copy as PENDING, asks the
node's atelet to PromoteSnapshot it, then commits the durable copy,
drops the local snapshot and the node pin, and marks the actor
SUSPENDED in one update. The assignment then runs again with no node
constraint, and the restore reads the durable copy. The fallback relies
on PromoteSnapshot still deleting the actor's local snapshots after the
upload.

A resume interrupted mid-detach finds the PENDING record and promotes
again; atelet answers from the uploaded manifest once the upload is
done. A failed promote leaves the actor PAUSED instead of crashing it,
since its local snapshot is still on the node. The no-capacity error
now wraps a sentinel so the fallback can recognize it; its message is
unchanged.
@zoez7

Copy link
Copy Markdown
Collaborator Author

Replaced by #2881

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api-machinery area/scheduling area/storage kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants