Repository navigation
Add WorkerService.ReportActorSnapshotState API for atelet snapshot reports - #2612
Merged
Zoe Zhao (zoez7) merged 5 commits intoOct 14, 2026
Merged
Zoe Zhao (zoez7) merged 5 commits into
Zoe Zhao (zoez7) merged 5 commits into
Conversation
With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node on its own timers. The control plane needs to learn about both to keep each snapshot's durable status, locality, and the actor's assigned_node correct. ReportActorSnapshotState lets atelet report the full snapshot state it holds for an actor on its node: the latest local snapshot, the one being uploaded, and the newest durable one. Each report supersedes older ones, so a lost report is repaired by the next. The messages mirror atelet's ActorSnapshotState, flattened because this API's Snapshot is the public resource. This adds only the API and its validation. The RPC returns Unimplemented until the handler lands.
atelet records when a checkpoint committed only for snapshots it took. A snapshot it restored from object storage, or found already uploaded, has no commit time on that node.
igooch
marked this pull request as ready for review
October 13, 2026 22:57
Zoe Zhao (zoez7)
approved these changes
Oct 13, 2026
| // Only atelet should call this, with its own client certificate. The | ||
| // control plane serves it only for an Actor on the caller's node. | ||
| // Idempotent: each report carries the whole state and replaces the previous | ||
| // one. Returns ABORTED while another operation holds the Actor; retry. |
Collaborator
There was a problem hiding this comment.
Returns ABORTED while another operation holds the Actor; retry.
SGTM.
Zoe Zhao (zoez7)
enabled auto-merge
October 13, 2026 23:26
2 tasks
Yongkun Anfernee Gui (anfernee)
pushed a commit
to anfernee/substrate
that referenced
this pull request
Oct 15, 2026
…nt-substrate#2622) Part of Actor Lifecycle v2 agent-substrate#2069. Second PR for agent-substrate#2570, after agent-substrate#2612 added the API. This serves `WorkerService.ReportActorSnapshotState`. With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node, on its own timers. The report is how ate-api-server learns about both. Nothing sends reports yet; that comes with atelet's asynchronous upload work. A report lists the snapshots atelet holds for an actor on its node. ate-api-server acts on two things in it: - **`latest_durable` means an upload finished.** If it names a recorded snapshot with the same uuid and upload URI, that snapshot becomes `COMPLETED`, with its manifest's fidelity. - **A report with neither `latest_local` nor `uploading` means atelet deleted its copies (a detach).** It covers this node's snapshots up to the reported `latest_durable` generation. Those localities are cleared, and a suspended actor with no snapshot left on the node is released from it. ate-api-server ignores the rest of the report. <details open> <summary>Flowchart: how a report is applied</summary> ```mermaid flowchart TD A["WorkerService.ReportActorSnapshotState(ReportActorSnapshotStateRequest)<br/>TODO: atelet does not send it yet"] --> B{"Caller presents atelet's certificate?"} B -- no --> X1["Unauthenticated or PermissionDenied"] B -- yes --> C{"Request passes validation?"} C -- no --> X2["InvalidArgument, including a report with none of<br/>latest_local, uploading, and latest_durable"] C -- yes --> E{"Actor exists, actor_uid equals Actor.metadata.uid,<br/>and Actor is on the caller's node?"} E -- no --> X4["NotFound"] E -- yes --> D{"Actor's lease free?"} D -- no --> X3["Aborted, atelet retries"] D -- yes --> E2{"Same check again, under the lease"} E2 -- no --> X4 E2 -- yes --> S{"status.state?"} S -- "CRASHED" --> X9["FailedPrecondition"] S -- "SUSPENDING, RESUMING,<br/>DELETING, REVERTING" --> X10["Aborted, atelet retries"] S -- "any other state" --> X12["Internal"] S -- "SUSPENDED or RUNNING" --> G{"latest_durable set?"} G -- no --> K{"latest_local and uploading both unset?"} G -- yes --> GF{"At or below the latest COMPLETED generation?"} GF -- yes --> K GF -- no --> GR{"Its generation is in status.snapshots?"} GR -- "no, and not newer than<br/>last_assigned_generation" --> K GR -- "no, and newer" --> X5["FailedPrecondition"] GR -- yes --> U{"Same uuid, and an upload URI is recorded?"} U -- no --> X6["FailedPrecondition"] U -- yes --> V{"Reported URI equals the recorded one?"} V -- no --> X7["InvalidArgument"] V -- yes --> H["Mark it COMPLETED"] H --> K K -- yes --> K2{"Latest snapshot on this node, up to<br/>latest_durable's generation, is durable?"} K2 -- no --> X8["FailedPrecondition, nothing recorded"] K2 -- yes --> J["Detach: clear this node's localities up to that generation.<br/>Clear assigned_node if status.state is SUSPENDED, it is this node,<br/>and no snapshot is left on this node"] K -- no --> P{"status changed?"} J --> P P -- no --> Q["Done, nothing written"] P -- yes --> R["Write the actor"] ``` </details> ## Outcomes | Report | Result | | --- | --- | | Any report for a `CRASHED` actor | `FailedPrecondition` | | Any report for a `SUSPENDING`, `RESUMING`, `DELETING`, or `REVERTING` actor | `Aborted`, retried | | `latest_durable` for a recorded snapshot with matching uuid and upload URI | That snapshot becomes `COMPLETED`, with the fidelity its manifest records | | `latest_durable` at or below the latest `COMPLETED` generation | No change | | `latest_durable` for a generation `status.snapshots` no longer has, such as one a revert discarded | No change | | `latest_durable` for an unrecorded generation newer than any assigned | `FailedPrecondition` | | `latest_durable` with another uuid, or for a snapshot with no recorded upload URI | `FailedPrecondition` | | `latest_durable` uploaded to a URI other than the recorded one | `InvalidArgument` | | None of `latest_local`, `uploading`, and `latest_durable` set | `InvalidArgument` | | Neither `latest_local` nor `uploading` set (a detach) | Clears this node's localities up to the `latest_durable` generation. Clears `assigned_node` if the actor is `SUSPENDED` there and no snapshot is left on this node. Refused with `FailedPrecondition` while the latest covered snapshot isn't durable | | A late detach, after a newer snapshot on this node | Leaves the newer snapshot and `assigned_node` alone | | A detach that's refused | Nothing is recorded, including an upload the same report carries | | A report that changes nothing | Nothing is written | ## Notes - **Nothing is pruned or deleted.** The existing workflows prune old snapshots and delete their objects at the next suspend or delete. - **A detach covers only generations up to its `latest_durable`,** so a late or repeated report can't clear a newer snapshot. Repeating any report is safe. - **A detach waits for a durable snapshot.** Clearing the locality of a snapshot that isn't durable would let the actor resume from an older one. - **A report applies as a whole or not at all.** atelet's detach uploads the latest snapshot first, so a refused detach is rare. - **The actor is checked before the lease is taken,** and again under it, so an atelet can't hold the lease of an actor on another node. - **Reports apply only to a `SUSPENDED` or `RUNNING` actor.** A transitioning actor's snapshots belong to its workflow, and a `CRASHED` actor's wait for a revert. - **Every UID or node refusal is `NotFound`,** so an atelet learns nothing about actors on other nodes. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of Actor Lifecycle v2 #2069. First PR for #2570.
With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node on its own timers. This adds the API atelet uses to tell ate-api-server about both.
ReportActorSnapshotStatecarries the full snapshot state atelet holds for an actor on its node. The state lists the latest local snapshot, the one being uploaded, and the newest durable one. Each report supersedes older ones.ActorSnapshotStateandSnapshotReportmirror atelet's messages in #2592, since a public proto can't import an internal one.SnapshotReportrepeats four fields of the publicSnapshot:uuid,generation,fidelity, andobject. It doesn't reuseSnapshotbecause that message requiresowner, which atelet doesn't know.Snapshotalso carries fields the control plane owns, such aslocalityand the durable copy's status, which a report shouldn't set.This PR adds only the proto, the generated code, and request validation with tests. The RPC returns
Unimplementeduntil the handler lands. The following PRs add the function that applies a report to the actor, thecontrolapimethod that applies it under the actor's lease, and theworkerservicehandler.Reports will be applied while holding the actor's lease, like any other change to an actor. If another workflow holds the lease, such as a resume, the report is rejected with
Abortedand atelet retries it.Open question for review: which name do we use,
ReportActorSnapshotStateorReportSnapshotStatus?