Skip to content

Add WorkerService.ReportActorSnapshotState API for atelet snapshot reports - #2612

Merged
Zoe Zhao (zoez7) merged 5 commits into
agent-substrate:mainfrom
igooch:report-actor-snapshot-state-proto
Oct 14, 2026
Merged

Zoe Zhao (zoez7) merged 5 commits into
agent-substrate:mainfrom
igooch:report-actor-snapshot-state-proto

Conversation

@igooch

@igooch igooch commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Part of Actor Lifecycle v2 #2069. First PR for #2570.

With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node on its own timers. This adds the API atelet uses to tell ate-api-server about both.

ReportActorSnapshotState carries the full snapshot state atelet holds for an actor on its node. The state lists the latest local snapshot, the one being uploaded, and the newest durable one. Each report supersedes older ones. ActorSnapshotState and SnapshotReport mirror atelet's messages in #2592, since a public proto can't import an internal one.

SnapshotReport repeats four fields of the public Snapshot: uuid, generation, fidelity, and object. It doesn't reuse Snapshot because that message requires owner, which atelet doesn't know. Snapshot also carries fields the control plane owns, such as locality and the durable copy's status, which a report shouldn't set.

This PR adds only the proto, the generated code, and request validation with tests. The RPC returns Unimplemented until the handler lands. The following PRs add the function that applies a report to the actor, the controlapi method that applies it under the actor's lease, and the workerservice handler.

Reports will be applied while holding the actor's lease, like any other change to an actor. If another workflow holds the lease, such as a resume, the report is rejected with Aborted and atelet retries it.

Open question for review: which name do we use, ReportActorSnapshotState or ReportSnapshotStatus?

  • Tests pass
  • Appropriate changes to documentation are included in the PR

With asynchronous suspend, atelet uploads a suspended actor's snapshot
and later detaches the actor from its node on its own timers. The
control plane needs to learn about both to keep each snapshot's durable
status, locality, and the actor's assigned_node correct.

ReportActorSnapshotState lets atelet report the full snapshot state it
holds for an actor on its node: the latest local snapshot, the one being
uploaded, and the newest durable one. Each report supersedes older ones,
so a lost report is repaired by the next. The messages mirror atelet's
ActorSnapshotState, flattened because this API's Snapshot is the public
resource.

This adds only the API and its validation. The RPC returns Unimplemented
until the handler lands.
atelet records when a checkpoint committed only for snapshots it took.
A snapshot it restored from object storage, or found already uploaded,
has no commit time on that node.
@igooch
igooch marked this pull request as ready for review October 13, 2026 22:57
// Only atelet should call this, with its own client certificate. The
// control plane serves it only for an Actor on the caller's node.
// Idempotent: each report carries the whole state and replaces the previous
// one. Returns ABORTED while another operation holds the Actor; retry.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returns ABORTED while another operation holds the Actor; retry.

SGTM.

@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 13, 2026
Merged via the queue into agent-substrate:main with commit e2163f9 Oct 14, 2026
14 checks passed
Yongkun Anfernee Gui (anfernee) pushed a commit to anfernee/substrate that referenced this pull request Oct 15, 2026
…nt-substrate#2622)

Part of Actor Lifecycle v2 agent-substrate#2069. Second PR for agent-substrate#2570, after agent-substrate#2612 added
the API.

This serves `WorkerService.ReportActorSnapshotState`. With asynchronous
suspend, atelet uploads a suspended actor's snapshot and later detaches
the actor from its node, on its own timers. The report is how
ate-api-server learns about both. Nothing sends reports yet; that comes
with atelet's asynchronous upload work.

A report lists the snapshots atelet holds for an actor on its node.
ate-api-server acts on two things in it:

- **`latest_durable` means an upload finished.** If it names a recorded
snapshot with the same uuid and upload URI, that snapshot becomes
`COMPLETED`, with its manifest's fidelity.
- **A report with neither `latest_local` nor `uploading` means atelet
deleted its copies (a detach).** It covers this node's snapshots up to
the reported `latest_durable` generation. Those localities are cleared,
and a suspended actor with no snapshot left on the node is released from
it.

ate-api-server ignores the rest of the report.

<details open>
<summary>Flowchart: how a report is applied</summary>

```mermaid
flowchart TD
    A["WorkerService.ReportActorSnapshotState(ReportActorSnapshotStateRequest)<br/>TODO: atelet does not send it yet"] --> B{"Caller presents atelet's certificate?"}
    B -- no --> X1["Unauthenticated or PermissionDenied"]
    B -- yes --> C{"Request passes validation?"}
    C -- no --> X2["InvalidArgument, including a report with none of<br/>latest_local, uploading, and latest_durable"]
    C -- yes --> E{"Actor exists, actor_uid equals Actor.metadata.uid,<br/>and Actor is on the caller's node?"}
    E -- no --> X4["NotFound"]
    E -- yes --> D{"Actor's lease free?"}
    D -- no --> X3["Aborted, atelet retries"]
    D -- yes --> E2{"Same check again, under the lease"}
    E2 -- no --> X4
    E2 -- yes --> S{"status.state?"}
    S -- "CRASHED" --> X9["FailedPrecondition"]
    S -- "SUSPENDING, RESUMING,<br/>DELETING, REVERTING" --> X10["Aborted, atelet retries"]
    S -- "any other state" --> X12["Internal"]
    S -- "SUSPENDED or RUNNING" --> G{"latest_durable set?"}
    G -- no --> K{"latest_local and uploading both unset?"}
    G -- yes --> GF{"At or below the latest COMPLETED generation?"}
    GF -- yes --> K
    GF -- no --> GR{"Its generation is in status.snapshots?"}
    GR -- "no, and not newer than<br/>last_assigned_generation" --> K
    GR -- "no, and newer" --> X5["FailedPrecondition"]
    GR -- yes --> U{"Same uuid, and an upload URI is recorded?"}
    U -- no --> X6["FailedPrecondition"]
    U -- yes --> V{"Reported URI equals the recorded one?"}
    V -- no --> X7["InvalidArgument"]
    V -- yes --> H["Mark it COMPLETED"]
    H --> K
    K -- yes --> K2{"Latest snapshot on this node, up to<br/>latest_durable's generation, is durable?"}
    K2 -- no --> X8["FailedPrecondition, nothing recorded"]
    K2 -- yes --> J["Detach: clear this node's localities up to that generation.<br/>Clear assigned_node if status.state is SUSPENDED, it is this node,<br/>and no snapshot is left on this node"]
    K -- no --> P{"status changed?"}
    J --> P
    P -- no --> Q["Done, nothing written"]
    P -- yes --> R["Write the actor"]
```

</details>

## Outcomes

| Report | Result |
| --- | --- |
| Any report for a `CRASHED` actor | `FailedPrecondition` |
| Any report for a `SUSPENDING`, `RESUMING`, `DELETING`, or `REVERTING`
actor | `Aborted`, retried |
| `latest_durable` for a recorded snapshot with matching uuid and upload
URI | That snapshot becomes `COMPLETED`, with the fidelity its manifest
records |
| `latest_durable` at or below the latest `COMPLETED` generation | No
change |
| `latest_durable` for a generation `status.snapshots` no longer has,
such as one a revert discarded | No change |
| `latest_durable` for an unrecorded generation newer than any assigned
| `FailedPrecondition` |
| `latest_durable` with another uuid, or for a snapshot with no recorded
upload URI | `FailedPrecondition` |
| `latest_durable` uploaded to a URI other than the recorded one |
`InvalidArgument` |
| None of `latest_local`, `uploading`, and `latest_durable` set |
`InvalidArgument` |
| Neither `latest_local` nor `uploading` set (a detach) | Clears this
node's localities up to the `latest_durable` generation. Clears
`assigned_node` if the actor is `SUSPENDED` there and no snapshot is
left on this node. Refused with `FailedPrecondition` while the latest
covered snapshot isn't durable |
| A late detach, after a newer snapshot on this node | Leaves the newer
snapshot and `assigned_node` alone |
| A detach that's refused | Nothing is recorded, including an upload the
same report carries |
| A report that changes nothing | Nothing is written |

## Notes

- **Nothing is pruned or deleted.** The existing workflows prune old
snapshots and delete their objects at the next suspend or delete.
- **A detach covers only generations up to its `latest_durable`,** so a
late or repeated report can't clear a newer snapshot. Repeating any
report is safe.
- **A detach waits for a durable snapshot.** Clearing the locality of a
snapshot that isn't durable would let the actor resume from an older
one.
- **A report applies as a whole or not at all.** atelet's detach uploads
the latest snapshot first, so a refused detach is rare.
- **The actor is checked before the lease is taken,** and again under
it, so an atelet can't hold the lease of an actor on another node.
- **Reports apply only to a `SUSPENDED` or `RUNNING` actor.** A
transitioning actor's snapshots belong to its workflow, and a `CRASHED`
actor's wait for a revert.
- **Every UID or node refusal is `NotFound`,** so an atelet learns
nothing about actors on other nodes.

- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
@igooch
igooch deleted the report-actor-snapshot-state-proto branch October 27, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants