Skip to content

atelet: Wait for a terminated actor's upload or detach to stop - #2943

Merged
Zoe Zhao (zoez7) merged 1 commit into
agent-substrate:mainfrom
msau42:terminate-waits-for-uploads
Oct 20, 2026
Merged

Zoe Zhao (zoez7) merged 1 commit into
agent-substrate:mainfrom
msau42:terminate-waits-for-uploads

Conversation

@msau42

@msau42 Michelle Au (msau42) commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Terminate now cancels the actor's upload or detach in progress and waits for it to stop before it forgets the actor, and keeps new ones from starting while it runs.

Once the control plane has terminated the actor, atelet is no longer uploading its snapshots in the background, so the control plane can delete their objects.

If atelet restarts while Terminate waits, it resumes the upload or detach from the state file until the control plane retries Terminate.

Part of #2069 and #2571

AI use: AI-assisted, I reviewed and tested it

  • Tests pass
  • Docs updated

Terminate now cancels the actor's upload or detach in progress and waits
for it to stop before it forgets the actor, and keeps new ones from
starting while it runs.

Once the control plane has terminated the actor, atelet is no longer
uploading its snapshots in the background, so the control plane can
delete their objects.

If atelet restarts while Terminate waits, it resumes the upload or
detach from the state file until the control plane retries Terminate.
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 20, 2026
Merged via the queue into agent-substrate:main with commit 87a3d9a Oct 20, 2026
15 checks passed
igooch added a commit to igooch/substrate that referenced this pull request Oct 21, 2026
… request times out (agent-substrate#2940)

Fixes agent-substrate#2910.

`DetachActor` ran its detach on the RPC's context. When a resume moved
an actor off a full node (agent-substrate#2881) and the caller left mid-upload, for
example when the router's resume budget ran out, the upload was
cancelled and the next resume started it over. A snapshot slower to
upload than that budget never moved. And because `DetachActor` cancelled
atelet's report, nothing corrected the record if the caller failed to
write it.

Now the detach runs on the tracker's context and always makes atelet's
report due.

```mermaid
sequenceDiagram
    participant A as ate-api-server
    participant L as atelet
    participant O as object storage
    A->>L: DetachActor
    L->>O: upload snapshot
    Note over A: resume budget runs out
    L-->>A: Canceled
    Note over L: the detach keeps running
    O-->>L: upload done
    L->>L: delete local copies
    L->>A: ReportActorSnapshotState
    A->>A: record the detach
```

- If the detach finishes in time, `DetachActor` returns its state as
before. Otherwise it returns the caller's error and the detach runs on.
A second call waits for it.
- Every detach makes the report due, including when the upload succeeds
and the prune fails.
- The upload timeout from agent-substrate#2831 now bounds a `DetachActor` call's
upload, since the caller's context no longer does.
- atelet's shutdown waits for a detach that outlived its call. Terminate
cancels it and waits, as agent-substrate#2943 does for any detach.

#### API changes

- `Atelet.DetachActor` (atelet internal). The proto comment now says
atelet reports the detach and that the detach outlives the call. Comment
only, with the regenerated `atelet_grpc.pb.go`.

#### Worth knowing

- Each move now sends one redundant report. It gets `Aborted` while the
actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet
drops it.
- agent-substrate#2831's subtest asserted that a `DetachActor` call ignores the upload
timeout. It now asserts the timeout applies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants