Repository navigation
atelet: Wait for a terminated actor's upload or detach to stop - #2943
Merged
Zoe Zhao (zoez7) merged 1 commit intoOct 20, 2026
Merged
Conversation
Michelle Au (msau42)
force-pushed
the
terminate-waits-for-uploads
branch
from
October 20, 2026 03:44
2a8a490 to
638e1f1
Compare
Terminate now cancels the actor's upload or detach in progress and waits for it to stop before it forgets the actor, and keeps new ones from starting while it runs. Once the control plane has terminated the actor, atelet is no longer uploading its snapshots in the background, so the control plane can delete their objects. If atelet restarts while Terminate waits, it resumes the upload or detach from the state file until the control plane retries Terminate.
Michelle Au (msau42)
force-pushed
the
terminate-waits-for-uploads
branch
from
October 20, 2026 04:52
638e1f1 to
753c4af
Compare
2 tasks done
Zoe Zhao (zoez7)
enabled auto-merge
October 20, 2026 16:35
Zoe Zhao (zoez7)
approved these changes
Oct 20, 2026
2 tasks done
igooch
added a commit
to igooch/substrate
that referenced
this pull request
Oct 21, 2026
… request times out (agent-substrate#2940) Fixes agent-substrate#2910. `DetachActor` ran its detach on the RPC's context. When a resume moved an actor off a full node (agent-substrate#2881) and the caller left mid-upload, for example when the router's resume budget ran out, the upload was cancelled and the next resume started it over. A snapshot slower to upload than that budget never moved. And because `DetachActor` cancelled atelet's report, nothing corrected the record if the caller failed to write it. Now the detach runs on the tracker's context and always makes atelet's report due. ```mermaid sequenceDiagram participant A as ate-api-server participant L as atelet participant O as object storage A->>L: DetachActor L->>O: upload snapshot Note over A: resume budget runs out L-->>A: Canceled Note over L: the detach keeps running O-->>L: upload done L->>L: delete local copies L->>A: ReportActorSnapshotState A->>A: record the detach ``` - If the detach finishes in time, `DetachActor` returns its state as before. Otherwise it returns the caller's error and the detach runs on. A second call waits for it. - Every detach makes the report due, including when the upload succeeds and the prune fails. - The upload timeout from agent-substrate#2831 now bounds a `DetachActor` call's upload, since the caller's context no longer does. - atelet's shutdown waits for a detach that outlived its call. Terminate cancels it and waits, as agent-substrate#2943 does for any detach. #### API changes - `Atelet.DetachActor` (atelet internal). The proto comment now says atelet reports the detach and that the detach outlives the call. Comment only, with the regenerated `atelet_grpc.pb.go`. #### Worth knowing - Each move now sends one redundant report. It gets `Aborted` while the actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet drops it. - agent-substrate#2831's subtest asserted that a `DetachActor` call ignores the upload timeout. It now asserts the timeout applies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Terminate now cancels the actor's upload or detach in progress and waits for it to stop before it forgets the actor, and keeps new ones from starting while it runs.
Once the control plane has terminated the actor, atelet is no longer uploading its snapshots in the background, so the control plane can delete their objects.
If atelet restarts while Terminate waits, it resumes the upload or detach from the state file until the control plane retries Terminate.
Part of #2069 and #2571
AI use: AI-assisted, I reviewed and tested it