Repository navigation
atelet: Bound background snapshot uploads with a timeout - #2831
Merged
Zoe Zhao (zoez7) merged 1 commit intoOct 19, 2026
Merged
Conversation
Michelle Au (msau42)
force-pushed
the
snapshot-upload-timeout
branch
2 times, most recently
from
October 18, 2026 18:06
4fd5761 to
2d07168
Compare
Michelle Au (msau42)
force-pushed
the
snapshot-upload-timeout
branch
2 times, most recently
from
October 19, 2026 21:26
6312f62 to
a6eec79
Compare
Background uploads and detaches ran under atelet's lifetime context with no per-call deadline. A wedged object-store plugin call or stalled stream could occupy a background worker indefinitely. Each background upload attempt, including the one a background detach runs first, now runs under a timeout: --snapshot-upload-timeout, plus the time the snapshot takes at --snapshot-upload-est-throughput-mib, so a large snapshot is not cut off. When an upload exceeds the timeout, atelet cancels that attempt, logs a warning, and retries the actor after the exponential backoff. Each consecutive failure doubles the next attempt's timeout, to at most 8 times the first. Foreground Detach RPCs remain bounded by their caller's context.
Michelle Au (msau42)
force-pushed
the
snapshot-upload-timeout
branch
from
October 19, 2026 21:57
a6eec79 to
edfe1e5
Compare
Zoe Zhao (zoez7)
approved these changes
Oct 19, 2026
Zoe Zhao (zoez7)
enabled auto-merge
October 19, 2026 22:04
igooch
added a commit
to igooch/substrate
that referenced
this pull request
Oct 21, 2026
… request times out (agent-substrate#2940) Fixes agent-substrate#2910. `DetachActor` ran its detach on the RPC's context. When a resume moved an actor off a full node (agent-substrate#2881) and the caller left mid-upload, for example when the router's resume budget ran out, the upload was cancelled and the next resume started it over. A snapshot slower to upload than that budget never moved. And because `DetachActor` cancelled atelet's report, nothing corrected the record if the caller failed to write it. Now the detach runs on the tracker's context and always makes atelet's report due. ```mermaid sequenceDiagram participant A as ate-api-server participant L as atelet participant O as object storage A->>L: DetachActor L->>O: upload snapshot Note over A: resume budget runs out L-->>A: Canceled Note over L: the detach keeps running O-->>L: upload done L->>L: delete local copies L->>A: ReportActorSnapshotState A->>A: record the detach ``` - If the detach finishes in time, `DetachActor` returns its state as before. Otherwise it returns the caller's error and the detach runs on. A second call waits for it. - Every detach makes the report due, including when the upload succeeds and the prune fails. - The upload timeout from agent-substrate#2831 now bounds a `DetachActor` call's upload, since the caller's context no longer does. - atelet's shutdown waits for a detach that outlived its call. Terminate cancels it and waits, as agent-substrate#2943 does for any detach. #### API changes - `Atelet.DetachActor` (atelet internal). The proto comment now says atelet reports the detach and that the detach outlives the call. Comment only, with the regenerated `atelet_grpc.pb.go`. #### Worth knowing - Each move now sends one redundant report. It gets `Aborted` while the actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet drops it. - agent-substrate#2831's subtest asserted that a `DetachActor` call ignores the upload timeout. It now asserts the timeout applies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background uploads and detaches ran under atelet's lifetime context with no per-call deadline. A wedged object-store plugin call or stalled stream could occupy a background worker indefinitely.
Each background upload attempt, including the one a background detach runs first, now runs under a timeout that scales based on the snapshot size.
--snapshot-upload-timeout is the initial timeout value
--snapshot-upload-est-throughput-mib is used to add on additional time based off of the snapshot size
When an upload exceeds the timeout, atelet cancels that attempt, logs a warning, and retries the actor after the exponential backoff. Foreground Detach RPCs remain bounded by their caller's context.
Part of #2069 and #2571