Skip to content

atelet: Bound background snapshot uploads with a timeout - #2831

Merged
Zoe Zhao (zoez7) merged 1 commit into
agent-substrate:mainfrom
msau42:snapshot-upload-timeout
Oct 19, 2026
Merged

Zoe Zhao (zoez7) merged 1 commit into
agent-substrate:mainfrom
msau42:snapshot-upload-timeout

Conversation

@msau42

@msau42 Michelle Au (msau42) commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Background uploads and detaches ran under atelet's lifetime context with no per-call deadline. A wedged object-store plugin call or stalled stream could occupy a background worker indefinitely.

Each background upload attempt, including the one a background detach runs first, now runs under a timeout that scales based on the snapshot size.
--snapshot-upload-timeout is the initial timeout value
--snapshot-upload-est-throughput-mib is used to add on additional time based off of the snapshot size

When an upload exceeds the timeout, atelet cancels that attempt, logs a warning, and retries the actor after the exponential backoff. Foreground Detach RPCs remain bounded by their caller's context.

Part of #2069 and #2571

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Comment thread docs/metrics/registry/metrics.yaml Resolved
Comment thread cmd/atelet/main.go Resolved
@msau42
Michelle Au (msau42) force-pushed the snapshot-upload-timeout branch 2 times, most recently from 6312f62 to a6eec79 Compare October 19, 2026 21:26
Background uploads and detaches ran under atelet's lifetime context with no per-call deadline. A wedged object-store plugin call or stalled stream could occupy a background worker indefinitely.

Each background upload attempt, including the one a background detach runs first, now runs under a timeout: --snapshot-upload-timeout, plus the time the snapshot takes at --snapshot-upload-est-throughput-mib, so a large snapshot is not cut off. When an upload exceeds the timeout, atelet cancels that attempt, logs a warning, and retries the actor after the exponential backoff. Each consecutive failure doubles the next attempt's timeout, to at most 8 times the first. Foreground Detach RPCs remain bounded by their caller's context.
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 19, 2026
Merged via the queue into agent-substrate:main with commit 69fef8d Oct 19, 2026
15 checks passed
igooch added a commit to igooch/substrate that referenced this pull request Oct 21, 2026
… request times out (agent-substrate#2940)

Fixes agent-substrate#2910.

`DetachActor` ran its detach on the RPC's context. When a resume moved
an actor off a full node (agent-substrate#2881) and the caller left mid-upload, for
example when the router's resume budget ran out, the upload was
cancelled and the next resume started it over. A snapshot slower to
upload than that budget never moved. And because `DetachActor` cancelled
atelet's report, nothing corrected the record if the caller failed to
write it.

Now the detach runs on the tracker's context and always makes atelet's
report due.

```mermaid
sequenceDiagram
    participant A as ate-api-server
    participant L as atelet
    participant O as object storage
    A->>L: DetachActor
    L->>O: upload snapshot
    Note over A: resume budget runs out
    L-->>A: Canceled
    Note over L: the detach keeps running
    O-->>L: upload done
    L->>L: delete local copies
    L->>A: ReportActorSnapshotState
    A->>A: record the detach
```

- If the detach finishes in time, `DetachActor` returns its state as
before. Otherwise it returns the caller's error and the detach runs on.
A second call waits for it.
- Every detach makes the report due, including when the upload succeeds
and the prune fails.
- The upload timeout from agent-substrate#2831 now bounds a `DetachActor` call's
upload, since the caller's context no longer does.
- atelet's shutdown waits for a detach that outlived its call. Terminate
cancels it and waits, as agent-substrate#2943 does for any detach.

#### API changes

- `Atelet.DetachActor` (atelet internal). The proto comment now says
atelet reports the detach and that the detach outlives the call. Comment
only, with the regenerated `atelet_grpc.pb.go`.

#### Worth knowing

- Each move now sends one redundant report. It gets `Aborted` while the
actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet
drops it.
- agent-substrate#2831's subtest asserted that a `DetachActor` call ignores the upload
timeout. It now asserts the timeout applies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants