Skip to content

Resume's detach should keep running after the caller leaves to finish upload large snapshots #2910

Closed

Description

@zoez7

Background

  • Resuming a suspended actor detaches it when its node has no room #2881 lets ResumeActor move a SUSPENDED actor off its full node. It calls atelet's DetachActor (atelet: implement DetachActor #2852) synchronously, under the actor lease, then writes the returned state to the actor record.
  • DetachActor uploads the latest local snapshot if it isn't durable yet and deletes the local copies. It also cancels atelet's pending report (cancelReportLocked in snapshottracker/detach.go). From then on, ate-api-server is the only writer that can record the detach.

Problem

  • The whole step runs on the lease context, which derives from the caller's request context. The upload can take minutes, and the step has no deadline of its own.
  • If the caller cancels during the upload, the upload is cancelled too, and the next resume starts it over. An upload that takes longer than the router's budget is never finished by a resume. Only atelet's background detach finishes it.

Proposal

  • Keep the detach running when the caller leaves, and keep atelet's report as the backstop writer:
    • atelet: DetachActor runs the detach under the tracker's context, not the RPC's, and finishes it the way a background detach does. If the detach finishes before the caller's context ends, DetachActor returns the new state. If it doesn't, DetachActor returns an error and the detach keeps running. When it finishes, atelet sends the new state in a ReportActorSnapshotState call. The detach marks that report due (markReportDueLocked) in both cases, rather than cancelling it. A call made while a detach is already running waits for that detach instead of starting another.
    • ate-api-server: if DetachActor returns within the request, resume records the state and continues on another node in the same call, as today.
      If the call or the record write fails, resume returns a retryable error. The background detach finishes, and its report updates the record. The next resume schedules the now durable-only actor on any node.

Activity

  1. self-assigned this
    on Oct 19, 2026
  2. added a commit that references this issue on Oct 21, 2026
    bb41c52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions