You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Resume's detach should keep running after the caller leaves to finish upload large snapshots #2910
DetachActor uploads the latest local snapshot if it isn't durable yet and deletes the local copies. It also cancels atelet's pending report (cancelReportLocked in snapshottracker/detach.go). From then on, ate-api-server is the only writer that can record the detach.
Problem
The whole step runs on the lease context, which derives from the caller's request context. The upload can take minutes, and the step has no deadline of its own.
If the caller cancels during the upload, the upload is cancelled too, and the next resume starts it over. An upload that takes longer than the router's budget is never finished by a resume. Only atelet's background detach finishes it.
Proposal
Keep the detach running when the caller leaves, and keep atelet's report as the backstop writer:
atelet: DetachActor runs the detach under the tracker's context, not the RPC's, and finishes it the way a background detach does. If the detach finishes before the caller's context ends, DetachActor returns the new state. If it doesn't, DetachActor returns an error and the detach keeps running. When it finishes, atelet sends the new state in a ReportActorSnapshotState call. The detach marks that report due (markReportDueLocked) in both cases, rather than cancelling it. A call made while a detach is already running waits for that detach instead of starting another.
ate-api-server: if DetachActor returns within the request, resume records the state and continues on another node in the same call, as today.
If the call or the record write fails, resume returns a retryable error. The background detach finishes, and its report updates the record. The next resume schedules the now durable-only actor on any node.
Activity
A resume that moves an actor finishes its snapshot upload even if the…