Skip to content

Commit bb41c52

Browse files
authored
A resume that moves an actor finishes its snapshot upload even if the request times out (#2940)
Fixes #2910. `DetachActor` ran its detach on the RPC's context. When a resume moved an actor off a full node (#2881) and the caller left mid-upload, for example when the router's resume budget ran out, the upload was cancelled and the next resume started it over. A snapshot slower to upload than that budget never moved. And because `DetachActor` cancelled atelet's report, nothing corrected the record if the caller failed to write it. Now the detach runs on the tracker's context and always makes atelet's report due. ```mermaid sequenceDiagram participant A as ate-api-server participant L as atelet participant O as object storage A->>L: DetachActor L->>O: upload snapshot Note over A: resume budget runs out L-->>A: Canceled Note over L: the detach keeps running O-->>L: upload done L->>L: delete local copies L->>A: ReportActorSnapshotState A->>A: record the detach ``` - If the detach finishes in time, `DetachActor` returns its state as before. Otherwise it returns the caller's error and the detach runs on. A second call waits for it. - Every detach makes the report due, including when the upload succeeds and the prune fails. - The upload timeout from #2831 now bounds a `DetachActor` call's upload, since the caller's context no longer does. - atelet's shutdown waits for a detach that outlived its call. Terminate cancels it and waits, as #2943 does for any detach. #### API changes - `Atelet.DetachActor` (atelet internal). The proto comment now says atelet reports the detach and that the detach outlives the call. Comment only, with the regenerated `atelet_grpc.pb.go`. #### Worth knowing - Each move now sends one redundant report. It gets `Aborted` while the actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet drops it. - #2831's subtest asserted that a `DetachActor` call ignores the upload timeout. It now asserts the timeout applies.
1 parent 4dd025e commit bb41c52

14 files changed

Lines changed: 324 additions & 138 deletions

File tree

‎cmd/ateapi/internal/controlapi/workflow_resume_detach.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -101,8 +101,8 @@ func (w *ActorWorkflow) detachFromFullNode(ctx context.Context, actorRef resourc
101101
return nil, apierror.Internal("atelet on node %q detached Actor %s but still reports a local snapshot", node, actorRef)
102102
}
103103

104-
// A conflict fails the resume with Aborted. The next resume detaches
105-
// again, and atelet returns the same state.
104+
// A conflict fails the resume with Aborted. atelet's report of the detach
105+
// records it, and so does the next resume, which detaches again.
106106
stored, err := w.recordDetach(ctx, actorRef, actor, node, state)
107107
if err != nil {
108108
return nil, err

‎cmd/atelet/internal/snapshottracker/actor.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -180,8 +180,8 @@ type actorSnapshotsRecord struct {
180180
// ReportDueSince is when the latest background change atelet owes a
181181
// report of (a new LatestDurable, or local snapshots cleared by a
182182
// detach) finished. Zero once no report is due: the control plane
183-
// acknowledged or refused it, a commit or Detach call returned the
184-
// state, or atelet gave up after reportQueue.giveUpAfter.
183+
// acknowledged or refused it, a commit returned the state, or atelet gave
184+
// up after reportQueue.giveUpAfter.
185185
ReportDueSince time.Time `json:"reportDueSince,omitzero"`
186186
}
187187

‎cmd/atelet/internal/snapshottracker/background.go‎

Lines changed: 22 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,13 @@ const (
3535
// DefaultDetachDelay is how long an actor stays suspended on this node by
3636
// default, its latest snapshot kept locally, before atelet detaches it.
3737
DefaultDetachDelay = 30 * time.Minute
38-
// DefaultUploadTimeout is how long one background upload attempt,
39-
// including the one a background detach runs first, may run by default,
40-
// on top of the time its snapshot takes at the estimated upload
41-
// throughput, before atelet cancels it. It keeps a wedged object-store
42-
// call from holding a background worker forever.
38+
// DefaultUploadTimeout is how long an upload attempt, background or part
39+
// of a detach, may run beyond its estimated upload time before atelet
40+
// cancels it, so a wedged object-store call can't hold it forever.
4341
DefaultUploadTimeout = 2 * time.Minute
4442
// DefaultUploadEstThroughput, in bytes per second of a snapshot's
45-
// allocated local size, is how fast atelet expects a background upload to
46-
// run by default. Each attempt's timeout allows for the snapshot to upload
43+
// allocated local size, is how fast atelet expects an upload to run by
44+
// default. Each attempt's timeout allows for the snapshot to upload
4745
// at this rate, so a large snapshot is not cut off.
4846
DefaultUploadEstThroughput = 100 << 20
4947
// maxUploadTimeoutDoublings caps how many times consecutive failures of
@@ -87,10 +85,9 @@ func (k opKind) String() string {
8785
}
8886
}
8987

90-
// logOp logs how an upload or detach ended. Foreground failures are returned
91-
// to the RPC caller, so only background failures are logged here: as a
92-
// warning, or at info level once ctx has ended, as when Terminate, an atelet
93-
// shutdown, or a newer durable snapshot cancels the operation.
88+
// logOp logs how an upload or detach ended. A failure logs as a warning, or at
89+
// info level once ctx has ended, as when Terminate, an atelet shutdown, or a
90+
// newer durable snapshot cancels the operation.
9491
func logOp(ctx context.Context, actorUID string, kind opKind, target SnapshotRef, uploaded, background bool, elapsed time.Duration, err error) {
9592
attrs := []any{
9693
slog.String("op", kind.String()),
@@ -103,7 +100,6 @@ func logOp(ctx context.Context, actorUID string, kind opKind, target SnapshotRef
103100
switch {
104101
case err == nil:
105102
slog.InfoContext(ctx, "Snapshot operation finished", append(attrs, slog.Bool("uploaded", uploaded))...)
106-
case !background:
107103
case ctx.Err() != nil:
108104
slog.InfoContext(ctx, "Snapshot operation canceled", append(attrs, slog.Any("err", err))...)
109105
default:
@@ -114,9 +110,8 @@ func logOp(ctx context.Context, actorUID string, kind opKind, target SnapshotRef
114110
// actorOp is an actor's operation in progress: an upload or a detach. An
115111
// actor has at most one, of any kind.
116112
//
117-
// A worker runs it in the background once it is due. An atelet RPC such as
118-
// Detach runs it in the RPC's goroutine instead, without going through the
119-
// background queue.
113+
// A worker runs it in the background once it is due. A Detach call begins one
114+
// directly instead, and it runs on if the caller leaves.
120115
//
121116
// The operation holds the actor's lock only while it begins and ends, not
122117
// while it runs:
@@ -139,6 +134,9 @@ type actorOp struct {
139134
// done is closed once the operation has ended, and has finished any change
140135
// it makes to the actor's local snapshots.
141136
done chan struct{}
137+
// state and err are a detach's result, set before done is closed.
138+
state *ateletpb.ActorSnapshotState
139+
err error
142140
}
143141

144142
// Start reads each actor's snapshot state file on this node, queues the actor
@@ -177,9 +175,13 @@ func (t *Tracker) Start() {
177175
}
178176
}
179177

180-
// Wait blocks until the workers have stopped, which they do once the tracker's
181-
// context ends, or until ctx ends. It returns ctx's error if ctx ended first.
178+
// Wait blocks until the workers and any detach a Detach call started have
179+
// stopped, which they do once the tracker's context ends. It returns ctx's
180+
// error if ctx ends first. Detach starts nothing after Wait begins.
182181
func (t *Tracker) Wait(ctx context.Context) error {
182+
t.foregroundMu.Lock()
183+
t.waiting = true
184+
t.foregroundMu.Unlock()
183185
stopped := make(chan struct{})
184186
go func() {
185187
t.workersDone.Wait()
@@ -235,8 +237,7 @@ func (t *Tracker) beginDueWork(actorUID string, kind opKind) func() {
235237
a := t.findLocked(actorUID)
236238
if a == nil || a.op != nil {
237239
// Forgotten, or an operation is already in progress on the actor: one
238-
// a worker of the other queue began, or one an atelet RPC such as
239-
// Detach runs in the RPC's goroutine. When that operation ends,
240+
// a worker of the other queue or a Detach call began. When it ends,
240241
// requeueLocked queues the actor again.
241242
return nil
242243
}
@@ -256,9 +257,9 @@ func (t *Tracker) beginDueWork(actorUID string, kind opKind) func() {
256257
// (see holdLocked).
257258
return nil
258259
}
259-
if finish, err := t.beginDetachLocked(t.ctx, a, true); err == nil {
260+
if _, run, err := t.beginDetachLocked(t.ctx, a, true); err == nil {
260261
t.instruments.recordDetachOverdue(t.ctx, now.Sub(due))
261-
return func() { _, _ = finish() }
262+
return run
262263
}
263264
}
264265
case opUpload:

‎cmd/atelet/internal/snapshottracker/background_test.go‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -440,9 +440,8 @@ func (h *recordingHandler) WithAttrs([]slog.Attr) slog.Handler { return h }
440440

441441
func (h *recordingHandler) WithGroup(string) slog.Handler { return h }
442442

443-
// logOp logs a finished operation at info level, a failed background one as a
444-
// warning, and a background one canceled by its context at info level. A
445-
// failed foreground operation goes back to its caller unlogged.
443+
// logOp logs a finished operation at info level, a failed one as a warning, and
444+
// one its context canceled at info level, in the background or not.
446445
func TestLogOp(t *testing.T) {
447446
canceled, cancel := context.WithCancel(context.Background())
448447
cancel()
@@ -457,7 +456,8 @@ func TestLogOp(t *testing.T) {
457456
{name: "finished", ctx: context.Background(), background: true, wantLevel: slog.LevelInfo, wantMsg: "Snapshot operation finished"},
458457
{name: "background failure", ctx: context.Background(), background: true, err: errors.New("boom"), wantLevel: slog.LevelWarn, wantMsg: "Snapshot operation failed"},
459458
{name: "background canceled", ctx: canceled, background: true, err: context.Canceled, wantLevel: slog.LevelInfo, wantMsg: "Snapshot operation canceled"},
460-
{name: "foreground failure", ctx: context.Background(), err: errors.New("boom")},
459+
{name: "foreground failure", ctx: context.Background(), err: errors.New("boom"), wantLevel: slog.LevelWarn, wantMsg: "Snapshot operation failed"},
460+
{name: "foreground canceled", ctx: canceled, err: context.Canceled, wantLevel: slog.LevelInfo, wantMsg: "Snapshot operation canceled"},
461461
} {
462462
t.Run(tc.name, func(t *testing.T) {
463463
h := &recordingHandler{}
@@ -468,12 +468,6 @@ func TestLogOp(t *testing.T) {
468468
logOp(tc.ctx, trackedActorUID, opUpload, testRef("s1", 1, time.Time{}), false, tc.background, time.Second, tc.err)
469469
h.mu.Lock()
470470
defer h.mu.Unlock()
471-
if tc.wantMsg == "" {
472-
if len(h.records) != 0 {
473-
t.Errorf("logged %q, want nothing", h.records[0].Message)
474-
}
475-
return
476-
}
477471
if len(h.records) != 1 || h.records[0].Message != tc.wantMsg || h.records[0].Level != tc.wantLevel {
478472
t.Errorf("logged %v, want one %v record %q", h.records, tc.wantLevel, tc.wantMsg)
479473
}

‎cmd/atelet/internal/snapshottracker/detach.go‎

Lines changed: 48 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -22,28 +22,30 @@ import (
2222
)
2323

2424
// beginDetachLocked begins a detach of the actor's latest local snapshot,
25-
// under a context derived from ctx, and returns the func that finishes it
26-
// once the caller releases the actor's lock. It begins nothing, and fails,
25+
// under a context derived from ctx, and returns its op and the func that runs
26+
// it once the caller releases the actor's lock. It begins nothing, and fails,
2727
// when no detach can make the snapshot durable (see detachTarget).
28-
func (t *Tracker) beginDetachLocked(ctx context.Context, a *actorSnapshots, background bool) (func() (*ateletpb.ActorSnapshotState, error), error) {
28+
func (t *Tracker) beginDetachLocked(ctx context.Context, a *actorSnapshots, background bool) (*actorOp, func(), error) {
2929
target, alreadyDurable, err := a.detachTarget()
3030
if err != nil {
31-
return nil, err
31+
return nil, nil, err
3232
}
3333
opCtx, op := t.beginOpLocked(ctx, a, opDetach, target)
34-
return func() (*ateletpb.ActorSnapshotState, error) {
35-
return t.finishDetach(opCtx, a.actorUID, op, alreadyDurable, background)
34+
return op, func() {
35+
defer close(op.done)
36+
op.state, op.err = t.finishDetach(opCtx, a.actorUID, op, alreadyDurable, background)
3637
}, nil
3738
}
3839

3940
// Detach makes the actor's latest local snapshot durable, then deletes the
4041
// actor's local snapshots. It is a no-op when atelet holds no local snapshot
4142
// for the actor. It fails with Aborted once a Run or Restore of the actor has
4243
// begun since its latest local snapshot, and with FailedPrecondition when no
43-
// detach can make the snapshot durable. It runs in the calling goroutine
44-
// rather than through the background queue, once any operation in progress on
45-
// the actor has ended. Callers serialize it with the actor's other lifecycle
46-
// RPCs (see Tracker).
44+
// detach can make the snapshot durable. It starts once the actor's operation
45+
// in progress ends, and runs on the tracker's context, not ctx. If ctx ends
46+
// first, Detach returns ctx's error and the detach runs on, recorded by its
47+
// report. After Wait begins, Detach fails with Unavailable. Callers serialize
48+
// it with the actor's other lifecycle RPCs (see Tracker).
4749
func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorSnapshotState, error) {
4850
if t.store == nil {
4951
return nil, apierror.Unimplemented("atelet has no snapshot store to detach actors with")
@@ -59,12 +61,10 @@ func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorS
5961
}
6062
if a == nil || a.LatestLocal == nil {
6163
// No local snapshot is stored on this node, so there is no work to be
62-
// done.
64+
// done. A report already due stays due, in case the caller doesn't
65+
// record this state.
6366
st := &ateletpb.ActorSnapshotState{}
6467
if a != nil {
65-
if a.reportDue() {
66-
t.clearReportDueLocked(a)
67-
}
6868
st = t.stateLocked(a)
6969
}
7070
t.unlock(actorUID)
@@ -76,21 +76,45 @@ func (t *Tracker) Detach(ctx context.Context, actorUID string) (*ateletpb.ActorS
7676
t.unlock(actorUID)
7777
return nil, apierror.Aborted("actor %s is in use on this node", actorUID)
7878
}
79-
finish, err := t.beginDetachLocked(ctx, a, false)
80-
t.unlock(actorUID)
79+
// Under foregroundMu, so no detach joins workersDone after Wait begins.
80+
t.foregroundMu.Lock()
81+
if t.waiting || t.ctx.Err() != nil {
82+
t.foregroundMu.Unlock()
83+
t.unlock(actorUID)
84+
return nil, apierror.Unavailable("atelet is shutting down")
85+
}
86+
op, run, err := t.beginDetachLocked(t.ctx, a, false)
8187
if err != nil {
88+
t.foregroundMu.Unlock()
89+
t.unlock(actorUID)
8290
return nil, err
8391
}
84-
return finish()
92+
t.workersDone.Go(run)
93+
t.foregroundMu.Unlock()
94+
t.unlock(actorUID)
95+
96+
select {
97+
case <-op.done:
98+
return op.state, t.detachCallError(ctx, op.err)
99+
case <-ctx.Done():
100+
return nil, ctx.Err()
101+
}
102+
}
103+
104+
// detachCallError returns err, or Unavailable if the tracker's shutdown ended
105+
// the detach, so the caller doesn't mistake it for its own cancellation.
106+
func (t *Tracker) detachCallError(ctx context.Context, err error) error {
107+
if err != nil && t.ctx.Err() != nil && ctx.Err() == nil {
108+
return apierror.Unavailable("atelet stopped during the detach: %v", err)
109+
}
110+
return err
85111
}
86112

87113
// finishDetach runs op, a detach of the actor's latest local snapshot: it
88114
// uploads the snapshot unless already durable, then deletes the actor's local
89115
// snapshots. background reports whether a worker runs it, rather than a Detach
90-
// call; a worker's detach makes a report of what it changed due, while a
91-
// Detach call returns the new state and cancels any pending report.
116+
// call. Either way it makes a report of what it changed due.
92117
func (t *Tracker) finishDetach(ctx context.Context, actorUID string, op *actorOp, alreadyDurable, background bool) (*ateletpb.ActorSnapshotState, error) {
93-
defer close(op.done)
94118
uploaded := false
95119
target := op.target
96120
detachStart := t.clock.Now()
@@ -102,7 +126,7 @@ func (t *Tracker) finishDetach(ctx context.Context, actorUID string, op *actorOp
102126
}
103127
if err == nil {
104128
t.updateOp(actorUID, func(a *actorSnapshots) {
105-
if t.recordUploadLocked(a, target, uploaded) && background {
129+
if t.recordUploadLocked(a, target, uploaded) {
106130
t.markReportDueLocked(a)
107131
}
108132
})
@@ -119,7 +143,8 @@ func (t *Tracker) finishDetach(ctx context.Context, actorUID string, op *actorOp
119143
recordDetach := err == nil
120144
if recordDetach {
121145
detachStart = t.clock.Now()
122-
// Forget or the caller may have ended ctx as the upload finished.
146+
// Forget or the tracker's shutdown may have ended ctx as the upload
147+
// finished.
123148
err = ctx.Err()
124149
}
125150
if err == nil {
@@ -129,11 +154,7 @@ func (t *Tracker) finishDetach(ctx context.Context, actorUID string, op *actorOp
129154
switch {
130155
case err == nil:
131156
t.clearLatestLocalLocked(a)
132-
if background {
133-
t.markReportDueLocked(a)
134-
} else {
135-
t.cancelReportLocked(a)
136-
}
157+
t.markReportDueLocked(a)
137158
case ctx.Err() == nil:
138159
// Detach again once the retry delay has passed, and no sooner
139160
// than the detach was due.

0 commit comments

Comments
 (0)