Skip to content

Commit 31ae728

Browse files
teemowQuentinBisson
authored andcommitted
ateapi, atelet: make a pause durable — upload its snapshot in the background and resume it on any worker
A pause checkpoints the workload onto its worker's node and nowhere else: the node holds the only copy of the actor's state, and the record pins the actor to that node (LocalSnapshotInfo.node_vms_with_local_snapshots → Constraints.RequiredNodes). A node that goes away — preempted, scaled down, rolled — takes the state along while the record still says PAUSED there. The next resume then finds no worker that applies, the scheduler reports the locality miss as ResourceExhausted "no free workers available" (the message of a full pool), the router parks the request for its whole budget and every retry repeats this; a suspend dials the atelet on the dead node and stays retryable forever, so the actor cannot be suspended or deleted either. Actors parked on a question for a person sit like this for hours. Pauses now become durable shortly after they complete. After FinalizePaused the control plane queues the actor with a background uploader (two workers per replica, rate-limited requeues, a resync every minute that also finds pauses from before an upgrade or a restart). The upload goes through the atelet on the snapshot's node, the path a suspend of a paused actor already uses, with two differences: the destination is named after the local snapshot, so a retry re-sends the same request and overwrites the same objects, and the atelet keeps the local copy (UploadPausedCheckpointRequest .keep_local) — the actor stays PAUSED and its node remains the fast path. On success the copy is recorded as the actor's external snapshot with the local snapshot it came from (ExternalSnapshot.source_local_snapshot_name), under a version precondition: no lease is held while the checkpoint streams, a resume that arrives meanwhile is not delayed, and an upload the actor has outrun is deleted again. The external snapshot the copy replaces is released once the record points at the copy. That marker is what the rest keys on. A paused actor's external snapshot is its durable copy only while it names the local snapshot the actor holds; the snapshot of an earlier suspend, or the upload of an earlier pause, is older state and never stands in for the pause. While no copy exists the node stays a hard constraint, exactly as before. Once one exists: - Placement prefers the snapshot's nodes (Constraints.PreferredNodes, ranked ahead of every other eligible worker, excluding none) instead of requiring them. Restore takes the local snapshot when the chosen worker is on such a node and the durable copy — with the scope the pause captured, memory included — anywhere else. A paused actor placed off its node with no copy is refused before any atelet is dialed rather than restored from an older snapshot. - A suspend commits the copy as captured: nothing is uploaded, no destination minted, the node not dialed except to drop the local copy best-effort (the new PruneLocalCheckpoints RPC, bounded to ten seconds). Narrowing the copy to the template's commit scope would need the node the actor may be leaving. - Deleting a paused actor collects the copy with the actor's prefix, as for a suspended actor, and asks the node to drop the local copy where an atelet answers. The upload uses the scope the pause captured, not the commit scope: the copy must restore exactly as the local snapshot would. Tests cover the marker, the resync selection and its grace, the constraint switch, the record commit with the replaced snapshot released, the stale upload discarded and the concurrent replica's record left alone, the suspend shortcut at mark and upload, the placement guard, the scheduler's preference and its fallbacks, and — end to end against the fake atelet — the upload after a pause, the resume onto another node restoring the copy after the snapshot's node lost its worker, the resume that still prefers the node, the suspend that uploads nothing, and the delete that releases both copies. Signed-off-by: Timo Derstappen <[email protected]>
1 parent 4ced1ec commit 31ae728

29 files changed

Lines changed: 1997 additions & 108 deletions

‎benchmarking/locust/common/ateapi_pb2_grpc.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -255,20 +255,28 @@ def SuspendActor(self, request, context):
255255
"""Suspend a given actor to a new snapshot. A running actor is checkpointed
256256
on its worker; a paused actor's node-local snapshot is uploaded, narrowed
257257
to the template's commit scope where required (Full capture, Data commit).
258+
A paused actor whose pause snapshot already has a durable copy (see
259+
ExternalSnapshot.source_local_snapshot_name) commits that copy as it was
260+
captured, without involving the node.
258261
"""
259262
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
260263
context.set_details('Method not implemented!')
261264
raise NotImplementedError('Method not implemented!')
262265

263266
def PauseActor(self, request, context):
264-
"""Pause a given actor and keep its snapshots on node VM.
267+
"""Pause a given actor and keep its snapshots on node VM. The control plane
268+
then uploads the pause snapshot to durable storage in the background and
269+
records the copy as the actor's external_snapshot; until that completes
270+
the actor can resume only on its node.
265271
"""
266272
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
267273
context.set_details('Method not implemented!')
268274
raise NotImplementedError('Method not implemented!')
269275

270276
def ResumeActor(self, request, context):
271-
"""Resume an actor from its latest snapshot.
277+
"""Resume an actor from its latest snapshot. A paused actor resumes from its
278+
node-local snapshot when a worker on that node is free, and from the
279+
snapshot's durable copy on any other eligible worker once one exists.
272280
"""
273281
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
274282
context.set_details('Method not implemented!')

‎cmd/ateapi/internal/controlapi/durable_pause.go‎

Lines changed: 411 additions & 0 deletions
Large diffs are not rendered by default.

‎cmd/ateapi/internal/controlapi/durable_pause_test.go‎

Lines changed: 523 additions & 0 deletions
Large diffs are not rendered by default.

‎cmd/ateapi/internal/controlapi/functionaltest/actor_test.go‎

Lines changed: 291 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3859,7 +3859,7 @@ func TestResumeActor_PausedOnGoneNodeCrashesAndDeletes(t *testing.T) {
38593859
}); err != nil {
38603860
t.Fatalf("ResumeActor kept answering %v after the node was deleted", resumeErr)
38613861
}
3862-
assertGrpcError(t, resumeErr, codes.DataLoss, fmt.Sprintf("actor %s/%s crashed: its local snapshot %q is on node(s) [%s], which no longer exist in the cluster, and it has no durable snapshot", testAtespace, name, paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(), node))
3862+
assertGrpcError(t, resumeErr, codes.DataLoss, fmt.Sprintf("actor %s/%s crashed: its local snapshot %q is on node(s) [%s], which no longer exist in the cluster, and it has no durable copy of it", testAtespace, name, paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(), node))
38633863
if !ateerrors.ActorCrashRequested(resumeErr) {
38643864
t.Errorf("ResumeActor error carries no crash directive: %v", resumeErr)
38653865
}
@@ -3880,3 +3880,293 @@ func TestResumeActor_PausedOnGoneNodeCrashesAndDeletes(t *testing.T) {
38803880
t.Fatalf("DeleteActor of the crashed actor failed: %v", err)
38813881
}
38823882
}
3883+
3884+
// startPauseUploads runs the service's background pause snapshot uploads for
3885+
// the test. Started only by the tests that assert on them, so every other
3886+
// test observes the atelet traffic of its own calls alone. The returned stop
3887+
// ends the uploads; call it before the test context is torn down.
3888+
func startPauseUploads(t *testing.T, tc *testContext) (stop func()) {
3889+
t.Helper()
3890+
ctx, cancel := context.WithCancel(context.Background())
3891+
tc.service.Start(ctx)
3892+
return cancel
3893+
}
3894+
3895+
// waitForDurablePause polls until the actor's pause snapshot has a durable
3896+
// copy: an external snapshot uploaded from the local snapshot it is paused on.
3897+
func waitForDurablePause(t *testing.T, tc *testContext, name string) *ateapipb.Actor {
3898+
t.Helper()
3899+
var actor *ateapipb.Actor
3900+
err := wait.PollUntilContextTimeout(context.Background(), 20*time.Millisecond, 10*time.Second, true, func(ctx context.Context) (bool, error) {
3901+
got, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{
3902+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name},
3903+
})
3904+
if err != nil {
3905+
return false, err
3906+
}
3907+
st := got.GetStatus()
3908+
if st.GetState() != ateapipb.ActorState_ACTOR_STATE_PAUSED || st.GetExternalSnapshot().GetSourceLocalSnapshotName() == "" ||
3909+
st.GetExternalSnapshot().GetSourceLocalSnapshotName() != st.GetLocalSnapshotInfo().GetSnapshotName() {
3910+
return false, nil
3911+
}
3912+
actor = got
3913+
return true, nil
3914+
})
3915+
if err != nil {
3916+
t.Fatalf("the pause snapshot of %s never got a durable copy: %v", name, err)
3917+
}
3918+
return actor
3919+
}
3920+
3921+
// pauseOnNode1 creates an actor, runs it on node1's only worker and pauses it
3922+
// there, returning the paused record.
3923+
func pauseOnNode1(t *testing.T, tc *testContext, name string) *ateapipb.Actor {
3924+
t.Helper()
3925+
if _, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
3926+
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name},
3927+
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
3928+
}}); err != nil {
3929+
t.Fatalf("CreateActor(%s) failed: %v", name, err)
3930+
}
3931+
if _, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{
3932+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name},
3933+
}); err != nil {
3934+
t.Fatalf("ResumeActor(%s) failed: %v", name, err)
3935+
}
3936+
if _, err := tc.client.PauseActor(context.Background(), &ateapipb.PauseActorRequest{
3937+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name},
3938+
}); err != nil {
3939+
t.Fatalf("PauseActor(%s) failed: %v", name, err)
3940+
}
3941+
paused, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{
3942+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name},
3943+
})
3944+
if err != nil {
3945+
t.Fatalf("GetActor(%s) failed: %v", name, err)
3946+
}
3947+
if got := paused.GetStatus().GetLocalSnapshotInfo().GetNodeVmsWithLocalSnapshots(); len(got) != 1 || got[0] != "node1" {
3948+
t.Fatalf("paused actor pinned to %v, want [node1]", got)
3949+
}
3950+
return paused
3951+
}
3952+
3953+
// TestPauseActor_MakesPauseDurable verifies a pause is followed by a
3954+
// background upload of its node-local snapshot: the atelet on the node is
3955+
// asked to upload the checkpoint and keep the local copy, and the actor —
3956+
// still PAUSED on its node — records the copy as its external snapshot,
3957+
// marked with the local snapshot it came from.
3958+
func TestPauseActor_MakesPauseDurable(t *testing.T) {
3959+
ns := namespaceForTest("ns-pause-durable")
3960+
tc := setupTest(t, ns)
3961+
defer tc.cleanup()
3962+
defer startPauseUploads(t, tc)()
3963+
3964+
createTemplate(t, tc, ns)
3965+
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
3966+
3967+
paused := pauseOnNode1(t, tc, "id1")
3968+
durable := waitForDurablePause(t, tc, "id1")
3969+
3970+
upload := tc.fakeAtelet.lastUploadRequest()
3971+
if upload == nil {
3972+
t.Fatal("expected atelet UploadPausedCheckpoint to be called")
3973+
}
3974+
if !upload.GetKeepLocal() {
3975+
t.Error("upload keep_local = false, want true: the actor stays paused on its node")
3976+
}
3977+
if got, want := upload.GetLocalSnapshotName(), paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(); got != want {
3978+
t.Errorf("upload local_snapshot_name = %q, want the pause snapshot %q", got, want)
3979+
}
3980+
if got := upload.GetDesiredScope(); got != ateletpb.SnapshotScope_SNAPSHOT_SCOPE_FULL {
3981+
t.Errorf("upload desired_scope = %v, want FULL (what the pause captured)", got)
3982+
}
3983+
3984+
external := durable.GetStatus().GetExternalSnapshot()
3985+
if got, want := external.GetSnapshotUri(), upload.GetDestinationSnapshotUri(); got != want {
3986+
t.Errorf("external snapshot URI = %q, want the upload destination %q", got, want)
3987+
}
3988+
if got := external.GetContentScope(); got != ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL {
3989+
t.Errorf("external snapshot ContentScope = %v, want FULL", got)
3990+
}
3991+
assertSnapshotOwnedByActor(t, durable, external.GetSnapshotUri())
3992+
assertSnapshotPresent(t, tc, external.GetSnapshotUri())
3993+
if diff := cmp.Diff(paused.GetStatus().GetLocalSnapshotInfo(), durable.GetStatus().GetLocalSnapshotInfo(), protocmp.Transform()); diff != "" {
3994+
t.Errorf("LocalSnapshotInfo changed by the upload (-paused +durable):\n%s", diff)
3995+
}
3996+
}
3997+
3998+
// TestResumeActor_PausedRestoresDurableCopyOffItsNode is the node-loss case:
3999+
// the worker on the paused actor's node is gone, another node has a free
4000+
// worker. With the pause snapshot's durable copy the actor resumes there,
4001+
// restoring that copy, instead of waiting out a capacity error that cannot
4002+
// clear.
4003+
func TestResumeActor_PausedRestoresDurableCopyOffItsNode(t *testing.T) {
4004+
ns := namespaceForTest("ns-resume-durable-copy")
4005+
tc := setupTest(t, ns)
4006+
defer tc.cleanup()
4007+
defer startPauseUploads(t, tc)()
4008+
4009+
createTemplate(t, tc, ns)
4010+
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
4011+
4012+
pauseOnNode1(t, tc, "id1")
4013+
durable := waitForDurablePause(t, tc, "id1")
4014+
4015+
// node1 goes away with its worker; node2 comes up with a free one.
4016+
deleteWorkerPod(t, tc, ns, "worker-1")
4017+
createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1")
4018+
setupAteletOnNode(t, tc, "atelet-node2", "node2")
4019+
tc.fakeAtelet.Reset()
4020+
4021+
resumed, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{
4022+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
4023+
})
4024+
if err != nil {
4025+
t.Fatalf("ResumeActor off the snapshot's node failed: %v", err)
4026+
}
4027+
if got := resumed.GetActor().GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_RUNNING {
4028+
t.Errorf("state = %v, want RUNNING", got)
4029+
}
4030+
if got := resumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-2" {
4031+
t.Errorf("resumed onto worker %q, want worker-2 (the worker on node2)", got)
4032+
}
4033+
restore := tc.fakeAtelet.lastRestoreRequest()
4034+
if restore == nil {
4035+
t.Fatal("expected atelet Restore to be called")
4036+
}
4037+
if got := restore.GetType(); got != ateletpb.CheckpointType_CHECKPOINT_TYPE_EXTERNAL {
4038+
t.Errorf("restore type = %v, want EXTERNAL (the durable copy)", got)
4039+
}
4040+
if got, want := restore.GetExternalConfig().GetSnapshotUri(), durable.GetStatus().GetExternalSnapshot().GetSnapshotUri(); got != want {
4041+
t.Errorf("restore snapshot URI = %q, want the pause's durable copy %q", got, want)
4042+
}
4043+
if got := restore.GetScope(); got != ateletpb.SnapshotScope_SNAPSHOT_SCOPE_FULL {
4044+
t.Errorf("restore scope = %v, want FULL: the copy restores as the pause would", got)
4045+
}
4046+
}
4047+
4048+
// TestResumeActor_PausedPrefersItsNodeOverOtherFreeWorkers verifies the
4049+
// durable copy does not cost the fast path: with free workers on the
4050+
// snapshot's node and elsewhere, the paused actor lands on its node and
4051+
// restores the local snapshot.
4052+
func TestResumeActor_PausedPrefersItsNodeOverOtherFreeWorkers(t *testing.T) {
4053+
ns := namespaceForTest("ns-resume-prefer-node")
4054+
tc := setupTest(t, ns)
4055+
defer tc.cleanup()
4056+
defer startPauseUploads(t, tc)()
4057+
4058+
createTemplate(t, tc, ns)
4059+
workerName := createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
4060+
4061+
paused := pauseOnNode1(t, tc, "id1")
4062+
waitForDurablePause(t, tc, "id1")
4063+
waitForWorkerAvailable(t, tc, workerName)
4064+
createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1")
4065+
setupAteletOnNode(t, tc, "atelet-node2", "node2")
4066+
tc.fakeAtelet.Reset()
4067+
4068+
resumed, err := tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{
4069+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
4070+
})
4071+
if err != nil {
4072+
t.Fatalf("ResumeActor failed: %v", err)
4073+
}
4074+
if got := resumed.GetActor().GetStatus().GetWorkerAssignment().GetWorkerPod(); got != "worker-1" {
4075+
t.Errorf("resumed onto worker %q, want worker-1 (the worker on the snapshot's node)", got)
4076+
}
4077+
restore := tc.fakeAtelet.lastRestoreRequest()
4078+
if restore == nil {
4079+
t.Fatal("expected atelet Restore to be called")
4080+
}
4081+
if got := restore.GetType(); got != ateletpb.CheckpointType_CHECKPOINT_TYPE_LOCAL {
4082+
t.Errorf("restore type = %v, want LOCAL (the node-local snapshot)", got)
4083+
}
4084+
if got, want := restore.GetLocalConfig().GetSnapshotName(), paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(); got != want {
4085+
t.Errorf("restore local snapshot = %q, want %q", got, want)
4086+
}
4087+
}
4088+
4089+
// TestSuspendActor_PausedWithDurableCopy_CommitsItWithoutTheNode verifies a
4090+
// suspend of a paused actor whose pause snapshot has a durable copy uploads
4091+
// nothing: the copy becomes the suspended actor's snapshot, the node pinning
4092+
// ends, and the node is only asked — best-effort — to drop the local copy.
4093+
func TestSuspendActor_PausedWithDurableCopy_CommitsItWithoutTheNode(t *testing.T) {
4094+
ns := namespaceForTest("ns-suspend-durable-copy")
4095+
tc := setupTest(t, ns)
4096+
defer tc.cleanup()
4097+
defer startPauseUploads(t, tc)()
4098+
4099+
createTemplate(t, tc, ns)
4100+
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
4101+
4102+
pauseOnNode1(t, tc, "id1")
4103+
durable := waitForDurablePause(t, tc, "id1")
4104+
copyURI := durable.GetStatus().GetExternalSnapshot().GetSnapshotUri()
4105+
tc.fakeAtelet.Reset()
4106+
4107+
suspended, err := tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{
4108+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
4109+
})
4110+
if err != nil {
4111+
t.Fatalf("SuspendActor failed: %v", err)
4112+
}
4113+
if tc.fakeAtelet.lastUploadRequest() != nil {
4114+
t.Error("atelet UploadPausedCheckpoint called; the pause snapshot already had a durable copy")
4115+
}
4116+
prune := tc.fakeAtelet.lastPruneRequest()
4117+
if prune == nil {
4118+
t.Error("expected atelet PruneLocalCheckpoints to be called for the local copy")
4119+
} else if got, want := prune.GetActorUid(), durable.GetMetadata().GetUid(); got != want {
4120+
t.Errorf("prune actor_uid = %q, want %q", got, want)
4121+
}
4122+
4123+
actor := suspended.GetActor()
4124+
if actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED {
4125+
t.Errorf("state = %v, want SUSPENDED", actor.GetStatus().GetState())
4126+
}
4127+
if got := actor.GetStatus().GetExternalSnapshot().GetSnapshotUri(); got != copyURI {
4128+
t.Errorf("snapshot URI = %q, want the durable copy %q kept", got, copyURI)
4129+
}
4130+
if actor.GetStatus().GetLocalSnapshotInfo() != nil {
4131+
t.Errorf("LocalSnapshotInfo = %v, want cleared", actor.GetStatus().GetLocalSnapshotInfo())
4132+
}
4133+
assertSnapshotPresent(t, tc, copyURI)
4134+
}
4135+
4136+
// TestDeleteActor_PausedWithDurableCopy_ReleasesBothCopies verifies deleting
4137+
// a paused actor collects the durable copy from object storage and asks the
4138+
// node to drop the local one.
4139+
func TestDeleteActor_PausedWithDurableCopy_ReleasesBothCopies(t *testing.T) {
4140+
ns := namespaceForTest("ns-delete-durable-copy")
4141+
tc := setupTest(t, ns)
4142+
defer tc.cleanup()
4143+
defer startPauseUploads(t, tc)()
4144+
4145+
createTemplate(t, tc, ns)
4146+
createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1")
4147+
4148+
pauseOnNode1(t, tc, "id1")
4149+
durable := waitForDurablePause(t, tc, "id1")
4150+
copyURI := durable.GetStatus().GetExternalSnapshot().GetSnapshotUri()
4151+
tc.fakeAtelet.Reset()
4152+
4153+
if _, err := tc.client.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{
4154+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
4155+
AnyState: true,
4156+
}); err != nil {
4157+
t.Fatalf("DeleteActor failed: %v", err)
4158+
}
4159+
assertSnapshotCollected(t, tc, copyURI)
4160+
prune := tc.fakeAtelet.lastPruneRequest()
4161+
if prune == nil {
4162+
t.Error("expected atelet PruneLocalCheckpoints to be called for the local copy")
4163+
} else if got, want := prune.GetActorUid(), durable.GetMetadata().GetUid(); got != want {
4164+
t.Errorf("prune actor_uid = %q, want %q", got, want)
4165+
}
4166+
_, err := tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{
4167+
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"},
4168+
})
4169+
if status.Code(err) != codes.NotFound {
4170+
t.Errorf("GetActor after delete = %v, want NotFound", err)
4171+
}
4172+
}

‎cmd/ateapi/internal/controlapi/functionaltest/main_test.go‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,8 @@ type FakeAteletServer struct {
130130
TerminateCalled bool
131131
TerminateRequest *ateletpb.TerminateRequest
132132
FailTerminate error
133+
PruneCalled bool
134+
PruneRequest *ateletpb.PruneLocalCheckpointsRequest
133135

134136
// objectStore, when set, receives the objects a checkpoint or an upload
135137
// writes, so the control plane's copy and release steps have real external
@@ -181,10 +183,45 @@ func (f *FakeAteletServer) Reset() {
181183
f.TerminateCalled = false
182184
f.TerminateRequest = nil
183185
f.FailTerminate = nil
186+
f.PruneCalled = false
187+
f.PruneRequest = nil
184188

185189
f.objectStore = nil
186190
}
187191

192+
// lastUploadRequest returns the last UploadPausedCheckpoint request, or nil.
193+
// Read under the lock: the control plane's pause uploads arrive in the
194+
// background.
195+
func (f *FakeAteletServer) lastUploadRequest() *ateletpb.UploadPausedCheckpointRequest {
196+
f.Lock.Lock()
197+
defer f.Lock.Unlock()
198+
199+
if f.UploadRequest == nil {
200+
return nil
201+
}
202+
return proto.Clone(f.UploadRequest).(*ateletpb.UploadPausedCheckpointRequest)
203+
}
204+
205+
// lastPruneRequest returns the last PruneLocalCheckpoints request, or nil.
206+
func (f *FakeAteletServer) lastPruneRequest() *ateletpb.PruneLocalCheckpointsRequest {
207+
f.Lock.Lock()
208+
defer f.Lock.Unlock()
209+
210+
if f.PruneRequest == nil {
211+
return nil
212+
}
213+
return proto.Clone(f.PruneRequest).(*ateletpb.PruneLocalCheckpointsRequest)
214+
}
215+
216+
func (f *FakeAteletServer) PruneLocalCheckpoints(ctx context.Context, req *ateletpb.PruneLocalCheckpointsRequest) (*ateletpb.PruneLocalCheckpointsResponse, error) {
217+
f.Lock.Lock()
218+
defer f.Lock.Unlock()
219+
220+
f.PruneCalled = true
221+
f.PruneRequest = proto.Clone(req).(*ateletpb.PruneLocalCheckpointsRequest)
222+
return &ateletpb.PruneLocalCheckpointsResponse{}, nil
223+
}
224+
188225
func (f *FakeAteletServer) UploadPausedCheckpoint(ctx context.Context, req *ateletpb.UploadPausedCheckpointRequest) (*ateletpb.UploadPausedCheckpointResponse, error) {
189226
f.Lock.Lock()
190227
defer f.Lock.Unlock()

0 commit comments

Comments
 (0)