Repository navigation
[Bug]: Retrying a Checkpoint whose snapshot already committed crashes the actor #2836
Description
Bug
If ate-api-server retries Atelet.Checkpoint after atelet has already committed the snapshot, the actor is crashed even though its snapshot is intact.
Expected behaviour
Atelet.Checkpoint should be idempotent on (actor_uid, snapshot_uuid).
- If the tracker has already committed this UUID with the same generation, fidelity and URI, Checkpoint returns the current ActorSnapshotState without calling the worker.
- On the ate-api side: FinalizeSuspended then completes and the actor ends SUSPENDED.
Suggested fix
// snapshottracker/lifecycle.go
func (t *Tracker) BeginCheckpoint(ctx, actorUID, snap) (committed *ActorSnapshotState, end func(), err error) {
lock(actorUID)
a := waitForDetachLocked(ctx, actorUID, checkGeneration(snap))
if a != nil {
// same uuid as LatestLocal / LatestDurable / Uploading ?
if ref := a.committedAs(snap.uuid); ref != nil {
if ref.generation != snap.generation || ref.uri != snap.uri || ref.fidelity != snap.fidelity {
return nil, nil, FailedPrecondition("snapshot %s committed with different params")
}
return a.state(), noop, nil
}
}
return nil, holdLocked(actorUID, a, snap, false), nil
}
// atelet main.go Checkpoint
committed, end, err := s.snapshots.BeginCheckpoint(ctx, uid, req.Snapshot)
if err != nil { return nil, err }
defer end()
if committed != nil {
if req.UploadMode == IMMEDIATE && committed.LatestDurable.uuid != req.uuid {
// first attempt was DEFERRED, retry is sync (mode can change on retry)
return nil, FailedPrecondition(...) // or: upload from the local copy, see below
}
return &CheckpointResponse{State: committed}, nil
}
... // existing path unchanged
Activity
chw120 commentedon Oct 19, 2026
I was planning to fold this suggested fix into #2602, but noticed that #2849 has already implemented it (along with the
BeginStartwait andIMMEDIATEpromotion on retry). The only remaining piece in #2602 is detaching the in-flightAtelet.Checkpointfrom caller cancellation (context.WithoutCancel(ctx)afterBeginCheckpoint), so that when a caller times out mid-checkpoint, the first attempt continues to completion and commits the snapshot instead of being canceled after the worker has already deleted the workload. We can either let #2849 merge first and narrow #2602 down to just thatWithoutCancelchange, or include it directly in #2849.Separately, #2623 covers the
ateworker-side cleanup when an earlier checkpoint attempt fails mid-flight beforeateletcommits the snapshot — without it, the next retry intoateworkerstill fails on leftover partial files in gVisor'scheckpoint-state/(runsc checkpointexit status 128) or a partially moved base in microVM'srestore-state/.msau42 commentedon Oct 19, 2026
I'm exploring if there's ways to simplify #2849. It's getting a bit complex trying to keep tracking of ongoing RPCs and I'm exploring a few things:
chw120 commentedon Oct 19, 2026
Michelle Au (@msau42) Sounds good! In case it's useful for what you're exploring: earlier revisions of #2602 up until f42b97b actually used *actorlock.Locks directly on Atelet in cmd/atelet/main.go to serialize Checkpoint per actor (before we rebased onto #2778 and moved the wait into BeginCheckpoint).
Feel free to take a look at that commit to see if it aligns with what you have in mind — if the actorlock.Locks approach makes more sense, I can also revert #2602 back to that version (with context.WithoutCancel). Please let me know which way you think is better to approach this.
atelet: Serialize lifecycle RPCs per actor (agent-substrate#2890)
aditya-shantanu commentedon Oct 20, 2026
Another way to reach this on
main4b4fbb2, without any client cancel: an atelet pod restart while a sync suspend is in flight. Chaos run on a 3-node GKE cluster (gVisor, 12 Locust users doing sync suspend/resume of 256 MiB glutton actors),kubectl delete pod atelet-jvb9q --grace-period=0 --forceat 20:56:34Z.Actor
benchmark/sb-3f3ef4d5-dcd9-48ac-83c9-94fc6060bb11, all on the same node:The first Checkpoint reached the worker and
runsc checkpointran (pause mode, so the sandbox exits afterwards); only the atelet-side commit was lost with the pod. The client sawUnavailable, which the API documents as retriable, retried four seconds later, and the retry ranrunsc checkpointagainst a container that no longer exists: exit 128, classified non-retriable, actor CRASHED. The worker still had the finished checkpoint on disk.So the "already committed" case includes "committed on the worker but never acknowledged by atelet": the replacement atelet needs to recover (or at least detect) a completed worker-side checkpoint before issuing a new one, and a checkpoint attempt that finds no container should not be reported as a crash when the previous attempt's result is unknown. The rest of that atelet restart was benign: 72
Unavailableerrors over about 4 s, 5Internal ... selected atelet has no assigned IPs(#646), one actor leftDELETINGwhose retried delete succeeded (#1665), no other crashes.chw120 commentedon Oct 20, 2026
Thanks Aditya Shantanu (@aditya-shantanu) for catching the
ateletrestart case!Since
ateletandateworkerrun in separate pods, a checkpoint can be interrupted at three points along the pipeline, and each layer can own its own recovery:Interrupted while the worker is still taking the snapshot (workload still alive)
Completed on the worker, but
ateletrestarted before committing the snapshot (the case above, where the workload is already stopped and the finished snapshot is still on the worker)ateletretries after the workload has already exited, the worker can detect and return the completed on-disk snapshot instead of trying to checkpoint a stopped container again.Committed by
atelet, but the response to the control plane was lostateletanswer the retry directly from its committed snapshot state (uploading or finishing cleanup if needed) without calling the worker again.Aditya Shantanu (@aditya-shantanu) Zoe Zhao (@zoez7) Michelle Au (@msau42) WDYT of this split across the layers?
aditya-shantanu commentedon Oct 21, 2026
Third trigger, a 180 s network partition between ate-api-server and atelet (NetworkPolicy on the ate-api-server pods,
main4b4fbb2, 12 users of sync suspend/resume). Suspends in flight when the partition started failed withUnavailable ... connection reset by peer(5), the clients retried, and 4 actors crashed withrunsc checkpoint: exit status 128because the first checkpoint had already run on the worker. Resumes in flight stayed RESUMING for the duration of the partition and completed after it healed; everything else recovered and the invariants were clean 2 minutes after the heal. So the only lasting damage from a control-plane partition is this retry path.atelet: detach Checkpoint from caller cancellation once underway (age…
ateworker: reuse completed checkpoint when workload is already stopped (
chw120 commentedon Oct 24, 2026
All 4 PRs covering the checkpoint retry failure windows have now been merged into main. Verified and resolved both the caller timeout / network partition retry case and the mid-checkpoint atelet restart case. Close the bug.
eren23 commentedon Oct 24, 2026
Thanks for the completed-checkpoint replay fix in #3015. I tested two later interruption points on
1756c0f7cb344035889da699981eeff4c8aef755, which includes it. Both still prevent a deferred checkpoint retry from completing in a fixture using production atelet andRecordCheckpoint/ReuseCheckpoint.checkpoint-state/.stagedlocal-checkpoint/<uuid>, beforeCommitLocalCommitDurableThe fixture reconstructs the tracker from disk and retries the same snapshot.
ReuseCheckpointdeclines replay because recorded files have moved away fromcheckpoint-state/. The synthetic worker then returnsFailedPreconditioninstead of executing the real worker fallback. Unchanged hashes establish preservation only in this fixture; the real gVisor fallback clears the checkpoint directory before attempting another capture. This test neither kills atelet nor runs tracker startup/background workers or the control-plane workflow.All five selected baseline packages passed with
-race, without skips. Three separate real gVisor MEMORY capture/replay/restore trials also passed, with one capture each, a matching 64 MiB memory witness, and a continuing counter after restore. Those trials do not exercise the two interruption points above.The tested
cmd/atelet/main.goand replay helper match main when checked on October 24. This exercises the staging introduced by #3048. The older open PR #1139 proposes destination recovery and interrupted-move handling; I have not tested it against this fixture. Is #1139 intended to cover these two points? I’m happy to contribute a focused regression test from my fork and coordinate with its author.Reproducer: exact tested fixture, preparation script, instructions, and evidence. The packet pins the upstream revision and checks source/fixture hashes before generating an overlay; no Kubernetes or gVisor installation is needed for this fixture.
AI-assisted investigation, fixture preparation, and draft.
eren23 commentedon Oct 24, 2026
Follow-up to my earlier fixture: the same two windows reproduce with actual atelet test-process SIGKILL/restart and a persistent real gVisor worker on
1756c0f7..staged: retry failed 3/3 and the real fallback emptied the observed snapshot directory.CommitLocal: retry failed 3/3; images/manifest remained, but tracker state was uncommitted.Each failed retry attempted another capture and reached exit 128:
FetchSpec failed: loading container: file does not exist. All nine controls passed, restoring the same 64 MiB memory witness after one capture. Five baseline packages passed 1,392 tests; cleanup and independent review passed.Minimal reproducer and instructions. Full raw logs are retained locally. Scope: same boot, test processes, direct runsc guest and simulated control-plane services; no full activation or worker/host-restart claim.
#1139 has related recovery ideas but uses older interfaces; I have not tested it as a fix. I can contribute focused regression cases and coordinate an adaptation with its author.
AI-assisted investigation, fixture, verification and draft.
chw120 commentedon Oct 26, 2026
Thanks Eren Akbulut (@eren23)! I opened two PRs:
I reran your TestCheckpointBoundaryAudit packet on #3218 and all 15 cases pass (6 failed on 1756c0f). The only fixture change is the first-stage hook, which now follows root.Link instead of root.Rename. Feel free to rerun your tests against them, including the real gVisor + SIGKILL packet, and review the PRs.
atelet, ateworker: finish a checkpoint retry after atelet restarts (a…
ateworker: reuse a completed checkpoint only on the same boot (agent-…