Repository navigation
ateworker: reuse completed checkpoint when workload is already stopped - #3015
Merged
Haven Xia (HavenXia) merged 1 commit intoOct 23, 2026
Conversation
Record a completion marker (.checkpoint-done.json) inside checkpoint-state/ once CheckpointWorkload has written the snapshot files and torn down the workload. If a retry arrives after the workload is gone (for example when atelet restarts or loses the response before moving checkpoint-state/ into local-checkpoints/), ReuseCheckpoint returns the recorded SnapshotFiles and DataSnapshotFiles while they are still intact on disk instead of failing on a missing sandbox.
Chenyi Wang (chw120)
force-pushed
the
ateworker-reuse-completed-checkpoint
branch
from
October 23, 2026 20:34
6660eb1 to
f84648d
Compare
Haven Xia (HavenXia)
approved these changes
Oct 23, 2026
Haven Xia (HavenXia)
enabled auto-merge
October 23, 2026 20:38
This was referenced Oct 26, 2026
Max Thompson (thompsonmax)
pushed a commit
to thompsonmax/substrate
that referenced
this pull request
Oct 28, 2026
…substrate#3213) Follow-up to agent-substrate#3015. Part of agent-substrate#2836. A retried CheckpointWorkload reuses the worker's record of a completed checkpoint. The worker does not sync the snapshot files, so after a node reboot the record and the files can survive with torn contents. The retry then reused them, and atelet stamped the local snapshot with the new boot ID, so its pre-reboot check no longer catches them. - RecordCheckpoint records the node's boot ID, and ReuseCheckpoint skips a record from another boot. After a reboot, the retry fails instead of committing files that may be torn. - atelet's boot ID reader moves to internal/bootid so the workers can use it. TestRecordAndReuseCheckpoint covers a record from another boot. AI use: AI-assisted. I reviewed and tested it. - [x] Tests pass --------- Signed-off-by: Chenyi Wang <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
ateletrestarts (or loses theCheckpointWorkloadresponse) afterateworkerhas already finished writing the snapshot intocheckpoint-state/and torn down the sandbox/VM,atelethas not yet movedcheckpoint-state/intolocal-checkpoints/<uuid>/or recorded the snapshot insnapshottracker. When the control plane retriesCheckpoint, the replacementateletcallsateworker.CheckpointWorkloadagain against a workload that no longer exists (runsc checkpoint: exit status 128).This PR records an atomic completion marker (
checkpoint-state/.checkpoint-done.json) at the end ofCheckpointWorkloadin bothateworker-gvisorandateworker-microvm. If a retry arrives when the workload is already unhosted (lookupActor(actorUID) == nil),ReuseCheckpointverifies that the marker's fidelity matches and all listedSnapshotFiles/DataSnapshotFilesare still present incheckpoint-state/, returning the cached response immediately instead of re-runningrunsc checkpointor cloud-hypervisor snapshot.Once
ateletmoves the snapshot files intolocal-checkpoints/<uuid>/or resets the actor directory viaresetActorDirs, the marker is invalidated/removed automatically.Part of #2836