Repository navigation
atelet, ateworker: finish a checkpoint retry after atelet restarts - #3218
Merged
Dmitry Berkovich (dberkov) merged 2 commits intoOct 27, 2026
Conversation
Chenyi Wang (chw120)
requested review from
Haven Xia (HavenXia),
Michelle Au (msau42) and
Zoe Zhao (zoez7)
October 26, 2026 18:34
moveLocalCheckpoint renamed the files the worker reported out of checkpoint-state before CommitLocal. If atelet died in between, the retried Checkpoint found no committed snapshot, and the worker could not answer from its completed checkpoint because the files had left checkpoint-state. The worker then checkpointed a workload that no longer existed. Stage the files as hard links instead, so checkpoint-state keeps them until atelet resets the actor's directories after the commit. A retry also replaces a local snapshot directory that an interrupted attempt published but never committed. Signed-off-by: Chenyi Wang <[email protected]>
CheckpointWorkload ran under the caller's context. If atelet died during runsc checkpoint or the cloud-hypervisor snapshot, the worker canceled it partway and wrote no completion record. The retry from the restarted atelet then checkpointed again and failed, because the first attempt had already paused or stopped the workload. Once CheckpointWorkload holds the actor's lock, it now runs under CheckpointContext, which keeps the caller's deadline but not its cancellation. Signed-off-by: Chenyi Wang <[email protected]>
Chenyi Wang (chw120)
force-pushed
the
checkpoint-retry-after-atelet-restart
branch
from
October 26, 2026 22:26
37c8782 to
2dffce9
Compare
Chenyi Wang (chw120)
added
the
prio/P0
Bowei Du (bowei)
added
area/gvisor
area/microVM
area/node
area/reliability
area/storage
kind/bug
Dmitry Berkovich (dberkov)
approved these changes
Oct 27, 2026
Chenyi Wang (chw120)
deleted the
checkpoint-retry-after-atelet-restart
branch
October 27, 2026 22:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2836.
A Checkpoint retry still failed with
runsc checkpoint: exit status 128when atelet died before the snapshot committed: while staging the local snapshot or after publishing it (both reported by Eren Akbulut (@eren23)), or while the worker was still capturing the workload, which the worker then canceled.TestRetriedCheckpointAfterUncommittedStaging and TestCheckpointContext cover the change, and the TestCheckpointBoundaryAudit fixture from #2836 passes all 15 cases, up from 9. The "pause move fails" row of TestFailedCheckpointKeepsLocalSnapshot is removed, since a leftover local snapshot directory is now replaced.
AI use: AI-assisted. I reviewed and tested it.