Repository navigation
Crash suspended actors when their latest snapshot is lost with the node - #3005
Closed
Eren Akbulut (eren23) wants to merge 1 commit into
Closed
Eren Akbulut (eren23) wants to merge 1 commit into
Eren Akbulut (eren23) wants to merge 1 commit into
Conversation
Eren Akbulut (eren23)
force-pushed
the
fix/resume-lost-upload
branch
from
October 21, 2026 04:41
3af06a1 to
d40db83
Compare
Collaborator
|
some overlap with #3010? |
1 of 2 tasks
Contributor
Author
|
Thanks igooch for reviewing #2486, and yes Davanum Srinivas (@dims), these overlap—I'll close this in favor of #3010 and bring the useful regression cases over there. |
Tom Gamble (gambtho)
pushed a commit
to gambtho/substrate
that referenced
this pull request
Oct 21, 2026
…#3010) agent-substrate#2883 lets resume forget an unreachable node when a paused actor's latest snapshot is durable. When that upload is still PENDING, the actor stays pinned. If the node was deleted, every resume returns `Unavailable`, and nothing says the snapshot is lost. Now, when resume finds no atelet on the pinned node and `NodeGone` reports the Node object deleted, it reads the actor again. If the upload was recorded during the lookup, it detaches as usual. Otherwise it crashes the actor. The crash keeps `assigned_node` and the PENDING snapshot. `RevertActor` then skips the terminate, deletes the pending objects, and returns the actor to its last completed durable snapshot. A node that still exists, or a failed Node lookup, keeps the current retry. There are two known limits. The check runs only after resume finds room on another node, so with no room anywhere resume still returns `RESOURCE_EXHAUSTED`. And a Node object deleted while the machine keeps running crashes the actor even though its checkpoint is still on that machine. That's the same trade agent-substrate#2486 makes. A PENDING copy may already be fully uploaded if atelet died before reporting it. Resume still crashes the actor, and revert deletes that copy. Recovering it needs ate-api-server to check for the manifest, which is a follow-up. No proto changes. `ResumeActor` can now crash the actor and return `FAILED_PRECONDITION` in this case, and `docs/api-guide.md` says so. Part of agent-substrate#2470 Supersedes agent-substrate#3005. Its "upload completes during lookup" regression case is adapted here. AI use: Claude Code helped write the change and tests. I reviewed them. - [ ] Tests pass - [x] Docs updated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A suspended actor whose node was deleted stays pinned when its latest snapshot upload is incomplete. Resume returns Unavailable indefinitely even when another node has capacity. Main already recovers actors with a completed latest local snapshot; this change handles the remaining loss case.
After NodeGone confirms deletion, reload the same suspended incarnation and check the latest generation. If its upload completed during the lookup, use the existing detach path. Otherwise commit a crash, clear the deleted node bindings, and retain snapshot records for revert/delete cleanup. Version or UID conflicts abort without emitting a crash event.
Follows merged #2486; part of #2470. Rebased onto
0279fe3dd, with only the resume commit remaining. Its patch is unchanged (git range-diffreports=). Node deletion does not fence a still-running machine. No-capacity-anywhere behavior stays unchanged, and this adds no new CSI shortcut.Validation on
d40db8358:0279fe3dd, and one completed-upload recovery control. With the source machine stopped but its Node still present, both versions returned Unavailable and stayed suspended. After Node deletion, the candidate crashed generation 3, preserved the completed generation-2 and pending generation-3 records, and allowed explicit revert/resume/delete; the parent remained suspended with Unavailable. Object-store listings confirmed generation 2 was complete and generation 3 had no uploaded objects. With the latest snapshot completed, the candidate restored on the surviving node with memory, durable-file and rootfs counters intact. This validates gVisor on local kind; it does not cover micro-VM or real CSI.AI assistance: Codex adapted the earlier local implementation, wrote and ran tests, reviewed the rebase, and prepared this description. The node-loss tests stopped each disposable machine before deleting its Node; they do not establish fencing by Node deletion alone.
3b779a823(runtime reporting atelet, ateapi: report worker runtimes per sandbox class #3022). On that local application, control API, functional, scheduling and hardware package tests passed three shuffled runs with the race detector. This is a compatibility check; the draft branch remains atd40db8358.