Skip to content

Crash suspended actors when their latest snapshot is lost with the node - #3005

Closed
Eren Akbulut (eren23) wants to merge 1 commit into
agent-substrate:mainfrom
eren23:fix/resume-lost-upload
Closed

Eren Akbulut (eren23) wants to merge 1 commit into
agent-substrate:mainfrom
eren23:fix/resume-lost-upload

Conversation

@eren23

@eren23 Eren Akbulut (eren23) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

A suspended actor whose node was deleted stays pinned when its latest snapshot upload is incomplete. Resume returns Unavailable indefinitely even when another node has capacity. Main already recovers actors with a completed latest local snapshot; this change handles the remaining loss case.

After NodeGone confirms deletion, reload the same suspended incarnation and check the latest generation. If its upload completed during the lookup, use the existing detach path. Otherwise commit a crash, clear the deleted node bindings, and retain snapshot records for revert/delete cleanup. Version or UID conflicts abort without emitting a crash event.

Follows merged #2486; part of #2470. Rebased onto 0279fe3dd, with only the resume commit remaining. Its patch is unchanged (git range-diff reports =). Node deletion does not fence a still-running machine. No-capacity-anywhere behavior stays unchanged, and this adds no new CSI shortcut.

Validation on d40db8358:

  • Controlapi and its functional tests pass three shuffled race runs, covering completed-upload controls, crash retries, lookup failures, concurrent upload/state/incarnation changes, cancellation and failed-write telemetry.
  • All repository verifiers pass. The full race run records 10,709 passing test/subtest results, eight skips, and only two macOS pidfd failures, reproduced on unchanged main. All 15 ateapi test packages pass in that full run.
  • An earlier three-run ateapi check had intermittent PostgreSQL certificate-authentication and outbox-partition failures. The PostgreSQL package passed the full run; both focused baseline checks and the full baseline package passed three runs, including the original shuffle seed. Their cause remains unconfirmed. This PR does not change that package or its dependencies.
  • A disposable three-node arm64 kind cluster with real gVisor workers passed one pending-upload candidate trial, the same trial against parent 0279fe3dd, and one completed-upload recovery control. With the source machine stopped but its Node still present, both versions returned Unavailable and stayed suspended. After Node deletion, the candidate crashed generation 3, preserved the completed generation-2 and pending generation-3 records, and allowed explicit revert/resume/delete; the parent remained suspended with Unavailable. Object-store listings confirmed generation 2 was complete and generation 3 had no uploaded objects. With the latest snapshot completed, the candidate restored on the surviving node with memory, durable-file and rootfs counters intact. This validates gVisor on local kind; it does not cover micro-VM or real CSI.
  • The original good-cop, adversarial and judge audits found no correctness blockers. A fresh review checked the rebase against the merged volume-detach and DELETING-state protections.

AI assistance: Codex adapted the earlier local implementation, wrote and ran tests, reviewed the rebase, and prepared this description. The node-loss tests stopped each disposable machine before deleting its Node; they do not establish fencing by Node deletion alone.

  • The published patch applies unchanged to main 3b779a823 (runtime reporting atelet, ateapi: report worker runtimes per sandbox class #3022). On that local application, control API, functional, scheduling and hardware package tests passed three shuffled runs with the race detector. This is a compatibility check; the draft branch remains at d40db8358.

@dims

Copy link
Copy Markdown
Collaborator

some overlap with #3010?

@eren23

Copy link
Copy Markdown
Contributor Author

Thanks igooch for reviewing #2486, and yes Davanum Srinivas (@dims), these overlap—I'll close this in favor of #3010 and bring the useful regression cases over there.

Tom Gamble (gambtho) pushed a commit to gambtho/substrate that referenced this pull request Oct 21, 2026
…#3010)

agent-substrate#2883 lets resume forget an unreachable node when a paused actor's
latest snapshot is durable. When that upload is still PENDING, the actor
stays pinned. If the node was deleted, every resume returns
`Unavailable`, and nothing says the snapshot is lost.

Now, when resume finds no atelet on the pinned node and `NodeGone`
reports the Node object deleted, it reads the actor again. If the upload
was recorded during the lookup, it detaches as usual. Otherwise it
crashes the actor. The crash keeps `assigned_node` and the PENDING
snapshot. `RevertActor` then skips the terminate, deletes the pending
objects, and returns the actor to its last completed durable snapshot. A
node that still exists, or a failed Node lookup, keeps the current
retry.

There are two known limits. The check runs only after resume finds room
on another node, so with no room anywhere resume still returns
`RESOURCE_EXHAUSTED`. And a Node object deleted while the machine keeps
running crashes the actor even though its checkpoint is still on that
machine. That's the same trade agent-substrate#2486 makes.

A PENDING copy may already be fully uploaded if atelet died before
reporting it. Resume still crashes the actor, and revert deletes that
copy. Recovering it needs ate-api-server to check for the manifest,
which is a follow-up.

No proto changes. `ResumeActor` can now crash the actor and return
`FAILED_PRECONDITION` in this case, and `docs/api-guide.md` says so.

Part of agent-substrate#2470

Supersedes agent-substrate#3005. Its "upload completes during lookup" regression case
is adapted here.

AI use: Claude Code helped write the change and tests. I reviewed them.

- [ ] Tests pass
- [x] Docs updated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants