Repository navigation
[Bug]: Since #2222, revert and delete --any-state cannot clear an actor whose node is gone #2470
Description
What happened?
Since #2222 (cc Michelle Au (@msau42)), revert and delete call atelet Terminate whenever the actor has AssignedNode set. Before, they skipped it when the actor had no worker assignment. When that node is gone for good, the call never succeeds, so I have no way left to revert the actor or to delete it.
Two kinds of actors keep AssignedNode with no worker. A PAUSED actor keeps it for its local snapshot. A CRASHED actor keeps it too, because crashBoundActor clears WorkerAssignment and leaves AssignedNode (workflow_worker_delete.go:224).
This is what I see on main with no atelet on the node.
| Start state | Verb | Result |
|---|---|---|
| PAUSED | revert | fails at CallAteletTerminate, actor stays REVERTING |
| CRASHED (worker pod gone) | revert | fails at CallAteletTerminate, actor stays REVERTING |
| REVERTING | resume, delete | FailedPrecondition |
| any of the above | delete --any-state | fails at CallAteletTerminate, actor stays DELETING |
The same happens with suspend from PAUSED, which #1502 already describes. What is new to me is that the two exits are closed too.
The CRASHED case is the one docs/api-guide.md describes for a drain, a scale down or a Spot reclaim, and it says RevertActor brings the actor back to its last snapshot. If the node went away with the pod, that revert now fails. The RevertActor doc also says it "needs no live worker to succeed".
I understand that ErrNoAteletOnNode is retryable on purpose (dialer.go:42), and that is right while atelet restarts. The problem is that nothing ends the retry when the node does not come back. I left a PAUSED actor for 20 minutes with atelet gone and nothing moved it. When I put atelet back on the node, all of the stuck verbs completed.
Expected Behavior
I expected revert to bring the actor back to its last durable snapshot, or at least delete --any-state to remove it, after its node is gone for good. One option I can think of is for ensureAteletTerminated (workflow_delete.go:124) to treat a node that no longer exists in the cluster as nothing left to terminate. A node that still exists would keep the current retry.
I'm happy to send a PR once there's a preferred direction.
Steps to Reproduce
I used a single node kind cluster, so I removed atelet from the node instead of deleting the node. The dialer finds atelet only by the pod's node name in its informer (informer.go:40), so a deleted node whose pods are gone gives the same ErrNoAteletOnNode.
- Create an actor, resume it, suspend it (so it has a durable snapshot), and resume it again.
- Pause it, or leave it running for the CRASHED case.
- Take atelet off the node.
kubectl -n ate-system patch ds <atelet-ds> --type merge \ -p '{"spec":{"template":{"spec":{"nodeSelector":{"lab.test/node-lost":"true"}}}}}' - For the CRASHED case, delete the actor's worker pod and wait for CRASHED.
- Run
kubectl ate revert actor, orkubectl ate delete actor --any-state. - To recover, set
"lab.test/node-lost": nullin the same patch. The stuck verbs then complete.
I also tried a node that is unreachable but still listed, by dropping traffic to the atelet pod on port 8085. Revert failed with Unavailable after 30 seconds and delete --any-state after 20 seconds, with the same stuck states.
Sandbox Runtime
gVisor
Agent Substrate Version / Commit SHA
Kubernetes Version & Environment
kind, Kubernetes v1.37.0, single node
Host OS & Architecture
macOS 26.2, arm64
Relevant Logs and Diagnostic Output
revert from PAUSED
internal server error: workflow failed at step DiscardWorker: workflow failed at step CallAteletTerminate:
while connecting to atelet on node "kind-control-plane": no atelet pod found on node: "kind-control-plane"
delete --any-state
internal server error: while terminating atelet: workflow failed at step CallAteletTerminate:
while connecting to atelet on node "kind-control-plane": no atelet pod found on node: "kind-control-plane"
resume after the failed revert
FailedPrecondition desc = AssignWorker prerequisite not met for Actor: default/trap-8446e1
(got: ACTOR_STATE_REVERTING, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)
Additional Context
Related to #1502 (suspend from PAUSED on a lost node), #1526 (lost worker pod, where revert was suggested as the way back) and #1706 (which tolerates ErrNoAteletOnNode on the worker delete path only).
Confirmation
- I have searched existing issues and verified that this is not a duplicate.
- I have verified that this issue occurs on the latest commit on
main.
Activity
Let revert and delete finish when the actor's node is gone
Let revert and delete finish when the actor's node is gone
Let revert and delete finish when the actor's node is gone
Let revert and delete finish when the actor's node is gone
Let revert and delete finish when the actor's node is gone
igooch commentedon Oct 19, 2026
Resume has the same problem from a paused actor, which the table above doesn't include.
A paused actor keeps
assigned_node, andschedulingConstraintspins its resume to that node (workflow_resume.go:666). When the node is gone, the resume fails withResourceExhausted"no worker has room for the actor", which looks like a capacity problem. #2881 makes resume detach an actor whose node is full. With that change, the detach finds no atelet on the node and the resume returnsUnavailable. Retrying doesn't help in either case.#2486 fixes revert and delete, so a manual revert followed by a resume will work, and it lands on the same durable snapshot. But nothing in the resume error tells anyone to revert.
Could we keep this issue open for resume when #2486 merges, by changing "Fixes" to "Part of"? The fix I have in mind would reuse #2486's
NodeGone. When the detach finds no atelet and the node is gone, resume would clear that node's localities andassigned_nodeand restore from the latest completed durable snapshot. If no completed durable snapshot covers the latest generation, it would crash the actor instead.Let revert and delete finish when the actor's node is gone
Let revert and delete finish when the actor's node is gone
Revert and delete no longer hang when an actor's node is deleted (age…
Crash a paused actor on resume when its node is gone (agent-substrate…
Resume restores an uploaded snapshot when its node is gone (agent-sub…
whysosuresh commentedon Oct 25, 2026
We hit this on a real cluster, not a simulated one, in case the data point helps with the resume half that is still open.
Setup: GKE, Substrate v0.5.0 (before #2486), gVisor. To copy what a GKE upgrade or repair does to a node, we cordoned the node running a worker and deleted that worker's pod. The WorkerPool recreated the worker on a new node, and the autoscaler deleted the old node about 12 minutes later.
Releasing actor from a worker whose pod is gone, thenActor crashed). Each had written a file after its last suspend, and that file was lost, as expected for a crash.RevertActorandDeleteActoron both failed the same way and never finished:v0.6.0 carries #2486, so revert and delete should clear these once we upgrade. +1 to keeping this open for resume, as igooch suggests. From the client's side, a resume error that says the node is gone (or a resume that restores from the last durable snapshot on its own) would save callers from guessing that a "no room" error means "revert first".
eren23 commentedon Oct 25, 2026
Thanks for the GKE confirmation; v0.6.0 also includes #3010 for the pending-snapshot resume case, so please let us know how revert/delete behave after your upgrade.
sfunkenhauser commentedon Nov 3, 2026
Between #2486, #3010 and #3094 I believe this should be fixed. Please re-open if that's not the case.