You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 68ec85b
Browse filesBrowse the repository at this point in the historyBrowse files
ateapi: fail a paused actor's resume fast when its node is gone, and crash it when no durable snapshot remains (LOCAL_SNAPSHOT_GONE)
A PAUSED actor's local snapshot lives on the node its worker ran on, and
the Resume workflow pins the actor to that node (Constraints.RequiredNodes).
When the node had left the cluster -- a spot interruption, a scale-down, a
node roll -- nothing noticed: the scheduler found no worker on the node and
AssignWorker answered ResourceExhausted "no free workers available", the
message of a full pool. The ingress dataplane parks a request on that
status, so every resume waited out the whole parking budget for an error
no wait could clear, the consumer saw a timeout, and the actor stayed
PAUSED for good: a suspend dialled the atelet on the dead node and stayed
retryable, so the actor could not be suspended and -- a delete suspends
first -- not be deleted either (agent-substrate#398 names the
case; agent-substrate#397 the empty-node variant).
The scheduler now says what its pass over the fleet saw when a node
restriction was in effect: Schedule returns a NoCapacityError -- still
errors.Is ErrNoCapacity -- carrying the restricted nodes, the eligible
workers found on them (all full) and the ACTIVE workers of any pool that
report them. AssignWorker classifies it:
- eligible workers on the node, all full: ResourceExhausted, "all
eligible workers on those nodes are busy", the node named, parked as
before;
- the node still in the cluster but no eligible worker on it (a pool
rolling, a pool moved off the node): ResourceExhausted with the node
named, still parked -- a worker may come back;
- the node gone from the cluster: the snapshot is lost. Without a
durable snapshot to fall back to the actor is crashed with the reason
LOCAL_SNAPSHOT_GONE (CRASHED, ate.actor.crashes labelled, deletable)
and the caller gets DataLoss with the crash directive; with one, the
actor is left as it is and the caller gets DataLoss naming the node.
A node counts as gone only when no ACTIVE worker of any pool reports it
and its Node object is absent, read from a Node informer ate-api-server
now runs (RBAC: nodes get/list/watch). A worker's absence alone is no
proof: a pool rolling, an ateom restarting or the atelet DaemonSet
updating leave a node without an ACTIVE worker for a while, with its disk
and the snapshot on it intact. Without a Node lister nothing is declared
lost.
The paused-origin suspend takes the same exit: an atelet missing on a node
that is gone crashes the actor (or fails fast when a durable snapshot
remains) instead of staying retryable forever, so the owner can delete
the actor.
DataLoss is the one status the dataplane never parks on -- it retries
Aborted, and while parking is enabled ResourceExhausted, FailedPrecondition
and Unavailable -- so the caller learns within one round trip. The atenet
router maps it to 410 with the cause in the body, the way it preserves the
capacity message.
The read side of agent-substrate#397: empty entries in
node_vms_with_local_snapshots are ignored when the constraints are built
and when the suspend picks the node to dial, so a record poisoned with
[""] by an older pause finalization is schedulable again (the write side
already refuses an unknown node).
Signed-off-by: Timo Derstappen <[email protected]>
t.Fatalf("ResumeActor kept answering %v after the node was deleted", resumeErr)
3861
+
}
3862
+
assertGrpcError(t, resumeErr, codes.DataLoss, fmt.Sprintf("actor %s/%s crashed: its local snapshot %q is on node(s) [%s], which no longer exist in the cluster, and it has no durable snapshot", testAtespace, name, paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(), node))
3863
+
if!ateerrors.ActorCrashRequested(resumeErr) {
3864
+
t.Errorf("ResumeActor error carries no crash directive: %v", resumeErr)
fmt.Errorf("actor %s: its local snapshot %q is on node(s) %v, which no longer exist in the cluster; the paused state is lost", actorRef, snapshot, nodes))
92
+
}
93
+
94
+
slog.LogAttrs(ctx, slog.LevelError, "Setting Actor to crashed: its local snapshot is lost with its node and it has no durable snapshot", attrs...)
fmt.Errorf("actor %s crashed: its local snapshot %q is on node(s) %v, which no longer exist in the cluster, and it has no durable snapshot", actorRef, snapshot, nodes))
100
+
}
101
+
102
+
// noFreeWorkerError turns the scheduler's refusal into the caller's status.
103
+
// Without a node restriction the pool is full: ResourceExhausted, which the
104
+
// router parks on until a worker frees up. Under one -- a PAUSED actor's
105
+
// local snapshot -- the scan says which of three states the actor is in:
106
+
//
107
+
// - eligible workers on the snapshot's node(s), all full: as above, with
108
+
// the node named, so the message does not point at cluster capacity;
109
+
// - the node(s) still in the cluster but no eligible worker on them (a
110
+
// pool rolling, a pool moved off the node): ResourceExhausted with the
111
+
// node named, still worth parking on -- a worker may come back;
112
+
// - the node(s) gone from the cluster: the snapshot is lost
113
+
// (localSnapshotLost).
114
+
//
115
+
// A node is gone only when no ACTIVE worker of any pool reports it and the
116
+
// Node object is absent; either signal alone keeps the actor waiting.
0 commit comments