Skip to content

Commit 68ec85b

Browse files
teemowQuentinBisson
authored andcommitted
ateapi: fail a paused actor's resume fast when its node is gone, and crash it when no durable snapshot remains (LOCAL_SNAPSHOT_GONE)
A PAUSED actor's local snapshot lives on the node its worker ran on, and the Resume workflow pins the actor to that node (Constraints.RequiredNodes). When the node had left the cluster -- a spot interruption, a scale-down, a node roll -- nothing noticed: the scheduler found no worker on the node and AssignWorker answered ResourceExhausted "no free workers available", the message of a full pool. The ingress dataplane parks a request on that status, so every resume waited out the whole parking budget for an error no wait could clear, the consumer saw a timeout, and the actor stayed PAUSED for good: a suspend dialled the atelet on the dead node and stayed retryable, so the actor could not be suspended and -- a delete suspends first -- not be deleted either (agent-substrate#398 names the case; agent-substrate#397 the empty-node variant). The scheduler now says what its pass over the fleet saw when a node restriction was in effect: Schedule returns a NoCapacityError -- still errors.Is ErrNoCapacity -- carrying the restricted nodes, the eligible workers found on them (all full) and the ACTIVE workers of any pool that report them. AssignWorker classifies it: - eligible workers on the node, all full: ResourceExhausted, "all eligible workers on those nodes are busy", the node named, parked as before; - the node still in the cluster but no eligible worker on it (a pool rolling, a pool moved off the node): ResourceExhausted with the node named, still parked -- a worker may come back; - the node gone from the cluster: the snapshot is lost. Without a durable snapshot to fall back to the actor is crashed with the reason LOCAL_SNAPSHOT_GONE (CRASHED, ate.actor.crashes labelled, deletable) and the caller gets DataLoss with the crash directive; with one, the actor is left as it is and the caller gets DataLoss naming the node. A node counts as gone only when no ACTIVE worker of any pool reports it and its Node object is absent, read from a Node informer ate-api-server now runs (RBAC: nodes get/list/watch). A worker's absence alone is no proof: a pool rolling, an ateom restarting or the atelet DaemonSet updating leave a node without an ACTIVE worker for a while, with its disk and the snapshot on it intact. Without a Node lister nothing is declared lost. The paused-origin suspend takes the same exit: an atelet missing on a node that is gone crashes the actor (or fails fast when a durable snapshot remains) instead of staying retryable forever, so the owner can delete the actor. DataLoss is the one status the dataplane never parks on -- it retries Aborted, and while parking is enabled ResourceExhausted, FailedPrecondition and Unavailable -- so the caller learns within one round trip. The atenet router maps it to 410 with the cause in the body, the way it preserves the capacity message. The read side of agent-substrate#397: empty entries in node_vms_with_local_snapshots are ignored when the constraints are built and when the suspend picks the node to dial, so a record poisoned with [""] by an older pause finalization is schedulable again (the write side already refuses an unknown node). Signed-off-by: Timo Derstappen <[email protected]>
1 parent 34d4d1a commit 68ec85b

22 files changed

Lines changed: 933 additions & 23 deletions

File tree

‎charts/substrate/templates/ate-api-server.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ rules:
2222
- apiGroups: [""]
2323
resources: ["pods"]
2424
verbs: ["get", "watch", "list"]
25+
# Nodes are read to tell a node that left the cluster from one whose workers
26+
# are momentarily absent: a PAUSED actor's local snapshot lives on a node, and
27+
# only a node that is gone makes it lost (see the Resume workflow).
28+
- apiGroups: [""]
29+
resources: ["nodes"]
30+
verbs: ["get", "watch", "list"]
2531
- apiGroups: ["ate.dev"]
2632
resources: ["workerpools", "sandboxconfigs", "csidriverconfigs"]
2733
verbs: ["get", "watch", "list"]

‎charts/substrate/tests/ate_api_server_test.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,15 @@ tests:
6868
content:
6969
name: AWS_ACCESS_KEY_ID
7070
any: true
71+
72+
- it: lets ate-api-server read Nodes, to tell a node that left the cluster from one whose workers are momentarily absent
73+
documentSelector:
74+
path: kind
75+
value: ClusterRole
76+
asserts:
77+
- contains:
78+
path: rules
79+
content:
80+
apiGroups: [""]
81+
resources: ["nodes"]
82+
verbs: ["get", "watch", "list"]

‎cmd/ateapi/internal/controlapi/functionaltest/actor_test.go‎

Lines changed: 97 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ import (
2525

2626
"github.com/agent-substrate/substrate/cmd/ateapi/internal/store"
2727
"github.com/agent-substrate/substrate/internal/ateattr"
28+
"github.com/agent-substrate/substrate/internal/ateerrors"
29+
"github.com/agent-substrate/substrate/internal/installdefaults"
2830
"github.com/agent-substrate/substrate/internal/proto/ateletpb"
2931
"github.com/agent-substrate/substrate/internal/resources"
3032
"github.com/agent-substrate/substrate/internal/volume"
@@ -38,7 +40,10 @@ import (
3840
"google.golang.org/protobuf/encoding/protowire"
3941
"google.golang.org/protobuf/proto"
4042
"google.golang.org/protobuf/testing/protocmp"
43+
corev1 "k8s.io/api/core/v1"
44+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
4145
"k8s.io/apimachinery/pkg/util/wait"
46+
"k8s.io/utils/ptr"
4247
)
4348

4449
// TestCreateActor_Success tests the happy path for creating an actor.
@@ -3269,11 +3274,12 @@ func TestResumeActor_RelocatesAfterSuspendFromPaused(t *testing.T) {
32693274
createWorkerPod(t, tc, ns, "worker-2", "node2", "pool1")
32703275
setupAteletOnNode(t, tc, "atelet-node2", "node2")
32713276

3272-
// Capacity exhaustion is ResourceExhausted.
3277+
// Capacity exhaustion is ResourceExhausted, and under the snapshot's node
3278+
// pin it names the node rather than pointing at cluster capacity.
32733279
_, err = tc.client.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{
32743280
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned},
32753281
})
3276-
assertGrpcError(t, err, codes.ResourceExhausted, "no free workers available")
3282+
assertGrpcError(t, err, codes.ResourceExhausted, "actor's local snapshot is on node(s) [node1] but all eligible workers on those nodes are busy")
32773283

32783284
suspended, err := tc.client.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{
32793285
Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: pinned},
@@ -3785,3 +3791,92 @@ func TestRevertActor_NotFound(t *testing.T) {
37853791
t.Fatalf("RevertActor = %v, want NotFound", err)
37863792
}
37873793
}
3794+
3795+
// TestResumeActor_PausedOnGoneNodeCrashesAndDeletes covers a PAUSED actor
3796+
// whose node left the cluster: its local snapshot went with the node, so the
3797+
// next resume cannot park on "no free workers" until a budget elapses. The
3798+
// control plane answers DataLoss with the crash directive as soon as it sees
3799+
// the node gone, the actor is CRASHED, and its owner can delete it.
3800+
func TestResumeActor_PausedOnGoneNodeCrashesAndDeletes(t *testing.T) {
3801+
ns := namespaceForTest("ns-resume-node-gone")
3802+
tc := setupTest(t, ns)
3803+
defer tc.cleanup()
3804+
ctx := context.Background()
3805+
3806+
createTemplate(t, tc, ns)
3807+
const node = "node-orphaned"
3808+
if _, err := tc.k8sClient.CoreV1().Nodes().Create(ctx, &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: node}}, metav1.CreateOptions{}); err != nil {
3809+
t.Fatalf("create Node %s: %v", node, err)
3810+
}
3811+
t.Cleanup(func() {
3812+
_ = tc.k8sClient.CoreV1().Nodes().Delete(context.Background(), node, metav1.DeleteOptions{})
3813+
})
3814+
// The pool's only worker is on the node under test, so the first sprint
3815+
// lands there and the pause records that node.
3816+
workerName := createWorkerPod(t, tc, ns, "worker-orphaned", node, "pool1")
3817+
setupAteletOnNode(t, tc, "atelet-orphaned", node)
3818+
3819+
const name = "actor-orphan"
3820+
if _, err := tc.client.CreateActor(ctx, &ateapipb.CreateActorRequest{Actor: &ateapipb.Actor{
3821+
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name},
3822+
ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"},
3823+
}}); err != nil {
3824+
t.Fatalf("CreateActor failed: %v", err)
3825+
}
3826+
if _, err := tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}}); err != nil {
3827+
t.Fatalf("ResumeActor failed: %v", err)
3828+
}
3829+
if _, err := tc.client.PauseActor(ctx, &ateapipb.PauseActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}}); err != nil {
3830+
t.Fatalf("PauseActor failed: %v", err)
3831+
}
3832+
paused, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}})
3833+
if err != nil {
3834+
t.Fatalf("GetActor failed: %v", err)
3835+
}
3836+
if got := paused.GetStatus().GetLocalSnapshotInfo().GetNodeVmsWithLocalSnapshots(); len(got) != 1 || got[0] != node {
3837+
t.Fatalf("paused actor pinned to %v, want [%s]", got, node)
3838+
}
3839+
waitForWorkerAvailable(t, tc, workerName)
3840+
// Free capacity elsewhere from now on, so the refusal below is about the
3841+
// node, not the pool.
3842+
createWorkerPod(t, tc, ns, "worker-elsewhere", "node1", "pool1")
3843+
3844+
// The node leaves the cluster: its worker, its atelet and the Node itself.
3845+
deleteWorkerPod(t, tc, ns, "worker-orphaned")
3846+
if err := tc.k8sClient.CoreV1().Pods(installdefaults.SystemNamespace).Delete(ctx, "atelet-orphaned", metav1.DeleteOptions{GracePeriodSeconds: ptr.To[int64](0)}); err != nil {
3847+
t.Fatalf("delete atelet pod: %v", err)
3848+
}
3849+
if err := tc.k8sClient.CoreV1().Nodes().Delete(ctx, node, metav1.DeleteOptions{}); err != nil {
3850+
t.Fatalf("delete Node %s: %v", node, err)
3851+
}
3852+
3853+
// Until the Node informer has seen the deletion the refusal is the
3854+
// retryable "no workers on that node"; then it is terminal.
3855+
var resumeErr error
3856+
if err := wait.PollUntilContextTimeout(ctx, 100*time.Millisecond, 10*time.Second, true, func(ctx context.Context) (bool, error) {
3857+
_, resumeErr = tc.client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}})
3858+
return status.Code(resumeErr) != codes.ResourceExhausted, nil
3859+
}); err != nil {
3860+
t.Fatalf("ResumeActor kept answering %v after the node was deleted", resumeErr)
3861+
}
3862+
assertGrpcError(t, resumeErr, codes.DataLoss, fmt.Sprintf("actor %s/%s crashed: its local snapshot %q is on node(s) [%s], which no longer exist in the cluster, and it has no durable snapshot", testAtespace, name, paused.GetStatus().GetLocalSnapshotInfo().GetSnapshotName(), node))
3863+
if !ateerrors.ActorCrashRequested(resumeErr) {
3864+
t.Errorf("ResumeActor error carries no crash directive: %v", resumeErr)
3865+
}
3866+
if got := ateerrors.ExtractReason(resumeErr); got != string(ateerrors.ReasonLocalSnapshotGone) {
3867+
t.Errorf("reason = %q, want %q", got, ateerrors.ReasonLocalSnapshotGone)
3868+
}
3869+
3870+
crashed, err := tc.client.GetActor(ctx, &ateapipb.GetActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}})
3871+
if err != nil {
3872+
t.Fatalf("GetActor failed: %v", err)
3873+
}
3874+
if got := crashed.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_CRASHED {
3875+
t.Fatalf("state = %v, want CRASHED", got)
3876+
}
3877+
3878+
// The actor is the owner's to delete now.
3879+
if _, err := tc.client.DeleteActor(ctx, &ateapipb.DeleteActorRequest{Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: name}}); err != nil {
3880+
t.Fatalf("DeleteActor of the crashed actor failed: %v", err)
3881+
}
3882+
}

‎cmd/ateapi/internal/controlapi/functionaltest/common_test.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu
133133
ateletFactory, ateletInformer := controlapi.AteletInformer(k8sClient, installdefaults.SystemNamespace)
134134
scFactory := informers.NewSharedInformerFactory(k8sClient, 0)
135135
scLister := scFactory.Storage().V1().StorageClasses().Lister()
136+
nodeLister := scFactory.Core().V1().Nodes().Lister()
136137

137138
substrateInformerFactory := externalversions.NewSharedInformerFactory(substrateClient, 0)
138139
workerPoolLister := substrateInformerFactory.Api().V1alpha1().WorkerPools().Lister()
@@ -215,6 +216,7 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu
215216
sandboxConfigLister,
216217
csiDriverConfigLister,
217218
scLister,
219+
nodeLister,
218220
dialer,
219221
instruments,
220222
"",
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
// Copyright 2026 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package controlapi
16+
17+
import (
18+
"context"
19+
"errors"
20+
"fmt"
21+
"log/slog"
22+
23+
"github.com/agent-substrate/substrate/cmd/ateapi/internal/scheduling"
24+
"github.com/agent-substrate/substrate/internal/ateattr"
25+
"github.com/agent-substrate/substrate/internal/ateerrors"
26+
"github.com/agent-substrate/substrate/internal/resources"
27+
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
28+
"google.golang.org/grpc/codes"
29+
"google.golang.org/grpc/status"
30+
apierrors "k8s.io/apimachinery/pkg/api/errors"
31+
)
32+
33+
// localSnapshotNodes returns the nodes a PAUSED actor's local snapshot is
34+
// recorded on, without empty entries. A record written before pause
35+
// finalization refused an unknown node can carry [""]; no worker has an
36+
// empty node name, so left in place that entry would exclude every worker
37+
// and make the actor unschedulable for good.
38+
func localSnapshotNodes(actor *ateapipb.Actor) []string {
39+
recorded := actor.GetStatus().GetLocalSnapshotInfo().GetNodeVmsWithLocalSnapshots()
40+
nodes := make([]string, 0, len(recorded))
41+
for _, n := range recorded {
42+
if n != "" {
43+
nodes = append(nodes, n)
44+
}
45+
}
46+
return nodes
47+
}
48+
49+
// localSnapshotNodesGone reports whether every node holding the actor's
50+
// local snapshot has left the cluster. It is answered from the Node informer,
51+
// the one signal a worker's absence cannot fake: a pool rolling, an ateom
52+
// restarting or the atelet DaemonSet updating leave a node without an ACTIVE
53+
// worker for a while, with its disk -- and the snapshot on it -- intact.
54+
// Without a lister nothing is known, and nothing is declared lost.
55+
func (w *ActorWorkflow) localSnapshotNodesGone(nodes []string) (bool, error) {
56+
if w.nodeLister == nil || len(nodes) == 0 {
57+
return false, nil
58+
}
59+
for _, name := range nodes {
60+
_, err := w.nodeLister.Get(name)
61+
if err == nil {
62+
return false, nil
63+
}
64+
if !apierrors.IsNotFound(err) {
65+
return false, fmt.Errorf("while looking up node %q: %w", name, err)
66+
}
67+
}
68+
return true, nil
69+
}
70+
71+
// localSnapshotLost is the terminal answer for a PAUSED actor whose local
72+
// snapshot went with its node. With no durable snapshot to fall back to the
73+
// actor is crashed: a paused actor holds no worker, so CRASHED releases
74+
// nothing here, but it makes the actor deletable and tells the consumer,
75+
// through the crash directive, that this session's runtime state is gone.
76+
// With a durable snapshot the actor is left as it is and the call fails at
77+
// once with the cause; parking would wait on a node that will not return.
78+
//
79+
// The status is DataLoss either way: a dataplane parks a request on
80+
// ResourceExhausted, FailedPrecondition and Unavailable, and this is not a
81+
// condition another operation moves the actor out of.
82+
func (w *ActorWorkflow) localSnapshotLost(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, opName string, nodes []string) error {
83+
snapshot := actor.GetStatus().GetLocalSnapshotInfo().GetSnapshotName()
84+
attrs := ateattr.ActorRefLogAttrs(actorRef)
85+
attrs = append(attrs, ateattr.FailureLogAttrs(ateattr.ReasonLocalSnapshotGone)...)
86+
attrs = append(attrs, slog.String("snapshot", snapshot), slog.Any("nodes", nodes))
87+
88+
if actor.GetStatus().GetExternalSnapshot().GetSnapshotUri() != "" {
89+
slog.LogAttrs(ctx, slog.LevelError, "Paused actor's local snapshot is lost with its node; a durable snapshot remains", attrs...)
90+
return ateerrors.NewGRPCError(ctx, codes.DataLoss, ateerrors.ReasonLocalSnapshotGone, nil,
91+
fmt.Errorf("actor %s: its local snapshot %q is on node(s) %v, which no longer exist in the cluster; the paused state is lost", actorRef, snapshot, nodes))
92+
}
93+
94+
slog.LogAttrs(ctx, slog.LevelError, "Setting Actor to crashed: its local snapshot is lost with its node and it has no durable snapshot", attrs...)
95+
if cerr := crashActor(ctx, w.store, actorRef, opName, ateattr.ReasonLocalSnapshotGone); cerr != nil {
96+
return cerr
97+
}
98+
return ateerrors.NewGRPCError(ctx, codes.DataLoss, ateerrors.ReasonLocalSnapshotGone, ateerrors.ActorCrashedMetadata(),
99+
fmt.Errorf("actor %s crashed: its local snapshot %q is on node(s) %v, which no longer exist in the cluster, and it has no durable snapshot", actorRef, snapshot, nodes))
100+
}
101+
102+
// noFreeWorkerError turns the scheduler's refusal into the caller's status.
103+
// Without a node restriction the pool is full: ResourceExhausted, which the
104+
// router parks on until a worker frees up. Under one -- a PAUSED actor's
105+
// local snapshot -- the scan says which of three states the actor is in:
106+
//
107+
// - eligible workers on the snapshot's node(s), all full: as above, with
108+
// the node named, so the message does not point at cluster capacity;
109+
// - the node(s) still in the cluster but no eligible worker on them (a
110+
// pool rolling, a pool moved off the node): ResourceExhausted with the
111+
// node named, still worth parking on -- a worker may come back;
112+
// - the node(s) gone from the cluster: the snapshot is lost
113+
// (localSnapshotLost).
114+
//
115+
// A node is gone only when no ACTIVE worker of any pool reports it and the
116+
// Node object is absent; either signal alone keeps the actor waiting.
117+
func (w *ActorWorkflow) noFreeWorkerError(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, err error) error {
118+
var restricted *scheduling.NoCapacityError
119+
if !errors.As(err, &restricted) {
120+
return status.Errorf(codes.ResourceExhausted, "no free workers available")
121+
}
122+
nodes := restricted.RequiredNodes
123+
switch {
124+
case restricted.EligibleOnRequiredNodes > 0:
125+
return status.Errorf(codes.ResourceExhausted, "actor's local snapshot is on node(s) %v but all eligible workers on those nodes are busy", nodes)
126+
case restricted.WorkersOnRequiredNodes > 0:
127+
return status.Errorf(codes.ResourceExhausted, "actor's local snapshot is on node(s) %v but no eligible workers exist on those nodes", nodes)
128+
}
129+
gone, err := w.localSnapshotNodesGone(nodes)
130+
if err != nil {
131+
return err
132+
}
133+
if !gone {
134+
return status.Errorf(codes.ResourceExhausted, "actor's local snapshot is on node(s) %v but no workers exist on those nodes", nodes)
135+
}
136+
return w.localSnapshotLost(ctx, actorRef, actor, ateattr.OperationResume, nodes)
137+
}

0 commit comments

Comments
 (0)