Skip to content

Commit 1b8f2e0

Browse files
authored
atelet: run at system-cluster-critical (agent-substrate#3328)
Part of agent-substrate#3101. When local snapshots fill a node's disk, the kubelet evicts atelet and rejects every replacement while the node has DiskPressure. Uploads and detaches stop, so nothing frees the space. Only atelet frees what substrate writes under `/var/lib/ate`, because the kubelet doesn't count that hostPath toward any pod and its image GC doesn't reach it. The kubelet never evicts a pod at priority 2000000000 or above, and it admits one under DiskPressure. So atelet now runs at `system-cluster-critical`. In test runs based off of agent-substrate#3101 atelet was never evicted for either `system-node-critical` or `system-cluster-critical` and uploaded on schedule. `system-node-critical` would also give atelet `oom_score_adj` -997. With no memory limits on atelet and objectstore-plugin, a leak in either would then make the kernel kill workers first. Per discussion in this PR the manifest comment says to switch once both containers have memory limits. GKE admits critical pods outside kube-system only in a namespace with a `ResourceQuota` scoped to their class. The manifest adds one that allows both classes, so switching later needs only the `priorityClassName` change. Its `pods: 1G` limit matches GKE's own `gcp-critical-pods` quota. Clusters that don't restrict the critical classes, like kind, admit atelet without it. The kubelet enforces an emptyDir `sizeLimit` only by evicting the pod, so the `asset-staging` limit no longer applies. I removed it and the test that checked `--asset-staging-capacity` fits under it. An install upgraded across this change keeps the old `ate-node-critical` PriorityClass. Delete it with `kubectl delete priorityclass ate-node-critical`. I used an AI agent (Claude Code) to run the agent-substrate#3101 tests, make this change and draft this description. --------- Signed-off-by: Ivy <[email protected]>
1 parent 17fa858 commit 1b8f2e0

2 files changed

Lines changed: 31 additions & 79 deletions

File tree

‎cmd/objectstore-plugin/main_test.go‎

Lines changed: 1 addition & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,7 @@
1414

1515
package main
1616

17-
import (
18-
"os"
19-
"strings"
20-
"testing"
21-
22-
appsv1 "k8s.io/api/apps/v1"
23-
"sigs.k8s.io/yaml"
24-
)
17+
import "testing"
2518

2619
func TestParseCapacity(t *testing.T) {
2720
for _, tc := range []struct {
@@ -43,57 +36,3 @@ func TestParseCapacity(t *testing.T) {
4336
}
4437
}
4538
}
46-
47-
// TestInstallStagingCapacityFitsVolume checks that the install's node plugin
48-
// stages no more than its asset-staging volume holds, so concurrent asset
49-
// downloads cannot get atelet's pod evicted.
50-
func TestInstallStagingCapacityFitsVolume(t *testing.T) {
51-
raw, err := os.ReadFile("../../manifests/ate-install/atelet.yaml")
52-
if err != nil {
53-
t.Fatal(err)
54-
}
55-
var ds *appsv1.DaemonSet
56-
for _, doc := range strings.Split(string(raw), "\n---") {
57-
var obj appsv1.DaemonSet
58-
if err := yaml.Unmarshal([]byte(doc), &obj); err != nil {
59-
t.Fatal(err)
60-
}
61-
if obj.Kind == "DaemonSet" && obj.Name == "atelet" {
62-
ds = &obj
63-
}
64-
}
65-
if ds == nil {
66-
t.Fatal("atelet.yaml has no atelet DaemonSet")
67-
}
68-
var capacityArg string
69-
for _, c := range ds.Spec.Template.Spec.InitContainers {
70-
if c.Name != "objectstore-plugin" {
71-
continue
72-
}
73-
for _, a := range c.Args {
74-
if v, ok := strings.CutPrefix(a, "--asset-staging-capacity="); ok {
75-
capacityArg = v
76-
}
77-
}
78-
}
79-
if capacityArg == "" {
80-
t.Fatal("the objectstore-plugin sidecar does not set --asset-staging-capacity")
81-
}
82-
capacity, err := parseCapacity(capacityArg)
83-
if err != nil {
84-
t.Fatal(err)
85-
}
86-
for _, v := range ds.Spec.Template.Spec.Volumes {
87-
if v.Name != "asset-staging" {
88-
continue
89-
}
90-
if v.EmptyDir == nil || v.EmptyDir.SizeLimit == nil {
91-
t.Fatal("asset-staging is not an emptyDir with a sizeLimit")
92-
}
93-
if limit := v.EmptyDir.SizeLimit.Value(); capacity >= limit {
94-
t.Errorf("--asset-staging-capacity=%s (%d bytes) leaves no room below the asset-staging sizeLimit of %d bytes", capacityArg, capacity, limit)
95-
}
96-
return
97-
}
98-
t.Fatal("atelet DaemonSet has no asset-staging volume")
99-
}

‎manifests/ate-install/atelet.yaml‎

Lines changed: 30 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -95,16 +95,31 @@ roleRef:
9595
name: atelet-endpointslices
9696
apiGroup: rbac.authorization.k8s.io
9797
---
98-
# 5. Create PriorityClass
99-
# Prioritize atelet over workers, which need it for actor activation.
100-
# Use the highest user-defined priority; system-node-critical requires kube-system.
101-
apiVersion: scheduling.k8s.io/v1
102-
kind: PriorityClass
98+
# 5. Allow critical pods in ate-system
99+
# atelet runs at system-cluster-critical so the kubelet neither evicts it
100+
# under node pressure nor rejects its replacement. Only atelet frees what
101+
# substrate writes under /var/lib/ate, so a node can't recover from
102+
# DiskPressure without it. system-node-critical also shields atelet from the
103+
# kernel OOM killer, so switch to it only once atelet and objectstore-plugin
104+
# have memory limits.
105+
# Clusters whose ResourceQuota admission config sets limitedResources for the
106+
# critical classes, like GKE, admit those pods only in a namespace with a
107+
# quota scoped to their class. This quota allows both classes, and its limit
108+
# never blocks atelet.
109+
# https://kubernetes.io/docs/concepts/policy/resource-quotas/#limiting-priorityclass-consumption-by-default
110+
apiVersion: v1
111+
kind: ResourceQuota
103112
metadata:
104-
name: ate-node-critical
105-
value: 1000000000
106-
globalDefault: false
107-
description: "Node-local ate components that every actor activation depends on."
113+
name: atelet-critical-pods
114+
namespace: ate-system
115+
spec:
116+
hard:
117+
pods: 1G
118+
scopeSelector:
119+
matchExpressions:
120+
- scopeName: PriorityClass
121+
operator: In
122+
values: ["system-node-critical", "system-cluster-critical"]
108123

109124
---
110125
# 6. Create DaemonSet
@@ -128,7 +143,7 @@ spec:
128143
prometheus.io/port: "9090"
129144
spec:
130145
serviceAccountName: atelet
131-
priorityClassName: ate-node-critical
146+
priorityClassName: system-cluster-critical
132147
# Budget for the full shutdown sequence: --drain-delay (0 for atelet) +
133148
# --drain-timeout (5m). Sized long because Checkpoint/Restore stream
134149
# multi-GiB snapshots and must not be force-cancelled mid-upload. The sum
@@ -172,8 +187,8 @@ spec:
172187
- --root=/var/lib/ate
173188
- --asset-staging-dir=/var/lib/objectstore-plugin/asset-staging
174189
# atelet's 8 GiB per-asset cap, so any asset fits. Each download
175-
# reserves its object's size; ones past the capacity wait their turn,
176-
# so the asset-staging volume below never fills up.
190+
# reserves its object's size; ones past the capacity wait their turn.
191+
# This bounds the asset-staging volume below.
177192
- --asset-staging-capacity=8Gi
178193
# The plugin listens on a Unix socket, which a gRPC probe cannot
179194
# reach, and the image has no shell, so the binary checks itself.
@@ -371,12 +386,10 @@ spec:
371386
volumes:
372387
- name: objectstore-plugin
373388
emptyDir: {}
374-
# Holds asset downloads until they are verified. The plugin keeps them
375-
# within --asset-staging-capacity; the limit adds 1Gi for filesystem
376-
# overhead, since kubelet evicts the pod if the volume exceeds it.
389+
# Holds asset downloads until they are verified.
390+
# --asset-staging-capacity bounds its size.
377391
- name: asset-staging
378-
emptyDir:
379-
sizeLimit: 9Gi
392+
emptyDir: {}
380393
- name: ate-base
381394
hostPath:
382395
path: /var/lib/ate

0 commit comments

Comments
 (0)