Repository navigation
Commit 1b8f2e0
authored
atelet: run at system-cluster-critical (agent-substrate#3328)
Part of agent-substrate#3101.
When local snapshots fill a node's disk, the kubelet evicts atelet and
rejects every replacement while the node has DiskPressure. Uploads and
detaches stop, so nothing frees the space. Only atelet frees what
substrate writes under `/var/lib/ate`, because the kubelet doesn't count
that hostPath toward any pod and its image GC doesn't reach it.
The kubelet never evicts a pod at priority 2000000000 or above, and it
admits one under DiskPressure. So atelet now runs at
`system-cluster-critical`. In test runs based off of agent-substrate#3101 atelet was
never evicted for either `system-node-critical` or
`system-cluster-critical` and uploaded on schedule.
`system-node-critical` would also give atelet `oom_score_adj` -997. With
no memory limits on atelet and objectstore-plugin, a leak in either
would then make the kernel kill workers first. Per discussion in this PR
the manifest comment says to switch once both containers have memory
limits.
GKE admits critical pods outside kube-system only in a namespace with a
`ResourceQuota` scoped to their class. The manifest adds one that allows
both classes, so switching later needs only the `priorityClassName`
change. Its `pods: 1G` limit matches GKE's own `gcp-critical-pods`
quota. Clusters that don't restrict the critical classes, like kind,
admit atelet without it.
The kubelet enforces an emptyDir `sizeLimit` only by evicting the pod,
so the `asset-staging` limit no longer applies. I removed it and the
test that checked `--asset-staging-capacity` fits under it.
An install upgraded across this change keeps the old `ate-node-critical`
PriorityClass. Delete it with `kubectl delete priorityclass
ate-node-critical`.
I used an AI agent (Claude Code) to run the agent-substrate#3101 tests, make this
change and draft this description.
---------
Signed-off-by: Ivy <[email protected]>1 parent 17fa858 commit 1b8f2e0
2 files changed
Lines changed: 31 additions & 79 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
| 17 | + | |
25 | 18 | | |
26 | 19 | | |
27 | 20 | | |
| |||
43 | 36 | | |
44 | 37 | | |
45 | 38 | | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
103 | 112 | | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
108 | 123 | | |
109 | 124 | | |
110 | 125 | | |
| |||
128 | 143 | | |
129 | 144 | | |
130 | 145 | | |
131 | | - | |
| 146 | + | |
132 | 147 | | |
133 | 148 | | |
134 | 149 | | |
| |||
172 | 187 | | |
173 | 188 | | |
174 | 189 | | |
175 | | - | |
176 | | - | |
| 190 | + | |
| 191 | + | |
177 | 192 | | |
178 | 193 | | |
179 | 194 | | |
| |||
371 | 386 | | |
372 | 387 | | |
373 | 388 | | |
374 | | - | |
375 | | - | |
376 | | - | |
| 389 | + | |
| 390 | + | |
377 | 391 | | |
378 | | - | |
379 | | - | |
| 392 | + | |
380 | 393 | | |
381 | 394 | | |
382 | 395 | | |
| |||
0 commit comments