atelet: run at system-cluster-critical by igooch · Pull Request #3328 · agent-substrate/substrate

igooch · 2026-10-27T17:57:01Z

Part of #3101.

When local snapshots fill a node's disk, the kubelet evicts atelet and rejects every replacement while the node has DiskPressure. Uploads and detaches stop, so nothing frees the space. Only atelet frees what substrate writes under /var/lib/ate, because the kubelet doesn't count that hostPath toward any pod and its image GC doesn't reach it.

The kubelet never evicts a pod at priority 2000000000 or above, and it admits one under DiskPressure. So atelet now runs at system-cluster-critical. In test runs based off of #3101 atelet was never evicted for either system-node-critical or system-cluster-critical and uploaded on schedule.

system-node-critical would also give atelet oom_score_adj -997. With no memory limits on atelet and objectstore-plugin, a leak in either would then make the kernel kill workers first. Per discussion in this PR the manifest comment says to switch once both containers have memory limits.

GKE admits critical pods outside kube-system only in a namespace with a ResourceQuota scoped to their class. The manifest adds one that allows both classes, so switching later needs only the priorityClassName change. Its pods: 1G limit matches GKE's own gcp-critical-pods quota. Clusters that don't restrict the critical classes, like kind, admit atelet without it.

The kubelet enforces an emptyDir sizeLimit only by evicting the pod, so the asset-staging limit no longer applies. I removed it and the test that checked --asset-staging-capacity fits under it.

An install upgraded across this change keeps the old ate-node-critical PriorityClass. Delete it with kubectl delete priorityclass ate-node-critical.

I used an AI agent (Claude Code) to run the #3101 tests, make this change and draft this description.

When local snapshots fill a node's disk, the kubelet evicts atelet and rejects its replacements until the pressure clears. Without atelet, nothing uploads or detaches the snapshots, so the node can stay in DiskPressure until someone frees space by hand. The kubelet never evicts a critical pod under node pressure and admits one while the node is under pressure. Some clusters admit critical pods outside kube-system only in a namespace with a matching ResourceQuota, so the manifest adds one scoped to system-node-critical. Signed-off-by: Ivy <[email protected]>

Benjamin Elder (BenTheElder)

Agent assisted sources and auditing, review by me :-)

Benjamin Elder (BenTheElder) · 2026-10-27T23:02:43Z

    spec:
      serviceAccountName: atelet
-      priorityClassName: ate-node-critical
+      priorityClassName: system-node-critical


Did we consider system-cluster-critical? The kubelet treats any priority >= 2000000000 as critical, so it still won't evict atelet or reject its replacement, but the -997 oom_score_adj only applies to the system-node-critical name, and it can't preempt cluster-critical pods.
https://github.com/kubernetes/kubernetes/blob/f28b4c9efbca5c5c0af716d9f2d5702667ee8a45/pkg/kubelet/types/pod_update.go#L188-L195

atelet and objectstore-plugin have no memory limits, and an OOM kill in a worker container takes the whole container and all of its actors with it, so I'd rather a leaking atelet be the one the kernel picks. If we stay at node-critical I think we need memory limits on both ... we might want to do that, but maybe not now.

We did consider it. atelet works much like the kubelet. ateapi reaches every worker through the atelet on its node and won't dial unless there is exactly one, so a node can't host actors without it. system-node-critical is the class for agents like that.

The -997 oom_score_adj is the main difference, and it's what we want when workers fill the node's memory. A worker OOM kill loses its actors' state. atelet restarting would stop activations and checkpoints on the whole node. Both classes already keep atelet safe from kubelet eviction, so the score only matters once the kernel has to pick.

I agree about the leak risk. -997 is what we want when workers fill the node, but a leaking atelet would take workers down with it. Should we add memory limits to atelet and objectstore-plugin in a follow-up, so a leak only restarts atelet? Does that need to be done as part of M3 or can it wait?

[...] so a node can't host actors without it

I mean, it can't interact with the actors, but they will stay up and not lose state if it goes down.

The -997 oom_score_adj is the main difference, and it's what we want when workers fill the node's memory. A worker OOM kill loses its actors' state. atelet restarting would stop activations and checkpoints on the whole node. Both classes already keep atelet safe from kubelet eviction, so the score only matters once the kernel has to pick.

Er, but my point is that if we do have to pick, we'd rather pick atelet than ate-worker. atelet can be recovered. ate-worker loses actor data.

[...] but a leaking atelet would take workers down with it [...]

how? if atelet crashes and restarts, it should re-discover the worker pod(s), without losing any critical state.

Should we add memory limits to atelet and objectstore-plugin in a follow-up, so a leak only restarts atelet?

Limits will require tuning, but I think we can accomplish the same by using system-cluster-critical instead of node-critical. Despite the names the only difference is a small amount of priority and the oom_score_adj.

If we're at the point of evicting a pod on one of these nodes, probably atelet actually is the most viable to evict.

Another system-node-critical pod would be something like kube-proxy, without which we can't use atelet <> ate-api anyhow.

My main concern is disk. Only atelet frees what substrate writes under /var/lib/ate, which includes local snapshots, actor image layers, actor bundles, and actor dirs. The kubelet doesn't count that hostPath toward any pod, and its image and container GC don't reach it (in #3101 image GC freed 0 bytes). So atelet has to stay on the node, or the node can't recover from disk pressure without manual cleanup. Agreed that both classes keep atelet on the node, so I'm assuming you mean "OOM kill [the container]" not "viable to evict [the pod]"?

On memory, I don't think system-cluster-critical gets us the same as limits. The kubelet's memory-pressure eviction skips critical pods and evicts the next pod in rank under either class. The class only matters when the kernel OOMs before the kubelet acts.

Memory pressure system-node-critical (-997) system-cluster-critical (~995 for a 128Mi request)

atelet leaks, kubelet acts first Workers evicted Workers evicted

atelet leaks, kernel acts first Workers OOM-killed atelet OOM-killed, restarts in place

Workers spike, kernel acts first A worker OOM-killed atelet or objectstore-plugin likely killed first, which frees little and interrupts suspends and uploads. Then a worker.

With a memory limit, an atelet leak OOM-kills only atelet before the node runs low, under either class. I slightly prefer system-node-critical with limits on atelet and objectstore-plugin in a follow-up. Although tuning them will take some work, since some of their memory use isn't bounded yet. If you feel strongly about system-cluster-critical, I can make the switch.

Er, but my point is that if we do have to pick, we'd rather pick atelet than ate-worker. atelet can be recovered. ate-worker loses actor data.

Hmm, but atelet is singleton per node that services all workers. If atelet is gone, nothing cleans up the shared snapshots, and all checkpoint could fail due to lack of disk space.

Memory limits will require tuning.

To me this is the main reason not to cherrypick this PR into release-1.0 branch. Do we have enough time to properly tune the memory limits? As a temporary workaround, Chenyi Wang (@chw120) igooch can we create a playbook on how to configure the atelet's detach timer and disk capacity to avoid running into #3101 ?

After GA we should look into separating /var/lib/ate/snapshots to its own dedicated disk instead of sharing the boot disk, it will prevent the issue where disk pressure doesn't trigger kubelet to evict atelet.

We also need to handle our own eviction BEFORE kubelet does. #3478 (comment)

Could we support both depending on whether limits.memory is set on atelet / objectstore-plugin?

Without limits.memory (default in atelet.yaml): Use system-cluster-critical. It prevents kubelet from evicting atelet under DiskPressure ([Bug]: Node-local snapshots from async suspends can fill a node's disk; kubelet then evicts the workers and atelet, and the node stays in DiskPressure #3101), without risking an unbounded atelet leak OOM-killing ateworker pods (so it's also safe for release-1.0).

With limits.memory configured (either by operators who set limits in their deployment, or in a follow-up PR after tuning): Recommend system-node-critical (and note this in the atelet.yaml comment), since limits.memory will restart only atelet if it leaks, while -997 protects atelet from node OOMs.

Allow both in the atelet-critical-pods ResourceQuota (values: ["system-node-critical", "system-cluster-critical"]) so GKE admits either one out of the box.

Both classes keep atelet on the node for disk, the kubelet skips any critical pod in evictPod, so that part is equal.

I read your table the other way for memory: With cluster-critical the worst case is an atelet restart, with node-critical a leak kills workers and loses actor data.

I'd take cluster-critical now (+1 to chw120's suggestion to allow both in the quota), and once atelet has tuned limits it gets -997 anyway if requests == limits, Guaranteed pods get the same adj regardless of class.

https://github.com/kubernetes/kubernetes/blob/a972b672c7612c4892557f3d6d2a952c741c5d1f/pkg/kubelet/qos/policy.go#L47-L58

I switched the default to system-cluster-critical as you proposed, and the quota now allows both classes so we can move to system-node-critical once atelet and objectstore-plugin have memory limits.

Reruns on my GKE test cluster at system-cluster-critical matched the node-critical runs, with atelet never evicted and every snapshot uploaded.

The kubelet enforces an emptyDir sizeLimit only by evicting the pod, and it never evicts a critical pod. --asset-staging-capacity is the only bound now, so remove the limit and the test that checked the capacity fits under it. Link the upstream docs and quota behind the critical-pods ResourceQuota. Signed-off-by: Ivy <[email protected]>

Benjamin Elder (BenTheElder) · 2026-10-29T05:06:54Z

cc Julian Gutierrez Oschmann (@juli4n)

ajaysundar.k (ajaysundark) · 2026-10-29T17:13:01Z

/cc

Benjamin Elder (BenTheElder) · 2026-11-02T23:13:12Z

-apiVersion: scheduling.k8s.io/v1
-kind: PriorityClass
+# 5. Allow system-node-critical pods in ate-system
+# atelet runs at system-node-critical because a node can't host actors


non-blocking: actors do stay up without atelet (see the priorityClassName thread), whichever class we land on can this give the disk reason instead: atelet is the only thing that frees /var/lib/ate.

Benjamin Elder (BenTheElder) · 2026-11-02T23:13:12Z

+# its replacement. Some clusters, like GKE, admit critical pods outside
+# kube-system only with a quota scoped to their class.
+# https://kubernetes.io/docs/concepts/policy/resource-quotas/#limiting-priorityclass-consumption-by-default
+# The limit matches the upstream kube-system quota for critical pods, so it


non-blocking: I think this was my goof the "default" here is really kube-up.sh

We should probably describe the upstream mechanism instead.
e.g. clusters whose ResourceQuota admission config sets limitedResources for the critical classes ...

Benjamin Elder (BenTheElder) · 2026-11-03T00:56:52Z

    spec:
      serviceAccountName: atelet
-      priorityClassName: ate-node-critical
+      priorityClassName: system-node-critical


Both classes keep atelet on the node for disk, the kubelet skips any critical pod in evictPod, so that part is equal.

I read your table the other way for memory: With cluster-critical the worst case is an atelet restart, with node-critical a leak kills workers and loses actor data.

I'd take cluster-critical now (+1 to chw120's suggestion to allow both in the quota), and once atelet has tuned limits it gets -997 anyway if requests == limits, Guaranteed pods get the same adj regardless of class.

https://github.com/kubernetes/kubernetes/blob/a972b672c7612c4892557f3d6d2a952c741c5d1f/pkg/kubelet/qos/policy.go#L47-L58

Benjamin Elder (BenTheElder) · 2026-11-03T00:57:38Z

cc Tim Hockin (@thockin) Eitan Yarmush (@EItanya)

Both critical classes keep the kubelet from evicting atelet and admit its replacement under DiskPressure. system-node-critical also gives atelet oom_score_adj -997, so with no memory limits a leak in atelet or objectstore-plugin makes the kernel kill workers first. Use system-cluster-critical until both containers have memory limits, and let the quota admit either class. Signed-off-by: Ivy <[email protected]>

Chenyi Wang (chw120)

LGTM. Thanks igooch for fixing it.

Signed-off-by: Ivy <[email protected]>

Part of #3101. When local snapshots fill a node's disk, the kubelet evicts atelet and rejects every replacement while the node has DiskPressure. Uploads and detaches stop, so nothing frees the space. Only atelet frees what substrate writes under `/var/lib/ate`, because the kubelet doesn't count that hostPath toward any pod and its image GC doesn't reach it. The kubelet never evicts a pod at priority 2000000000 or above, and it admits one under DiskPressure. So atelet now runs at `system-cluster-critical`. In test runs based off of #3101 atelet was never evicted for either `system-node-critical` or `system-cluster-critical` and uploaded on schedule. `system-node-critical` would also give atelet `oom_score_adj` -997. With no memory limits on atelet and objectstore-plugin, a leak in either would then make the kernel kill workers first. Per discussion in this PR the manifest comment says to switch once both containers have memory limits. GKE admits critical pods outside kube-system only in a namespace with a `ResourceQuota` scoped to their class. The manifest adds one that allows both classes, so switching later needs only the `priorityClassName` change. Its `pods: 1G` limit matches GKE's own `gcp-critical-pods` quota. Clusters that don't restrict the critical classes, like kind, admit atelet without it. The kubelet enforces an emptyDir `sizeLimit` only by evicting the pod, so the `asset-staging` limit no longer applies. I removed it and the test that checked `--asset-staging-capacity` fits under it. An install upgraded across this change keeps the old `ate-node-critical` PriorityClass. Delete it with `kubectl delete priorityclass ate-node-critical`. I used an AI agent (Claude Code) to run the #3101 tests, make this change and draft this description. --------- Signed-off-by: Ivy <[email protected]> (cherry picked from commit 1b8f2e0) Signed-off-by: dberkov <[email protected]>

Move Commit from the v1.0.0-rc.2 tag (e59281c) to v1.0.0-rc.3 (4ee8f66ac237adfc3303086b12275aefc150fd6f), five commits later. go.mod is unchanged (go 1.27.0), so MinGoVersion stays; substrate-pin-check agrees. What rc.3 changes for the installer: - atelet runs at system-cluster-critical (agent-substrate/substrate#3328) instead of upstream's own ate-node-critical PriorityClass, which the manifests no longer ship. GKE's ResourceQuota admission config limits the system-* classes to namespaces with a quota scoped to them, and rc.3's atelet.yaml ships that quota (atelet-critical-pods in ate-system, scopeSelector on PriorityClass, pods: 1G) for exactly that reason, so `deploy ate-system` admits the DaemonSet with no change here. The system pools' taints and labels, the cordon and placement checks (which look at ate-api-server, not atelet) and the autoscaling step are untouched by it. An upgrade from rc.2 leaves the orphaned ate-node-critical PriorityClass behind, unused and harmless. - The gVisor durable directory root is now mode 0777 (#3457), the objectstore-plugin exchange skips the system roots for a `.svc` issuer, and the control-plane binaries resolve gRPC peers through a k8s resolver. None of these touch what the installer passes. - cmd/ate-setup and docs/guides/upgrade.md are byte-identical to rc.2, so the micro-VM staging path, the delete and demo commands, the cordon and placement checks, and the install record format are as at rc.2. All eleven core images are published on GHCR at v1.0.0-rc.3; the demo images still are not. Verified: make substrate-pin-check, make verify, go test ./... in tools/setup-gcp, and in installer/ gofmt, go vet and go test for the root, internal and uishots packages, the ui suite included. The dry-run frames render; guard.svg is regenerated for the new short commit and the other four are byte-identical. TestFetchTreesFetchesBothCommits fails on this machine before and after the change, for an environmental reason unrelated to the pin.

igooch force-pushed the atelet-system-node-critical branch from 876a493 to 61b99ec Compare October 27, 2026 18:14

igooch force-pushed the atelet-system-node-critical branch from 61b99ec to 9cbf717 Compare October 27, 2026 18:16

igooch marked this pull request as ready for review October 27, 2026 18:17

igooch requested a review from Benjamin Elder (BenTheElder) October 27, 2026 18:38

igooch added this to the M3 milestone Oct 27, 2026

Bowei Du (bowei) added area/node area/reliability kind/bug Something isn't working / bugfixes labels Oct 27, 2026

igooch mentioned this pull request Oct 27, 2026

docs: size the node disk for local snapshots #3329

Open

Benjamin Elder (BenTheElder) reviewed Oct 28, 2026

View reviewed changes

igooch requested a review from Benjamin Elder (BenTheElder) October 28, 2026 17:25

Benjamin Elder (BenTheElder) self-assigned this Oct 29, 2026

Dmitry Berkovich (dberkov) requested a review from Tim Hockin (thockin) October 29, 2026 17:13

Dmitry Berkovich (dberkov) added the milestone-ack label Oct 29, 2026

igooch requested a review from Julian Gutierrez Oschmann (juli4n) November 2, 2026 17:23

Benjamin Elder (BenTheElder) reviewed Nov 3, 2026

View reviewed changes

igooch changed the title ~~atelet: run at system-node-critical~~ atelet: run at system-cluster-critical Nov 3, 2026

igooch requested review from Benjamin Elder (BenTheElder), Chenyi Wang (chw120) and Zoe Zhao (zoez7) November 3, 2026 03:26

Chenyi Wang (chw120) approved these changes Nov 3, 2026

View reviewed changes

Dmitry Berkovich (dberkov) assigned Eitan Yarmush (EItanya) Nov 3, 2026

igooch force-pushed the atelet-system-node-critical branch from 9958dd9 to 6de26e5 Compare November 3, 2026 18:41

atelet: describe the quota admission mechanism

bc0c919

Signed-off-by: Ivy <[email protected]>

igooch force-pushed the atelet-system-node-critical branch from 6de26e5 to bc0c919 Compare November 3, 2026 18:41

Eitan Yarmush (EItanya) approved these changes Nov 4, 2026

View reviewed changes

Tim Hockin (thockin) added this pull request to the merge queue Nov 4, 2026

Benjamin Elder (BenTheElder) approved these changes Nov 4, 2026

View reviewed changes

Merged via the queue into agent-substrate:main with commit 1b8f2e0 Nov 4, 2026
5 checks passed

Benjamin Elder (BenTheElder) mentioned this pull request Nov 4, 2026

[Bug]: Node-local snapshots from async suspends can fill a node's disk; kubelet then evicts the workers and atelet, and the node stays in DiskPressure #3101

Open

2 tasks done

Dmitry Berkovich (dberkov) mentioned this pull request Nov 5, 2026

[release-1.0] Cherry-pick 5 main changes through 2289ade15 #3645

Merged

2 tasks done

Aditya Shantanu (aditya-shantanu) mentioned this pull request Nov 5, 2026

Pin Substrate to v1.0.0-rc.3 ai-on-gke/substrate-gke#149

Merged

igooch mentioned this pull request Nov 9, 2026

atelet: keep node caches across a clean reboot #3740

Open

Memory pressure	`system-node-critical` (-997)	`system-cluster-critical` (~995 for a 128Mi request)
atelet leaks, kubelet acts first	Workers evicted	Workers evicted
atelet leaks, kernel acts first	Workers OOM-killed	atelet OOM-killed, restarts in place
Workers spike, kernel acts first	A worker OOM-killed	atelet or objectstore-plugin likely killed first, which frees little and interrupts suspends and uploads. Then a worker.

Conversation

igooch commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 29, 2026

Uh oh!

ajaysundar.k (ajaysundark) commented Oct 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) commented Nov 3, 2026

Uh oh!

Chenyi Wang (chw120) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

igooch commented Oct 27, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) Oct 27, 2026 •

edited

Loading