Repository navigation
fix(kubernetes-nodes): announce role labels and GPUs for scale from zero - #4949
Draft
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Draft
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Conversation
github-actions
Bot
added
size/M
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 tasks done
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
2 times, most recently
from
October 27, 2026 21:57
700b686 to
edb15a6
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
fix/kubernetes-nodes-scale-from-zero
branch
2 times, most recently
from
October 29, 2026 13:55
c713900 to
fca7b60
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
from
October 29, 2026 13:55
edb15a6 to
66c6a82
Compare
A pool scaled to zero has no Node, so the cluster-autoscaler builds its template node from the MachineDeployment's capacity annotations alone. The labels annotation carried only the pool's own labels, and nothing announced GPUs. A pod selecting a role, such as the tenant ingress-nginx, or selecting gpu=on, never scaled its pool up from zero, and neither did a pod requesting a GPU. The labels annotation now also carries the role labels and, on a GPU pool, gpu=on. A GPU pool announces its GPUs as nvidia.com/gpu with the number of attached GPUs: inside the tenant the device plugin serves every attached GPU under that name, whatever host device the pool names. HAMi splits each GPU into several, so with HAMi enabled a pool scaled to zero is planned for fewer GPUs than it serves. GPU nodes still do not carry the cluster-api/accelerator label the autoscaler needs to hold a new node as unready until its GPUs show up, so a scale-up may request extra nodes while the device plugin starts. Setting it goes through the bootstrap config and would roll every GPU pool. These are MachineDeployment annotations, so no machine rolls. A pool without roles or GPUs renders as before. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
from
October 29, 2026 15:56
66c6a82 to
f317a98
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
fix/kubernetes-nodes-scale-from-zero
branch
from
October 29, 2026 15:56
fca7b60 to
5955b33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
A tenant worker pool scaled to zero didn't scale up for pods that need one of its roles, its GPU label or its GPUs. Such pods stayed Pending. Examples: the tenant ingress-nginx on a role pool, a pod with
nodeSelector: {gpu: "on"}, a pod requestingnvidia.com/gpu. With no Node in the pool, cluster-autoscaler builds the template node from the MachineDeployment's capacity annotations only, and those had no role labels, nogpu=onand no GPUs. Now they have all three. GPUs are announced asnvidia.com/gpuwith the number of attached GPUs. The commit message explains that name and the HAMi undercount.Stacked on #4927, the base branch is
feat/kubernetes-nodes-labels-taints.One thing is left out on purpose and needs your call. GPU nodes don't get the
cluster-api/acceleratorlabel. The autoscaler uses it to treat a new node as unready until its GPUs show up, so without it the autoscaler may add extra nodes while the device plugin starts. Adding the label would roll every existing GPU pool.No machine rolls with this change, only MachineDeployment annotations move. A pool without roles or GPUs renders byte for byte as before, and its golden snapshots didn't change. I checked the format against the cluster-autoscaler 1.32.0 source and covered it with helm unit tests. Not tested on a cluster yet.
Screenshots
No UI changes.
Downstream repositories
The diff touches only
templates/nodegroup.yamland the chart's tests: no values, schema, README or API change.Release note