Skip to content

Commit b666805

Browse files
committed
fix(kubernetes-nodes): size reservations from the cores the VM actually gets
On the kubevirt substrate resources.cpu is written into domain.cpu.cores as spelled and decoded as YAML, so "010" gives the VM eight cores, while the kubelet reservations and the cluster-autoscaler capacity annotation parsed the same string as decimal ten. The autoscaler then planned for ten cores on a node that has eight. Keep the YAML-decoded value the range check already computes and derive both readers from it on kubevirt. The emitted cores line and the machine template name do not change for any spelling, so no pool rolls; for a leading-zero pool the capacity annotation and the reservations now match the VM. Proxmox parses the quantity as decimal on both sides and is unchanged. The field description says the value is decoded as YAML. This closes #4554 without the migration and the render-time refusal it proposes: a refusal breaks stored pools on their next upgrade, a schema pattern breaks updates of existing CRs, and a platform-wide migration of every KubernetesNodes and nodeGroups entry is a lot of blast radius for a spelling nobody has reported in a live cluster. Closes #4554 Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent aed7fa8 commit b666805

7 files changed

Lines changed: 107 additions & 7 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/README.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,16 @@
104104
or 1e1 pass as the integers they decode to, and a leading-zero
105105
spelling passes as whatever YAML reads it as. A bare numeric zero
106106
is left alone: it is falsy, so the pool renders without domain.cpu
107-
and is sized by its instancetype. */}}
107+
and is sized by its instancetype.
108+
109+
The decoded value is kept: it is the core count the VM actually gets,
110+
and everything below that derives a number from resources.cpu on this
111+
substrate - the kubelet reservations and the cluster-autoscaler
112+
capacity annotation - reads it from here rather than parsing the raw
113+
string as a decimal. The two readings differ for a leading zero:
114+
YAML reads "010" as octal, so the VM has 8 cores, and a decimal parse
115+
would have the autoscaler plan for 10 on a node that has 8. */}}
116+
{{- $kubevirtCores := "" }}
108117
{{- $cpuValue := (.Values.resources | default dict).cpu }}
109118
{{- if and (not $isProxmox) (not (kindIs "invalid" $cpuValue)) (not (and (not (kindIs "string" $cpuValue)) (eq (float64 $cpuValue) 0.0))) }}
110119
{{- $cpu := printf "%v" $cpuValue }}
@@ -113,6 +122,7 @@
113122
{{- if not (and $isNumber (gt (float64 $cores) 0.0) (lt (float64 $cores) 4294967296.0) (eq (floor $cores) (float64 $cores))) }}
114123
{{- fail (printf "nodeGroup %s: resources.cpu (%s) must be a whole number of vCPU cores between 1 and 4294967295, such as 2. On the kubevirt substrate it becomes the worker VM's domain.cpu.cores, which KubeVirt declares as an unsigned 32-bit integer, so a millicore or fractional quantity cannot size the guest. Set a core count, or remove the field." $groupName $cpu) }}
115124
{{- end }}
125+
{{- $kubevirtCores = printf "%.0f" (float64 $cores) }}
116126
{{- end }}
117127
{{- /* Assembled once in _helpers.tpl and read back here and in
118128
talos-reconcile-job.yaml, so the TalosConfigTemplate name this file
@@ -745,7 +755,11 @@ spec:
745755
{{- end }}
746756
{{- $effectiveCpu := "" }}
747757
{{- if and $group.resources $group.resources.cpu }}
748-
{{- $effectiveCpu = $group.resources.cpu | toString }}
758+
{{- /* On kubevirt the count the VM gets is the YAML-decoded one kept at
759+
the top of this file (see the note there); proxmox hands the raw
760+
quantity to the machine template through cpuToMillicores, so it
761+
reads the same string the template does. */}}
762+
{{- $effectiveCpu = ternary ($group.resources.cpu | toString) $kubevirtCores $isProxmox }}
749763
{{- else if and $instanceType $instanceType.spec $instanceType.spec.cpu $instanceType.spec.cpu.guest }}
750764
{{- $effectiveCpu = $instanceType.spec.cpu.guest | toString }}
751765
{{- end }}

‎packages/apps/kubernetes-nodes/tests/resources_cpu_test.yaml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,63 @@ tests:
188188
- failedTemplate:
189189
errorPattern: 'nodeGroup md0: resources.cpu \(0\) must be a whole number of vCPU cores between 1 and 4294967295'
190190

191+
# A leading zero is read as octal by YAML, so "010" gives the VM eight cores.
192+
# The kubelet reservations and the autoscaler capacity annotation follow
193+
# that reading rather than a decimal parse of the string, so the chart
194+
# agrees with itself: 8000m minus the two 400m reservations (5% of 8 cores,
195+
# inside the [50m, 500m] clamp). The template name is the one main renders
196+
# for this value, so the machine template does not move.
197+
- it: sizes reservations from the cores a leading-zero value gives the VM
198+
set:
199+
resources:
200+
cpu: "010"
201+
memory: 4Gi
202+
asserts:
203+
- equal:
204+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
205+
value: 8
206+
documentSelector:
207+
path: kind
208+
value: KubevirtMachineTemplate
209+
- equal:
210+
path: metadata.name
211+
value: kubernetes-myk8s-md0-ce2335
212+
documentSelector:
213+
path: kind
214+
value: KubevirtMachineTemplate
215+
- equal:
216+
path: metadata.annotations["capacity.cluster-autoscaler.kubernetes.io/cpu"]
217+
value: "7200m"
218+
documentSelector:
219+
path: kind
220+
value: MachineDeployment
221+
222+
- it: sizes reservations for a plain eight exactly as for the octal spelling
223+
set:
224+
resources:
225+
cpu: "8"
226+
memory: 4Gi
227+
documentSelector:
228+
path: kind
229+
value: MachineDeployment
230+
asserts:
231+
- equal:
232+
path: metadata.annotations["capacity.cluster-autoscaler.kubernetes.io/cpu"]
233+
value: "7200m"
234+
235+
- it: keeps the decimal reading for a value without a leading zero
236+
set:
237+
resources:
238+
cpu: "10"
239+
memory: 4Gi
240+
documentSelector:
241+
path: kind
242+
value: MachineDeployment
243+
asserts:
244+
- equal:
245+
path: metadata.annotations["capacity.cluster-autoscaler.kubernetes.io/cpu"]
246+
value: "9000m"
247+
191248
# The schema's quantity pattern admits a leading minus.
192249
- it: refuses a negative core count
193250
set:
@@ -294,3 +351,32 @@ tests:
294351
- equal:
295352
path: spec.template.spec.numCores
296353
value: 1
354+
355+
# No YAML round trip on this substrate: the quantity is parsed as decimal
356+
# for the machine template and for the reservations alike, so a leading
357+
# zero is ten cores on both sides and nothing here changes for it.
358+
- it: reads a leading zero as decimal on proxmox, on both sides
359+
set:
360+
substrate: proxmox
361+
proxmox:
362+
templateTags: [capmox-template, talos]
363+
full: true
364+
network:
365+
bridge: vmbr0
366+
dnsServers: ["8.8.8.8"]
367+
resources:
368+
cpu: "010"
369+
memory: 4Gi
370+
asserts:
371+
- equal:
372+
path: spec.template.spec.numCores
373+
value: 10
374+
documentSelector:
375+
path: kind
376+
value: ProxmoxMachineTemplate
377+
- equal:
378+
path: metadata.annotations["capacity.cluster-autoscaler.kubernetes.io/cpu"]
379+
value: "9000m"
380+
documentSelector:
381+
path: kind
382+
value: MachineDeployment

‎packages/apps/kubernetes-nodes/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@
6868
}
6969
},
7070
"resources": {
71-
"description": "Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.",
71+
"description": "Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the kubelet reservations and autoscaler capacity follow that reading.",
7272
"type": "object",
7373
"default": {},
7474
"properties": {

‎packages/apps/kubernetes-nodes/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ diskSize: 20Gi
6767
## @param {[]string} roles - List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/<r>`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.
6868
roles: []
6969

70-
## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time.
70+
## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the kubelet reservations and autoscaler capacity follow that reading.
7171
resources: {}
7272

7373
## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.

‎packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)