Skip to content

Commit 11910e0

Browse files
authored
fix(kubernetes-nodes): refuse a resources.cpu that cannot be a core count (#4386)
## What this PR does A worker pool's CPU value could land in the VM as something KubeVirt cannot accept. `resources.cpu` is typed as a quantity, but on the kubevirt substrate the template wrote it unchanged into `domain.cpu.cores`, which is `uint32` in KubeVirt and an integer in the KubevirtMachineTemplate CRD. With `resources.cpu: 500m` the chart rendered `cores: 500m`. That string was truthy, so it also counted as explicit sizing and dropped the pool's `instanceType` from the VM, and then the API server refused the template anyway. The render now decodes the value as YAML, the same way `cores: <raw>` is decoded, and stops unless the result is a whole number from 1 to 4294967295. `500m`, `"1.5"` and `"1e10"` now fail at render time with an error naming the field, before the value reaches the VM template or the instancetype decision. `"0"` and `"-2"` were already refused on `main` and still are. Only inputs that could not produce a working pool are refused. Every value that rendered a pool before renders the same bytes now, so the content-hashed template keeps its name and nothing rolls. That includes `2`, `"2.0"`, `"+2"` and `"1e1"`, and also two odd cases I kept on purpose because pools may already carry them. An unquoted `0` is falsy, so the pool renders without `domain.cpu` and is sized by its instancetype, as on `main`. A leading-zero value such as `"010"` is still emitted as written and keeps the reading it gets on `main` today, where YAML treats it as octal. I checked both against a render of `main`, and the tests pin the template names that `main` produces for them. This is where kubernetes-nodes and vm-instance now differ. Since #4405, vm-instance refuses a leading zero and an unquoted `0`, and this chart does not. Stored worker pools can carry these values, because migration 56 copies `resources` verbatim, so refusing them here without a migration would fail those pools' releases on their next upgrade. #4554 tracks the migration and the refusal after it. The proxmox substrate is not checked. It converts the quantity to millicores and rounds up to whole cores, so `500m` is a valid size there and becomes one core. The field description now says what the value means on each substrate. Memory is left alone. It goes to `domain.memory.guest`, which is a `resource.Quantity`, so a quantity string is valid there. The issue suggested coercing to an integer, as vm-instance did before #4405. I didn't, because `int64` turns `500m` into `0` without an error. The new helm-unittest suite covers each input form, both preserved cases, the upper bound, the instancetype case and the proxmox path. Two neighbouring descriptions still say only "CPU available.": `nodeGroups[].resources.cpu` in `packages/extra/computeplane` and in the parent `kubernetes` chart. computeplane passes `resources` straight into this chart, so the new check applies to its pools as well. I left both alone to keep this change to one chart. Closes #4150 ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes-nodes): on the kubevirt substrate, a worker pool whose resources.cpu cannot be a whole vCPU core count (for example 500m, 1.5 or a value above 4294967295) now fails at render time with a clear error, instead of rendering a VM template that is refused later or dropping the pool's instanceType. Values that rendered a working pool before render unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * KubeVirt CPU settings now accept whole-core values from 1 to 4,294,967,295. Fractional, millicore, negative, and larger values are rejected; an unquoted `0` is treated as unset. * Proxmox rounds fractional and millicore CPU values up to whole cores. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents fb7e780 + 4583ee5 commit 11910e0

7 files changed

Lines changed: 323 additions & 5 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/README.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,28 @@
9292
{{- fail "podCpuLimit/podCpuRequest size the virt-launcher pod and are kubevirt-only" }}
9393
{{- end }}
9494
{{- end }}
95+
{{- /* On kubevirt, resources.cpu is emitted verbatim into domain.cpu.cores,
96+
an integer in the KubevirtMachineTemplate CRD and a uint32 in
97+
KubeVirt, and the same raw value decides below whether the
98+
instancetype matcher is dropped. A quantity such as 500m counts as
99+
sizing there and is then refused by the CRD, so a value that cannot
100+
be a core count stops here. Proxmox needs no such check: it converts
101+
the quantity to millicores and rounds up to whole cores.
102+
103+
The value is decoded as YAML, the way `cores: <raw>` is, so +2, 2.0
104+
or 1e1 pass as the integers they decode to, and a leading-zero
105+
spelling passes as whatever YAML reads it as. A bare numeric zero
106+
is left alone: it is falsy, so the pool renders without domain.cpu
107+
and is sized by its instancetype. */}}
108+
{{- $cpuValue := (.Values.resources | default dict).cpu }}
109+
{{- if and (not $isProxmox) (not (kindIs "invalid" $cpuValue)) (not (and (not (kindIs "string" $cpuValue)) (eq (float64 $cpuValue) 0.0))) }}
110+
{{- $cpu := printf "%v" $cpuValue }}
111+
{{- $cores := (printf "v: %s" $cpu | fromYaml).v }}
112+
{{- $isNumber := or (kindIs "int" $cores) (kindIs "int64" $cores) (kindIs "float64" $cores) }}
113+
{{- if not (and $isNumber (gt (float64 $cores) 0.0) (lt (float64 $cores) 4294967296.0) (eq (floor $cores) (float64 $cores))) }}
114+
{{- fail (printf "nodeGroup %s: resources.cpu (%s) must be a whole number of vCPU cores between 1 and 4294967295, such as 2. On the kubevirt substrate it becomes the worker VM's domain.cpu.cores, which KubeVirt declares as an unsigned 32-bit integer, so a millicore or fractional quantity cannot size the guest. Set a core count, or remove the field." $groupName $cpu) }}
115+
{{- end }}
116+
{{- end }}
95117
{{- /* Assembled once in _helpers.tpl and read back here and in
96118
talos-reconcile-job.yaml, so the TalosConfigTemplate name this file
97119
hashes into configRef and the one the Job hashes into TCT_NAME cannot
Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
suite: worker pool resources.cpu
2+
3+
# resources.cpu is typed as a quantity, but on the kubevirt substrate it lands
4+
# in the worker VM's domain.cpu.cores, which KubeVirt declares as uint32 and
5+
# the KubevirtMachineTemplate CRD as an integer. Only a whole number of cores
6+
# has a meaning there, and the same value decides whether the instancetype
7+
# matcher is dropped, so a value that cannot be one stops the render. Values
8+
# that already rendered a working pool keep rendering the same bytes: the
9+
# template name is a hash of its content, so any change would roll the pool.
10+
11+
templates:
12+
- templates/nodegroup.yaml
13+
release:
14+
name: kubernetes-nodes-myk8s-md0
15+
namespace: tenant-test
16+
set:
17+
cluster: myk8s
18+
_cluster:
19+
cluster-domain: cozy.local
20+
version: "v1.35"
21+
minReplicas: 0
22+
maxReplicas: 3
23+
instanceType: ""
24+
diskSize: 20Gi
25+
storageClass: replicated
26+
tests:
27+
- it: renders a quoted whole number as an integer core count
28+
set:
29+
resources:
30+
cpu: "2"
31+
memory: 4Gi
32+
documentSelector:
33+
path: kind
34+
value: KubevirtMachineTemplate
35+
asserts:
36+
- equal:
37+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
38+
value: 2
39+
40+
- it: renders an unquoted integer as the same core count
41+
set:
42+
resources:
43+
cpu: 4
44+
memory: 4Gi
45+
documentSelector:
46+
path: kind
47+
value: KubevirtMachineTemplate
48+
asserts:
49+
- equal:
50+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
51+
value: 4
52+
53+
- it: renders a whole float as an integer core count
54+
set:
55+
resources:
56+
cpu: 2.0
57+
memory: 4Gi
58+
documentSelector:
59+
path: kind
60+
value: KubevirtMachineTemplate
61+
asserts:
62+
- equal:
63+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
64+
value: 2
65+
66+
# The quantity grammar spells one whole number several ways, and each has
67+
# always rendered: YAML decodes the emitted value to the integer.
68+
- it: renders a quoted whole decimal as the same core count
69+
set:
70+
resources:
71+
cpu: "2.0"
72+
memory: 4Gi
73+
documentSelector:
74+
path: kind
75+
value: KubevirtMachineTemplate
76+
asserts:
77+
- equal:
78+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
79+
value: 2
80+
81+
- it: renders an explicit plus sign as the same core count
82+
set:
83+
resources:
84+
cpu: "+2"
85+
memory: 4Gi
86+
documentSelector:
87+
path: kind
88+
value: KubevirtMachineTemplate
89+
asserts:
90+
- equal:
91+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
92+
value: 2
93+
94+
- it: renders an exponent spelling as the core count it decodes to
95+
set:
96+
resources:
97+
cpu: "1e1"
98+
memory: 4Gi
99+
documentSelector:
100+
path: kind
101+
value: KubevirtMachineTemplate
102+
asserts:
103+
- equal:
104+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
105+
value: 10
106+
107+
# Both ends of the range the error message promises.
108+
- it: renders the smallest core count
109+
set:
110+
resources:
111+
cpu: "1"
112+
memory: 4Gi
113+
documentSelector:
114+
path: kind
115+
value: KubevirtMachineTemplate
116+
asserts:
117+
- equal:
118+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
119+
value: 1
120+
121+
- it: renders the largest core count uint32 can hold
122+
set:
123+
resources:
124+
cpu: "4294967295"
125+
memory: 4Gi
126+
documentSelector:
127+
path: kind
128+
value: KubevirtMachineTemplate
129+
asserts:
130+
- equal:
131+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
132+
value: 4294967295
133+
134+
- it: refuses a millicore quantity
135+
set:
136+
resources:
137+
cpu: 500m
138+
memory: 4Gi
139+
asserts:
140+
- failedTemplate:
141+
errorPattern: 'nodeGroup md0: resources.cpu \(500m\) must be a whole number of vCPU cores between 1 and 4294967295'
142+
143+
- it: refuses a millicore spelling of whole cores
144+
set:
145+
resources:
146+
cpu: 2000m
147+
memory: 4Gi
148+
asserts:
149+
- failedTemplate:
150+
errorPattern: 'nodeGroup md0: resources.cpu \(2000m\) must be a whole number of vCPU cores between 1 and 4294967295'
151+
152+
- it: refuses a fractional core count
153+
set:
154+
resources:
155+
cpu: "1.5"
156+
memory: 4Gi
157+
asserts:
158+
- failedTemplate:
159+
errorPattern: 'nodeGroup md0: resources.cpu \(1.5\) must be a whole number of vCPU cores between 1 and 4294967295'
160+
161+
# An unquoted zero is falsy, so it has always rendered as if the field were
162+
# unset: no domain.cpu, sizing left to the instancetype. The template name
163+
# below is what the chart rendered for it before this guard existed.
164+
- it: keeps rendering an unquoted zero as an unset field
165+
set:
166+
resources:
167+
cpu: 0
168+
memory: 4Gi
169+
documentSelector:
170+
path: kind
171+
value: KubevirtMachineTemplate
172+
asserts:
173+
- notExists:
174+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu
175+
- equal:
176+
path: metadata.name
177+
value: kubernetes-myk8s-md0-28d812
178+
179+
# A quoted zero is truthy, so it never took the unset path. It was refused
180+
# before this guard existed as well, by the kubelet CPU reservation check,
181+
# which cannot fit any reservation into an effective CPU of zero cores.
182+
- it: refuses a quoted zero
183+
set:
184+
resources:
185+
cpu: "0"
186+
memory: 4Gi
187+
asserts:
188+
- failedTemplate:
189+
errorPattern: 'nodeGroup md0: resources.cpu \(0\) must be a whole number of vCPU cores between 1 and 4294967295'
190+
191+
# The schema's quantity pattern admits a leading minus.
192+
- it: refuses a negative core count
193+
set:
194+
resources:
195+
cpu: "-2"
196+
memory: 4Gi
197+
asserts:
198+
- failedTemplate:
199+
errorPattern: 'nodeGroup md0: resources.cpu \(-2\) must be a whole number of vCPU cores between 1 and 4294967295'
200+
201+
# A leading zero is emitted verbatim, as it always was, and YAML reads 010
202+
# as octal. The template name is what the chart rendered for it before this
203+
# guard existed, so a pool sized this way does not roll.
204+
- it: keeps rendering a leading zero byte-identically
205+
set:
206+
resources:
207+
cpu: "010"
208+
memory: 4Gi
209+
documentSelector:
210+
path: kind
211+
value: KubevirtMachineTemplate
212+
asserts:
213+
- equal:
214+
path: metadata.name
215+
value: kubernetes-myk8s-md0-ce2335
216+
217+
# KubeVirt declares cores as uint32, so a count past its ceiling cannot size
218+
# the guest even though it is a whole number.
219+
- it: refuses a core count past the uint32 ceiling
220+
set:
221+
resources:
222+
cpu: "4294967296"
223+
memory: 4Gi
224+
asserts:
225+
- failedTemplate:
226+
errorPattern: 'nodeGroup md0: resources.cpu \(4294967296\) must be a whole number of vCPU cores between 1 and 4294967295'
227+
228+
- it: refuses an exponent spelling past the uint32 ceiling
229+
set:
230+
resources:
231+
cpu: "1e10"
232+
memory: 4Gi
233+
asserts:
234+
- failedTemplate:
235+
errorPattern: 'nodeGroup md0: resources.cpu \(1e10\) must be a whole number of vCPU cores between 1 and 4294967295'
236+
237+
- it: leaves domain.cpu out when resources.cpu is unset
238+
set:
239+
resources:
240+
memory: 4Gi
241+
documentSelector:
242+
path: kind
243+
value: KubevirtMachineTemplate
244+
asserts:
245+
- notExists:
246+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu
247+
248+
- it: lets a whole core count replace the instancetype matcher
249+
set:
250+
instanceType: u1.medium
251+
resources:
252+
cpu: "2"
253+
memory: 4Gi
254+
documentSelector:
255+
path: kind
256+
value: KubevirtMachineTemplate
257+
asserts:
258+
- notExists:
259+
path: spec.template.spec.virtualMachineTemplate.spec.instancetype
260+
- equal:
261+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
262+
value: 2
263+
264+
# The case the precedence rule got wrong: the raw string was truthy, so the
265+
# matcher was dropped for a value that cannot size the VM.
266+
- it: refuses a millicore quantity beside an instanceType instead of dropping the matcher
267+
set:
268+
instanceType: u1.medium
269+
resources:
270+
cpu: 500m
271+
memory: 4Gi
272+
asserts:
273+
- failedTemplate:
274+
errorPattern: 'nodeGroup md0: resources.cpu \(500m\) must be a whole number of vCPU cores between 1 and 4294967295'
275+
276+
# Proxmox converts the quantity to millicores and rounds up, so a millicore
277+
# value is a valid size there and the kubevirt-only guard must not see it.
278+
- it: leaves a millicore quantity to the proxmox substrate
279+
set:
280+
substrate: proxmox
281+
proxmox:
282+
templateTags: [capmox-template, talos]
283+
full: true
284+
network:
285+
bridge: vmbr0
286+
dnsServers: ["8.8.8.8"]
287+
resources:
288+
cpu: 500m
289+
memory: 4Gi
290+
documentSelector:
291+
path: kind
292+
value: ProxmoxMachineTemplate
293+
asserts:
294+
- equal:
295+
path: spec.template.spec.numCores
296+
value: 1

‎packages/apps/kubernetes-nodes/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@
7373
"default": {},
7474
"properties": {
7575
"cpu": {
76-
"description": "CPU available.",
76+
"description": "Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.",
7777
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
7878
"anyOf": [
7979
{

‎packages/apps/kubernetes-nodes/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ storageClass: replicated
3232
##
3333

3434
## @typedef {struct} Resources - Explicit CPU and memory configuration for a node.
35-
## @field {quantity} [cpu] - CPU available.
35+
## @field {quantity} [cpu] - Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.
3636
## @field {quantity} [memory] - Memory (RAM) available.
3737

3838
## @typedef {struct} GPU - GPU configuration.

‎packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)