Skip to content

Commit 6b4c7eb

Browse files
committed
fix(kubernetes): make node group resources optional when instanceType is set
A node group is sized either by instanceType or by explicit resources, but the generated values schema required resources on every node group. A node group with instanceType and no resources, or an empty resources that is dropped during typed-CR to HelmRelease translation, failed schema validation with "missing property 'resources'" even though sizing was fully specified. Mark resources optional in the node group definition and regenerate the schema, API types, README, and ApplicationDefinition. The chart template already emits domain.cpu/memory only when resources is present, so a node group sized purely by instanceType now passes validation and installs. instanceType and explicit resources are mutually exclusive: KubeVirt rejects a VirtualMachine that references an instancetype and also sets domain.cpu or domain.memory. Guard this combination at render time with a clear error instead of emitting a VM the instancetype admission webhook would reject. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent f5c408d commit 6b4c7eb

7 files changed

Lines changed: 103 additions & 8 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ See the reference for components utilized in this service:
117117
| `nodeGroups[name].diskSize` | System disk size for the worker VM. Carries the Talos OS image (factory.talos.dev raw artifact streamed in by CDI), kubelet state, containerd image cache, and any local-path PVCs. Pre-Talos installs used a separate disk-kubelet PVC for kubelet/containerd state; on Talos this is consolidated onto the single system disk imaged from the factory artifact. | `quantity` | `20Gi` |
118118
| `nodeGroups[name].storageClass` | StorageClass for worker node persistent disks. When empty, uses the management cluster default StorageClass (the one annotated storageclass.kubernetes.io/is-default-class: true). NOTE: deliberately not marked immutable — the field is optional and undefaulted, so a strict `self == oldSelf` rule would block any future attempt to set it on an existing node group. | `string` | `""` |
119119
| `nodeGroups[name].roles` | List of node roles. | `[]string` | `[]` |
120-
| `nodeGroups[name].resources` | CPU and memory resources for each worker node. | `object` | `{}` |
120+
| `nodeGroups[name].resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. Use one or the other — combining a non-empty `instanceType` with explicit `resources` is rejected at render time, because instanceType CPU/memory cannot be overridden. | `object` | `{}` |
121121
| `nodeGroups[name].resources.cpu` | CPU available. | `quantity` | `""` |
122122
| `nodeGroups[name].resources.memory` | Memory (RAM) available. | `quantity` | `""` |
123123
| `nodeGroups[name].gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -341,6 +341,14 @@ metadata:
341341
name: {{ .Release.Name }}
342342
namespace: {{ .Release.Namespace }}
343343
{{- range $groupName, $group := .Values.nodeGroups }}
344+
{{- /* instanceType and explicit resources are alternative, mutually exclusive
345+
sizing sources. KubeVirt rejects a VirtualMachine that references an
346+
instancetype while also setting domain.cpu/memory, because instancetype
347+
CPU/memory cannot be overridden. Fail fast here with a clear render-time
348+
error instead of letting the conflict surface as a late admission rejection. */}}
349+
{{- if and $group.instanceType $group.resources (or $group.resources.cpu $group.resources.memory) }}
350+
{{- fail (printf "nodeGroup %s: set either instanceType or explicit resources (cpu/memory), not both — KubeVirt rejects a VM that overrides instanceType CPU/memory" $groupName) }}
351+
{{- end }}
344352
{{/* Resolve instanceType once per group — reused for kubelet reservations and capacity annotations. */}}
345353
{{- $instanceType := dict }}
346354
{{- if $group.instanceType }}

‎packages/apps/kubernetes/tests/cluster_test.yaml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,3 +292,91 @@ tests:
292292
asserts:
293293
- failedTemplate:
294294
errorMessage: 'nodeGroup "md0": ephemeralStorage is no longer supported and should have been automatically migrated to diskSize by platform migration 41. If you see this error after upgrading, the migration did not run — check the cozystack-migration-hook Job logs in cozy-system.'
295+
296+
###############################################
297+
# resources is optional when instanceType set #
298+
###############################################
299+
300+
# A node group may be sized either by instanceType or by explicit resources.
301+
# When instanceType is the sizing source, resources may be omitted entirely.
302+
# helm-unittest enforces values.schema.json, so the render below only succeeds
303+
# if the schema does NOT require resources on each node group.
304+
#
305+
# instanceType is set to "" here only to bypass the live-cluster lookup() that
306+
# validates a real VirtualMachineClusterInstancetype; the schema acceptance of
307+
# an omitted resources key is independent of the instanceType value.
308+
#
309+
# The "worker" group sorts after "md0", so its KubevirtMachineTemplate is
310+
# document index 8 (md0 → 4, worker → 8), matching the multi-group layout above.
311+
312+
- it: accepts a node group that omits resources and renders no explicit cpu/memory
313+
release:
314+
name: test-k8s
315+
namespace: tenant-test
316+
set:
317+
nodeGroups:
318+
worker:
319+
minReplicas: 1
320+
maxReplicas: 3
321+
instanceType: ""
322+
diskSize: "20Gi"
323+
roles:
324+
- worker
325+
asserts:
326+
- isKind:
327+
of: KubevirtMachineTemplate
328+
documentIndex: 8
329+
- equal:
330+
path: spec.template.spec.virtualMachineTemplate.spec.template.metadata.labels["cluster.x-k8s.io/deployment-name"]
331+
value: test-k8s-worker
332+
documentIndex: 8
333+
- notExists:
334+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu
335+
documentIndex: 8
336+
- notExists:
337+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.memory
338+
documentIndex: 8
339+
340+
# Complementary case: when explicit resources ARE provided (md0 in common.yaml
341+
# has cpu: 2 / memory: 4Gi), they take effect on the VM domain. Confirms the fix
342+
# only relaxes the schema requirement and does not drop honoured resources.
343+
- it: still renders explicit cpu and memory when resources are provided
344+
release:
345+
name: test-k8s
346+
namespace: tenant-test
347+
asserts:
348+
- isKind:
349+
of: KubevirtMachineTemplate
350+
documentIndex: 4
351+
- equal:
352+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.cpu.cores
353+
value: 2
354+
documentIndex: 4
355+
- equal:
356+
path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.memory.guest
357+
value: 4Gi
358+
documentIndex: 4
359+
360+
# instanceType and explicit resources are mutually exclusive sizing sources:
361+
# KubeVirt rejects a VM that references an instancetype and also sets explicit
362+
# CPU/memory. The chart fails fast at render time rather than emitting a VM that
363+
# is later rejected by the instancetype admission webhook.
364+
- it: fails when both instanceType and explicit resources are set
365+
release:
366+
name: test-k8s
367+
namespace: tenant-test
368+
set:
369+
nodeGroups:
370+
md0:
371+
minReplicas: 0
372+
maxReplicas: 10
373+
instanceType: "u1.medium"
374+
diskSize: 20Gi
375+
roles:
376+
- worker
377+
resources:
378+
cpu: 2
379+
memory: 4Gi
380+
asserts:
381+
- failedTemplate:
382+
errorMessage: "nodeGroup md0: set either instanceType or explicit resources (cpu/memory), not both — KubeVirt rejects a VM that overrides instanceType CPU/memory"

‎packages/apps/kubernetes/values.schema.json‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,7 @@
4040
"diskSize",
4141
"instanceType",
4242
"maxReplicas",
43-
"minReplicas",
44-
"resources"
43+
"minReplicas"
4544
],
4645
"properties": {
4746
"diskSize": {
@@ -128,7 +127,7 @@
128127
"default": 0
129128
},
130129
"resources": {
131-
"description": "CPU and memory resources for each worker node.",
130+
"description": "Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. Use one or the other — combining a non-empty `instanceType` with explicit `resources` is rejected at render time, because instanceType CPU/memory cannot be overridden.",
132131
"type": "object",
133132
"properties": {
134133
"cpu": {

‎packages/apps/kubernetes/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ storageClass: replicated
8585
## @field {string} [storageClass] - StorageClass for worker node persistent disks. When empty, uses the management cluster default StorageClass (the one annotated storageclass.kubernetes.io/is-default-class: true). NOTE: deliberately not marked immutable — the field is optional and undefaulted, so a strict `self == oldSelf` rule would block any future attempt to set it on an existing node group.
8686
## @x-cozystack-options {source: storageclass}
8787
## @field {[]string} roles - List of node roles.
88-
## @field {Resources} resources - CPU and memory resources for each worker node.
88+
## @field {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. Use one or the other — combining a non-empty `instanceType` with explicit `resources` is rejected at render time, because instanceType CPU/memory cannot be overridden.
8989
## @field {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
9090
## @field {Kubelet} [kubelet] - Kubelet resource reservations for this node group.
9191

‎packages/system/kubernetes-rd/cozyrds/kubernetes.yaml‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)