Skip to content

Commit 38b13f9

Browse files
fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two templates have to agree on it: talos-reconcile-job.yaml creates the object, nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the rendering was not enough — each file also assembled its own $group dict, one with ten keys and one with four, so the two names agreed only by coincidence of the keys the spec helper happens to read. Dropping "gpus" from the Job's dict diverged them (ffdbe1 vs 0bbd2a on a GPU pool) with the whole suite still green, which is the "templates do not exist" deadlock this fix exists to make impossible. Move the assembly into a single kubernetes-nodes.group helper, read back with fromYaml at both call sites the way the parent chart already reads kubernetes.nodeGroups, so there is one input as well as one rendering. Cover it too, since the seam can be reintroduced: the only literal pair in this chart rendered with instanceType "", gpus [] and kubelet {}, all falsy, so a dict missing those keys was indistinguishable from one carrying them. Add a GPU-shaped pair across the two suites, and the notMatchRegex guard against the old fixed name that the parent suite already has. Also correct a comment in the parent chart's Job that still described the computed locals as .group, removed when the machine config moved into _talosconfigtemplate.tpl. No rendered output changes: 209 + 17 helm unittests pass, render-parity.sh reports GOLDEN PARITY, and the golden hashes are unchanged. Assisted-By: Claude <[email protected]> Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 4206b9a commit 38b13f9

6 files changed

Lines changed: 104 additions & 20 deletions

File tree

‎packages/apps/kubernetes-nodes/templates/_helpers.tpl‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,3 +105,42 @@ it reconciles.
105105
{{- end -}}
106106
{{- end -}}
107107
{{- end -}}
108+
109+
{{/*
110+
This pool's node group, in the shape the parent kubernetes chart's
111+
`kubernetes.nodeGroups` helper emits — the SINGLE assembly of the dict that
112+
feeds every consumer in this chart:
113+
114+
* nodegroup.yaml, which renders the KubevirtMachineTemplate and the
115+
MachineDeployment (and hashes the former into its name);
116+
* talos-reconcile-job.yaml, which renders the TalosConfigTemplate spec and
117+
hashes it into the name the Job applies it under.
118+
119+
Emitted as YAML and read back with `fromYaml` at each call site, mirroring
120+
`include "kubernetes.nodeGroups" $ | fromYaml` in the parent chart. A Helm
121+
template cannot return a dict and a Helm variable cannot cross a file
122+
boundary, so this round-trip is what lets one assembly serve two files.
123+
124+
Why it has to be one assembly. The TalosConfigTemplate name is a hash of the
125+
spec this dict renders, and two templates have to agree on that name: the Job
126+
creates the object, the MachineDeployment's bootstrap.configRef points at it.
127+
Assembling the dict twice makes them agree only by coincidence of the keys the
128+
spec helper happens to read — drop a key on one side (`gpus`, say) and the two
129+
names diverge, CAPI blocks on "templates do not exist" forever and no worker is
130+
ever created. Sharing the *rendering* is not enough; the input has to be shared
131+
too. Keep the key set identical to the parent chart's group shape.
132+
*/}}
133+
{{- define "kubernetes-nodes.group" -}}
134+
{{- dict
135+
"minReplicas" .Values.minReplicas
136+
"maxReplicas" .Values.maxReplicas
137+
"instanceType" .Values.instanceType
138+
"diskSize" .Values.diskSize
139+
"storageClass" .Values.storageClass
140+
"roles" .Values.roles
141+
"resources" .Values.resources
142+
"gpus" .Values.gpus
143+
"kubelet" .Values.kubelet
144+
"logSerialConsole" .Values.logSerialConsole
145+
| toYaml -}}
146+
{{- end -}}

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -31,18 +31,11 @@
3131
{{- $groupName := include "kubernetes-nodes.groupName" . }}
3232
{{- include "kubernetes-nodes.assertNoForeignPool" . }}
3333
{{- $kubevirtmachinetemplateNames := list }}
34-
{{- $group := dict
35-
"minReplicas" .Values.minReplicas
36-
"maxReplicas" .Values.maxReplicas
37-
"instanceType" .Values.instanceType
38-
"diskSize" .Values.diskSize
39-
"storageClass" .Values.storageClass
40-
"roles" .Values.roles
41-
"resources" .Values.resources
42-
"gpus" .Values.gpus
43-
"kubelet" .Values.kubelet
44-
"logSerialConsole" .Values.logSerialConsole
45-
}}
34+
{{- /* Assembled once in _helpers.tpl and read back here and in
35+
talos-reconcile-job.yaml, so the TalosConfigTemplate name this file
36+
hashes into configRef and the one the Job hashes into TCT_NAME cannot
37+
be computed from different inputs. */}}
38+
{{- $group := include "kubernetes-nodes.group" . | fromYaml }}
4639
{{- define "kubevirtmachinetemplate" -}}
4740
spec:
4841
{{- /* Talos has no SSH; use 'none' so CAPK does not block Machine.Ready

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -270,17 +270,18 @@ spec:
270270
type: RuntimeDefault
271271
{{- end }}
272272
{{- $dnsDomain := include "kubernetes.tenantClusterDomain" $ }}
273-
{{- $group := dict
274-
"instanceType" .Values.instanceType
275-
"resources" .Values.resources
276-
"gpus" .Values.gpus
277-
"kubelet" .Values.kubelet
278-
}}
279273
{{- /* The TalosConfigTemplate spec is immutable, so it can only ever be
280274
created — never updated — and its name therefore carries a hash of the
281275
machine config it holds (see _talosconfigtemplate.tpl). nodegroup.yaml
282276
computes the same name for
283-
MachineDeployment.spec.template.spec.bootstrap.configRef. */}}
277+
MachineDeployment.spec.template.spec.bootstrap.configRef.
278+
279+
Both sides read the group from the same helper rather than each
280+
assembling its own dict: the spec helper reads a subset of the keys, so
281+
two assemblies would agree only by coincidence of that subset, and a key
282+
dropped on one side alone would point the MachineDeployment at a
283+
template this Job never creates. */}}
284+
{{- $group := include "kubernetes-nodes.group" $ | fromYaml }}
284285
{{- $tctArgs := dict "root" $ "group" $group }}
285286
{{- $tctSpec := include "kubernetes.talosConfigTemplateSpec" $tctArgs }}
286287
{{- $tctName := printf "%s-%s-%s" $clusterName $groupName (include "kubernetes.talosConfigTemplateHash" $tctArgs) }}

‎packages/apps/kubernetes-nodes/tests/nodegroup_test.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,23 @@ tests:
8080
path: spec.template.spec.bootstrap.configRef.kind
8181
value: TalosConfigTemplate
8282

83+
# Paired with the GPU-shaped test in talos_reconcile_job_test.yaml: a
84+
# machine-config change rotates the name, and both templates must rotate it
85+
# to the same value. See that file for why the fixture carries a truthy
86+
# `gpus` rather than reusing the all-falsy defaults above.
87+
- it: a machine-config change rotates the name the MachineDeployment references
88+
documentSelector:
89+
path: kind
90+
value: MachineDeployment
91+
set:
92+
gpus:
93+
- name: nvidia.com/GH200
94+
asserts:
95+
# Same literal as the paired test in talos_reconcile_job_test.yaml.
96+
- equal:
97+
path: spec.template.spec.bootstrap.configRef.name
98+
value: kubernetes-myk8s-md0-0bbd2a
99+
83100
- it: names the KubevirtMachineTemplate with the parent cluster prefix and a content hash
84101
documentSelector:
85102
path: kind

‎packages/apps/kubernetes-nodes/tests/talos_reconcile_job_test.yaml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,34 @@ tests:
4747
- matchRegex:
4848
path: spec.template.spec.containers[0].command[2]
4949
pattern: 'name: \$\{TCT_NAME\}'
50+
# ... and never the fixed <release>-<group> name, which is what the
51+
# immutability webhook rejected on every reconcile after the first.
52+
- notMatchRegex:
53+
path: spec.template.spec.containers[0].command[2]
54+
pattern: 'name: \$\{RELEASE\}-\$\{GROUP_NAME\}'
55+
56+
# A GPU node group adds `nodeLabels: {gpu: "on"}` to the machine config, so
57+
# the spec — and therefore the name — differs from the pair above. Two things
58+
# are pinned here. First, the propagation that was impossible before the fix:
59+
# new content lands on a NEW template instead of a rejected mutation, and CAPI
60+
# rolls the workers onto it. Second, and the reason the fixture is GPU-shaped
61+
# rather than another falsy default: the baseline above renders with
62+
# instanceType "", gpus [] and kubelet {}, all falsy, so it cannot tell apart
63+
# a group dict that carries those keys from one that omits them. The name is a
64+
# hash of the spec these values render, and the Job and the MachineDeployment
65+
# must compute it from the same dict — assembling it twice would let a dropped
66+
# key diverge the two names invisibly and leave CAPI blocked on "templates do
67+
# not exist" with no worker ever created. Truthy `gpus` makes such a
68+
# divergence fail here.
69+
- it: a machine-config change rotates the name the Job applies
70+
documentSelector:
71+
path: kind
72+
value: Job
73+
set:
74+
gpus:
75+
- name: nvidia.com/GH200
76+
asserts:
77+
# Same literal as the paired test in nodegroup_test.yaml.
78+
- equal:
79+
path: spec.template.spec.containers[0].env[?(@.name=="TCT_NAME")].value
80+
value: kubernetes-myk8s-md0-0bbd2a

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,10 @@ subjects:
214214
a content hash of the whole spec (see the header comment). The dot
215215
passed in is a deepCopy of the root context with the per-nodeGroup
216216
computed values set on it, so $.Release / $.Values resolve as usual
217-
and the loop/computed locals are read as .group, .groupName, etc. */}}
217+
and the loop/computed locals are read as .groupName, .dnsDomain,
218+
.tctName, .tctSpec and .kubectlImage. The node group itself is not
219+
among them: the machine config it feeds is rendered outside this
220+
template, into .tctSpec (see _talosconfigtemplate.tpl). */}}
218221
{{- define "kubernetes.talos-reconcile-job-spec" -}}
219222
spec:
220223
{{- /* Each pod attempt runs the ~10-minute (120 × 5s) input-wait loop, so this

0 commit comments

Comments
 (0)