Skip to content

Commit 2002266

Browse files
fix(kubernetes): name TalosConfigTemplate by content hash
TalosConfigTemplate.spec is immutable — vtalosconfigtemplate.cluster.x-k8s.io denies any mutation with "TalosConfigTemplate.Spec is immutable" — but the talos-reconcile Job applied it under a fixed <release>-<group> name on every reconcile. Once the template existed, no change to the rendered worker machine config could reach the cluster: the apply was rejected server-side while the HelmRelease stayed Ready. Observed with machine.network.searchDomains after a clusterDomain correction; the same held for certSANs, machine.registries and nameservers. Name the template after a 6-char hash of its rendered spec, the idiom KubevirtMachineTemplate already uses in the same chart, and point MachineDeployment.spec.template.spec.bootstrap.configRef at that name. A content change now creates a new template instead of attempting a rejected mutation, and CAPI rolls the workers onto it. The spec is rendered from one named template shared by the Job (which embeds it in its heredoc) and the MachineDeployment (which hashes it into configRef), so the two cannot compute different names. The runtime values the Job substitutes (CA certificates, Service ClusterIP, tokens) stay shell placeholders and are not hashed, so a certificate rotation does not roll workers. The talos-reconcile Role pins its TalosConfigTemplate get, patch, update and delete rights to the pool's own template, so it names the hashed template too. A server-side apply of a template that does not exist yet is still a PATCH on that name: a rule left on <release>-<group> would leave the Job Forbidden and the pool with no template to scale onto. The replace-on-immutable path is kept and pointed at the same name, because the runtime values sit outside the hash: a new apiserver address still renders a different spec under an unchanged name. Upgrade note: existing clusters roll their Talos workers once, because the configRef moves from <release>-<group> to the hashed name and CAPI creates a new MachineSet for it. The rendered machine config itself is unchanged by this commit. The pre-existing fixed-name template is left in place — the outgoing MachineSet still references it — and is garbage-collected with the KamajiControlPlane when the cluster is deleted. A pool sized by instanceType rather than by explicit resources has a second roll trigger. Its kubelet reservations are resolved through a live lookup of the VirtualMachineClusterInstancetype, so that object's size is part of the hashed spec even though the chart does not own it: a platform-side change to the instancetype rotates the name and rolls the pool again. That is the mechanism working as intended - the reservations really did change - but it is a roll the operator did not initiate, and the KubevirtMachineTemplate hash this mirrors does not have the property, since it embeds the instancetype by name and never its resolved size. Fixes #3515 Signed-off-by: Mattia Eleuteri <[email protected]> Assisted-by: LLM
1 parent b86fd7a commit 2002266

8 files changed

Lines changed: 456 additions & 286 deletions

File tree

‎packages/apps/kubernetes-nodes/Makefile‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,18 @@ generate:
1010
../../../hack/update-crd.sh
1111

1212
# Keep the worker-image inputs (Talos<->Kubernetes support matrix and the
13-
# kubectl.tag used by the talos-reconcile Job) in lockstep with the parent
14-
# kubernetes chart. They are copied, not symlinked, so `helm package` bundles
15-
# them; run this after the parent chart's `make update`.
13+
# kubectl.tag used by the talos-reconcile Job) and the worker machine-config
14+
# template in lockstep with the parent kubernetes chart. They are copied, not
15+
# symlinked, so `helm package` bundles them; run this after the parent chart's
16+
# `make update`.
17+
#
18+
# _talosconfigtemplate.tpl must stay byte-identical to the parent's copy: the
19+
# TalosConfigTemplate it renders is content-hash named, so any divergence would
20+
# point this pool's MachineDeployment at a different bootstrap template than the
21+
# parent rendered for the same group and roll every live worker. tests/render-
22+
# parity.sh compares the MachineDeployment across both charts and fails on it.
1623
update:
1724
cp ../kubernetes/files/versions.yaml files/versions.yaml
1825
cp ../kubernetes/images/kubectl.tag images/kubectl.tag
26+
cp ../kubernetes/templates/talos/_talosconfigtemplate.tpl templates/_talosconfigtemplate.tpl
1927
$(MAKE) generate

‎packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl‎

Lines changed: 296 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@
106106
"logSerialConsole" .Values.logSerialConsole
107107
"podCpuLimit" .Values.podCpuLimit
108108
"podCpuRequest" .Values.podCpuRequest
109+
"kernelModules" .Values.kernelModules
109110
}}
110111
{{- $_ := set $group "proxmox" .Values.proxmox }}
111112
{{- define "proxmoxmachinetemplate" -}}
@@ -863,7 +864,13 @@ spec:
863864
the TalosConfigTemplate with a KamajiControlPlane ownerReference for
864865
natural GC. The MachineDeployment below references the TCT name; CAPI's
865866
Machine controller blocks on its absence and unblocks the moment the Job
866-
applies it. */}}
867+
applies it.
868+
869+
The name it references carries a hash of the rendered machine config,
870+
computed here from the same named template the Job renders into its
871+
heredoc, because TalosConfigTemplate.spec is immutable and can only ever
872+
be created, never updated — see _talosconfigtemplate.tpl. */}}
873+
{{- $talosConfigTemplateName := printf "%s-%s-%s" $clusterName $groupName (include "kubernetes.talosConfigTemplateHash" (dict "root" $ "group" $group)) }}
867874
---
868875
{{- $context := deepCopy $ }}
869876
{{- $_ := set $context "group" $group }}
@@ -959,7 +966,7 @@ spec:
959966
configRef:
960967
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
961968
kind: TalosConfigTemplate
962-
name: {{ $clusterName }}-{{ $groupName }}
969+
name: {{ $talosConfigTemplateName }}
963970
namespace: {{ $.Release.Namespace }}
964971
clusterName: {{ $clusterName }}
965972
infrastructureRef:

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 57 additions & 260 deletions
Large diffs are not rendered by default.

‎packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ matches the golden render of a GPU pool:
109109
configRef:
110110
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
111111
kind: TalosConfigTemplate
112-
name: kubernetes-myk8s-md0
112+
name: kubernetes-myk8s-md0-940c60
113113
namespace: tenant-test
114114
clusterName: kubernetes-myk8s
115115
infrastructureRef:
@@ -262,7 +262,7 @@ matches the golden render of a kubelet-override pool:
262262
configRef:
263263
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
264264
kind: TalosConfigTemplate
265-
name: kubernetes-myk8s-md0
265+
name: kubernetes-myk8s-md0-db97de
266266
namespace: tenant-test
267267
clusterName: kubernetes-myk8s
268268
infrastructureRef:
@@ -418,7 +418,7 @@ matches the golden render of the pool objects:
418418
configRef:
419419
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
420420
kind: TalosConfigTemplate
421-
name: kubernetes-myk8s-md0
421+
name: kubernetes-myk8s-md0-ffdbe1
422422
namespace: tenant-test
423423
clusterName: kubernetes-myk8s
424424
infrastructureRef:

‎packages/apps/kubernetes-nodes/tests/nodegroup_test.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,14 +61,21 @@ tests:
6161
- notExists:
6262
path: spec.replicas
6363

64+
# The TalosConfigTemplate spec is immutable, so the object can only ever be
65+
# created, never updated: the name carries a content hash of the machine config
66+
# it holds (cozystack/cozystack#3515). The Job in talos-reconcile-job.yaml
67+
# applies it under the same name — see the TCT_NAME assertion below.
6468
- it: points the MachineDeployment bootstrap at the pool TalosConfigTemplate
6569
documentSelector:
6670
path: kind
6771
value: MachineDeployment
6872
asserts:
6973
- equal:
7074
path: spec.template.spec.bootstrap.configRef.name
71-
value: kubernetes-myk8s-md0
75+
value: kubernetes-myk8s-md0-ffdbe1
76+
- matchRegex:
77+
path: spec.template.spec.bootstrap.configRef.name
78+
pattern: ^kubernetes-myk8s-md0-[0-9a-f]{6}$
7279
- equal:
7380
path: spec.template.spec.bootstrap.configRef.kind
7481
value: TalosConfigTemplate
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
suite: talos-reconcile Job TalosConfigTemplate name
2+
3+
# TalosConfigTemplate.spec is immutable (webhook
4+
# vtalosconfigtemplate.cluster.x-k8s.io), so the object can only ever be created,
5+
# never updated: applied under a fixed name it freezes this pool's worker machine
6+
# config at first creation and every later change is rejected server-side
7+
# (cozystack/cozystack#3515). The name therefore carries a content hash of the
8+
# rendered spec.
9+
#
10+
# The literal below must stay identical to the MachineDeployment
11+
# bootstrap.configRef assertion in nodegroup_test.yaml — helm-unittest cannot
12+
# compare two templates in one assertion, and a divergence leaves CAPI blocked on
13+
# "templates do not exist" with no worker ever created.
14+
15+
templates:
16+
- templates/talos-reconcile-job.yaml
17+
release:
18+
name: kubernetes-nodes-myk8s-md0
19+
namespace: tenant-test
20+
set:
21+
cluster: myk8s
22+
_cluster:
23+
cluster-domain: cozy.local
24+
version: "v1.35"
25+
minReplicas: 0
26+
maxReplicas: 3
27+
instanceType: ""
28+
diskSize: 20Gi
29+
storageClass: replicated
30+
roles:
31+
- ingress-nginx
32+
resources:
33+
cpu: "2"
34+
memory: 4Gi
35+
gpus: []
36+
kubelet: {}
37+
38+
tests:
39+
- it: applies the TalosConfigTemplate under that same hashed name
40+
documentSelector:
41+
path: kind
42+
value: Job
43+
asserts:
44+
- equal:
45+
path: spec.template.spec.containers[0].env[?(@.name=="TCT_NAME")].value
46+
value: kubernetes-myk8s-md0-ffdbe1
47+
- matchRegex:
48+
path: spec.template.spec.containers[0].command[2]
49+
pattern: 'name: \$\{TCT_NAME\}'

‎packages/apps/kubernetes-nodes/tests/talos_tct_immutable_test.yaml‎

Lines changed: 23 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,13 @@ tests:
2626
- matchRegex:
2727
path: spec.template.spec.containers[0].command[2]
2828
pattern: '(?m)^\s*\*"is immutable"\*\)\s*$'
29-
# The target in full: a delete naming anything but this pool's own template
30-
# would take out a sibling pool's, and a prefix match would not see it.
29+
# The target in full: a delete naming anything but the template this Job
30+
# applies would take out a sibling pool's, or the pre-hash one the
31+
# outgoing MachineSet still references, and a prefix match would not see
32+
# it.
3133
- matchRegex:
3234
path: spec.template.spec.containers[0].command[2]
33-
pattern: 'kubectl delete talosconfigtemplate "\$\{RELEASE\}-\$\{GROUP_NAME\}" -n "\$NS"'
35+
pattern: 'kubectl delete talosconfigtemplate "\$\{TCT_NAME\}" -n "\$NS"'
3436

3537
# Any other apply error must stop the Job: a delete followed by a create that
3638
# fails for the same reason would leave the pool with no template at all.
@@ -51,13 +53,15 @@ tests:
5153
path: kind
5254
value: Role
5355
asserts:
54-
- contains:
55-
path: rules
56-
content:
57-
apiGroups: ["bootstrap.cluster.x-k8s.io"]
58-
resources: ["talosconfigtemplates"]
59-
resourceNames: ["kubernetes-myk8s-md0"]
60-
verbs: ["delete"]
56+
# The name is this pool's content-hash one. Which hash is not pinned here:
57+
# that it equals the name the Job applies and the MachineDeployment
58+
# references is hack/check-talosconfigtemplate-name-agreement.bats' job.
59+
- matchRegex:
60+
path: rules[?(@.resources[0] == "talosconfigtemplates" && @.verbs[0] == "delete")].resourceNames[0]
61+
pattern: '^kubernetes-myk8s-md0-[0-9a-f]{6}$'
62+
- lengthEqual:
63+
path: rules[?(@.resources[0] == "talosconfigtemplates" && @.verbs[0] == "delete")].resourceNames
64+
count: 1
6165
# delete must stay bound to this pool's own template: an unnamed rule
6266
# carrying it, alone or folded into the unnamed create, would let the Job
6367
# remove a sibling pool's template.
@@ -82,13 +86,15 @@ tests:
8286
path: kind
8387
value: Role
8488
asserts:
85-
- contains:
86-
path: rules
87-
content:
88-
apiGroups: ["bootstrap.cluster.x-k8s.io"]
89-
resources: ["talosconfigtemplates"]
90-
resourceNames: ["kubernetes-myk8s-md0"]
91-
verbs: ["get", "patch", "update"]
89+
# The name is this pool's content-hash one. Which hash is not pinned here:
90+
# that it equals the name the Job applies and the MachineDeployment
91+
# references is hack/check-talosconfigtemplate-name-agreement.bats' job.
92+
- matchRegex:
93+
path: rules[?(@.resources[0] == "talosconfigtemplates" && @.verbs[0] == "get")].resourceNames[0]
94+
pattern: '^kubernetes-myk8s-md0-[0-9a-f]{6}$'
95+
- lengthEqual:
96+
path: rules[?(@.resources[0] == "talosconfigtemplates" && @.verbs[0] == "get")].resourceNames
97+
count: 1
9298
- notContains:
9399
path: rules
94100
content:

0 commit comments

Comments
 (0)