Skip to content

Commit 51aedc9

Browse files
committed
fix(kubernetes-nodes): let the reconcile Job replace an immutable TalosConfigTemplate
The CRD marks spec immutable, so kubectl apply fails the moment an input the render depends on genuinely changes. In practice that input is the apiserver address, which moves when a cluster is rebuilt or its Service type changes — and the Job then died on the error, leaving an operator to delete the template by hand before workers could be created again. It now replaces the object on that specific failure and re-raises every other one. Matching the message narrowly is deliberate: deleting on an unrecognised error would turn an unrelated fault, an expired token or a webhook outage, into worker churn. Existing Machines are untouched either way, their bootstrap data having been rendered when they joined; only Machines created afterwards read the new template, which is the point of rewriting it. The delete permission is a separate rule naming this pool's own template, so the Job cannot remove anyone else's. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent 6541295 commit 51aedc9

3 files changed

Lines changed: 93 additions & 4 deletions

File tree

‎hack/talos-reconcile-heredoc_test.bats‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# therefore render packages/apps/kubernetes-nodes.
1212
#
1313
# The reconcile Job applies the TalosConfigTemplate via an UNQUOTED
14-
# `cat <<EOF | kubectl apply -f -` heredoc, so every line of its body is subject
14+
# `cat <<EOF` heredoc (redirected to a file, then applied), so every line of its body is subject
1515
# to shell parameter expansion and command substitution at Job runtime. The
1616
# `talos.registryMirrors` knob and the Talos image coordinates render free-form
1717
# tenant-facing input into that heredoc. A helm-unittest string `matchRegex`
@@ -51,7 +51,7 @@ VALS
5151
> "$work/cmd.sh"
5252
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
5353
awk '
54-
/^cat <<EOF \| kubectl apply/ { print "cat <<EOF"; inblock=1; next }
54+
/^cat <<EOF([[:space:]]*\||[[:space:]]*>)/ { print "cat <<EOF"; inblock=1; next }
5555
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
5656
inblock { print }
5757
' "$work/cmd.sh" > "$work/heredoc.sh"
@@ -86,7 +86,7 @@ VALS
8686
> "$work/cmd.sh"
8787
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
8888
awk '
89-
/^cat <<EOF \| kubectl apply/ { print "cat <<EOF"; inblock=1; next }
89+
/^cat <<EOF([[:space:]]*\||[[:space:]]*>)/ { print "cat <<EOF"; inblock=1; next }
9090
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
9191
inblock { print }
9292
' "$work/cmd.sh" > "$work/heredoc.sh"

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,14 @@ rules:
7171
- apiGroups: ["bootstrap.cluster.x-k8s.io"]
7272
resources: ["talosconfigtemplates"]
7373
verbs: ["get", "create", "patch", "update"]
74+
{{- /* delete is separate and named: the CRD marks spec immutable, so a
75+
genuinely changed render can only be applied by replacing the object.
76+
Restricting it to this pool's own template keeps the Job unable to
77+
remove anyone else's. */}}
78+
- apiGroups: ["bootstrap.cluster.x-k8s.io"]
79+
resources: ["talosconfigtemplates"]
80+
resourceNames: ["{{ $clusterName }}-{{ $groupName }}"]
81+
verbs: ["delete"]
7482
---
7583
{{- /* The tenant-root chart installs a default-deny egress policy on every
7684
pod in the tenant namespace; without an explicit allow, this Job
@@ -241,7 +249,7 @@ spec:
241249
# Apply the TalosConfigTemplate. ownerReference back to the parent
242250
# KamajiControlPlane makes the standard Kubernetes garbage collector
243251
# clean it up when the parent cluster is deleted.
244-
cat <<EOF | kubectl apply -f -
252+
cat <<EOF > /tmp/tct.yaml
245253
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
246254
kind: TalosConfigTemplate
247255
metadata:
@@ -385,6 +393,31 @@ spec:
385393
service:
386394
disabled: true
387395
EOF
396+
397+
# The CRD marks spec immutable, so apply fails the moment any input
398+
# this render depends on has genuinely changed -- the apiserver
399+
# address is the one that moves in practice. The object then has to
400+
# be replaced rather than patched. Existing Machines are unaffected:
401+
# their bootstrap data was rendered when they joined. Only Machines
402+
# created afterwards read the new template, which is the intent.
403+
#
404+
# Every other apply failure still stops the Job. Matching on the
405+
# message is narrow on purpose: silently deleting and recreating on
406+
# an unrecognised error would turn an unrelated fault into worker
407+
# churn.
408+
if ! APPLY_OUT="$(kubectl apply -f /tmp/tct.yaml 2>&1)"; then
409+
case "$APPLY_OUT" in
410+
*"is immutable"*|*"Spec is immutable"*)
411+
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} spec changed; replacing the immutable object"
412+
kubectl delete talosconfigtemplate "${RELEASE}-${GROUP_NAME}" -n "$NS"
413+
kubectl apply -f /tmp/tct.yaml
414+
;;
415+
*)
416+
echo "$APPLY_OUT" >&2
417+
exit 1
418+
;;
419+
esac
420+
fi
388421
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} applied"
389422
resources:
390423
requests:
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
suite: TalosConfigTemplate replacement on an immutable spec
2+
# The CRD marks spec immutable, so the Job's kubectl apply fails the moment any
3+
# input the render depends on genuinely changes — the apiserver address being
4+
# the one that moves in practice. Before this, that failure stopped the Job and
5+
# an operator had to delete the template by hand. What is pinned here is that
6+
# the Job replaces it, that it only does so for that specific failure, and that
7+
# it may delete nothing else.
8+
templates:
9+
- templates/talos-reconcile-job.yaml
10+
release:
11+
name: kubernetes-nodes-myk8s-md0
12+
namespace: tenant-test
13+
set:
14+
cluster: myk8s
15+
_cluster:
16+
cluster-domain: cozy.local
17+
version: "v1.35"
18+
tests:
19+
- it: replaces the template when the apiserver reports it immutable
20+
documentSelector:
21+
path: kind
22+
value: Job
23+
asserts:
24+
- matchRegex:
25+
path: spec.template.spec.containers[0].command[2]
26+
pattern: 'is immutable'
27+
- matchRegex:
28+
path: spec.template.spec.containers[0].command[2]
29+
pattern: 'kubectl delete talosconfigtemplate'
30+
31+
# Deleting on an unrecognised error would turn an unrelated fault — an expired
32+
# token, a webhook outage — into worker churn, so the branch must re-raise.
33+
- it: still fails the Job on any other apply error
34+
documentSelector:
35+
path: kind
36+
value: Job
37+
asserts:
38+
- matchRegex:
39+
path: spec.template.spec.containers[0].command[2]
40+
pattern: 'echo "\$APPLY_OUT" >&2'
41+
- matchRegex:
42+
path: spec.template.spec.containers[0].command[2]
43+
pattern: 'exit 1'
44+
45+
- it: may delete only this pool's own template
46+
documentSelector:
47+
path: kind
48+
value: Role
49+
asserts:
50+
- contains:
51+
path: rules
52+
content:
53+
apiGroups: ["bootstrap.cluster.x-k8s.io"]
54+
resources: ["talosconfigtemplates"]
55+
resourceNames: ["kubernetes-myk8s-md0"]
56+
verbs: ["delete"]

0 commit comments

Comments
 (0)