Skip to content

Commit 9afe7f1

Browse files
fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two templates have to agree on it: talos-reconcile-job.yaml creates the object, nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the rendering was not enough — each file also assembled its own $group dict, one carrying every pool value and one only the few the spec reads, so the two names agreed only by coincidence of the keys the spec helper happens to read. Dropping "gpus" from the Job's dict diverged them on a GPU pool with the whole suite still green, which is the "templates do not exist" deadlock this fix exists to make impossible. Move the assembly into a single kubernetes-nodes.group helper, read back with fromYaml at both call sites, so there is one input as well as one rendering. Cover it too, since the seam can be reintroduced. The chart's helm-unittest suites pin each name against its own literal in its own file, which catches a divergence but reports it as one literal having moved — indistinguishable from a stale fixture, so re-pasting the new value is the obvious and wrong repair. hack/check-talosconfigtemplate-name-agreement.bats renders the chart once and compares the two values against each other, so the failure states the invariant instead of a number. It asserts on a GPU-shaped pool as well as the default one: the default renders gpus [], kubelet {} and instanceType "" as falsy, so a consumer reading a subset of the group is indistinguishable there from one reading all of it. kernelModules gets a case of its own, because it has no default and stays unset on both of those shapes. A last case pins that the shapes hash differently, so the others cannot hold vacuously. The same guard compares the talos-reconcile Role's resourceNames on TalosConfigTemplates against that name. The Job applies server-side, and that is a PATCH on the name even when it creates the object, so a Role naming anything else leaves the Job Forbidden and the pool without a template, with the same silent outcome. Document that talos.registryMirrors and kernelModules sit inside the hashed spec, so changing either rolls the pool's existing workers instead of only affecting nodes created later. The kernelModules text described the fixed-name template as the present state and the content-hash name as a future one. No rendered output changes: the golden render snapshots are unchanged. Signed-off-by: Mattia Eleuteri <[email protected]> Assisted-by: LLM
1 parent 2002266 commit 9afe7f1

12 files changed

Lines changed: 324 additions & 79 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# The TalosConfigTemplate is named after a content hash of the worker machine
4+
# config, and three places in kubernetes-nodes have to arrive at that name:
5+
#
6+
# * talos-reconcile-job.yaml creates the object, passing the name to the Job
7+
# as TCT_NAME;
8+
# * the same file's Role pins the Job's get/patch/update and delete rights on
9+
# TalosConfigTemplates to it by resourceNames;
10+
# * nodegroup.yaml points MachineDeployment.spec.template.spec.bootstrap
11+
# .configRef.name at it.
12+
#
13+
# If the Job and the MachineDeployment disagree, the MachineDeployment
14+
# references a TalosConfigTemplate that is never created. If the Role disagrees,
15+
# the Job's server-side apply -- a PATCH on the name, even for a create -- is
16+
# Forbidden and the template is never created either. CAPI does not fail in
17+
# either case: it blocks on "templates do not exist" and no worker is ever
18+
# built, so the divergence surfaces as a cluster that quietly never scales
19+
# rather than as a render error.
20+
#
21+
# The chart's helm-unittest suites pin both names, but each pins its own
22+
# literal in its own file. That catches a divergence, yet the output only says
23+
# one literal moved -- which reads as a stale fixture, and re-pasting the new
24+
# value is then the obvious and wrong fix. This guard renders the chart ONCE and
25+
# compares the two values against each other, so the failure states the
26+
# invariant instead of a number.
27+
#
28+
# Rendered with explicit resources rather than an instanceType: an
29+
# instanceType-sized pool resolves its kubelet reservations through a live
30+
# `lookup` of the VirtualMachineClusterInstancetype, which is empty outside a
31+
# cluster. The invariant under test does not depend on which sizing source is
32+
# used -- both consumers read the same assembled group either way.
33+
#
34+
# Compatible with both `bats` directly and the in-repo cozytest.sh runner, which
35+
# runs each @test in a fresh subshell under `set -u` and honors no setup() or
36+
# teardown(), so TMP is provisioned inline per test and cleaned without an EXIT
37+
# trap (docs/agents/e2e-testing.md bans those here).
38+
39+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
40+
CHART="$REPO_ROOT/packages/apps/kubernetes-nodes"
41+
42+
render_pool() {
43+
# $1: output file, remaining args: extra --set flags for the pool shape
44+
out="$1"; shift
45+
helm template kubernetes-nodes-myk8s-md0 "$CHART" \
46+
--namespace tenant-test \
47+
--set cluster=myk8s \
48+
--set _cluster.cluster-domain=cozy.local \
49+
--set version=v1.35 \
50+
--set minReplicas=0 --set maxReplicas=3 \
51+
--set instanceType="" \
52+
--set diskSize=20Gi --set storageClass=replicated \
53+
--set 'roles[0]=ingress-nginx' \
54+
--set resources.cpu=2 --set resources.memory=4Gi \
55+
"$@" > "$out" 2>"$out.err"
56+
[ -s "$out" ] || { cat "$out.err" >&2; return 1; }
57+
}
58+
59+
job_tct_name() {
60+
awk '/name: TCT_NAME/ { getline; sub(/^[[:space:]]*value:[[:space:]]*/, ""); gsub(/"/, ""); print; exit }' "$1"
61+
}
62+
63+
role_tct_names() {
64+
# One line per Role rule on talosconfigtemplates that carries resourceNames.
65+
awk '/resources: \["talosconfigtemplates"\]/ { getline; if ($0 ~ /resourceNames:/) { sub(/.*resourceNames:[[:space:]]*\[/, ""); sub(/\].*/, ""); gsub(/"/, ""); print } }' "$1"
66+
}
67+
68+
md_config_ref_name() {
69+
awk '/configRef:/ { f=1; next } f && /name:/ { sub(/^[[:space:]]*name:[[:space:]]*/, ""); print; exit }' "$1"
70+
}
71+
72+
assert_agreement() {
73+
tmp="$1"; shape="$2"
74+
job=$(job_tct_name "$tmp/r.yaml")
75+
md=$(md_config_ref_name "$tmp/r.yaml")
76+
77+
# An empty capture means the render stopped emitting one of the two fields,
78+
# which would make a naive string comparison pass on "" == "".
79+
if [ -z "$job" ] || [ -z "$md" ]; then
80+
echo "FAIL: could not read both names out of the $shape render."
81+
echo " Job TCT_NAME: '${job:-<missing>}'"
82+
echo " MachineDeployment ref: '${md:-<missing>}'"
83+
echo "Either the field moved or the render is incomplete; this guard cannot"
84+
echo "compare what it cannot find, so it fails rather than pass vacuously."
85+
return 1
86+
fi
87+
88+
if [ "$job" != "$md" ]; then
89+
echo "FAIL: the two TalosConfigTemplate names must be equal, and are not ($shape pool)."
90+
echo " talos-reconcile-job.yaml creates: $job"
91+
echo " nodegroup.yaml references: $md"
92+
echo "The MachineDeployment now points at a TalosConfigTemplate nothing creates."
93+
echo "CAPI blocks on \"templates do not exist\" and builds no worker, without"
94+
echo "failing the release. Both names come from a hash of the group assembled"
95+
echo "by kubernetes-nodes.group in templates/_helpers.tpl: they diverge when a"
96+
echo "call site stops feeding that helper, or when a key it carries is dropped."
97+
echo "Fix the input they share; do not re-pin either literal."
98+
return 1
99+
fi
100+
101+
# Two named rules: get/patch/update, and delete. Fewer means one went unnamed
102+
# or disappeared, and the per-rule comparison below would pass on nothing.
103+
roles=$(role_tct_names "$tmp/r.yaml")
104+
if [ "$(printf '%s\n' "$roles" | grep -c .)" -ne 2 ]; then
105+
echo "FAIL: expected two named TalosConfigTemplate rules in the talos-reconcile Role ($shape pool), found:"
106+
printf ' %s\n' "${roles:-<none>}"
107+
return 1
108+
fi
109+
for r in $roles; do
110+
if [ "$r" != "$md" ]; then
111+
echo "FAIL: the talos-reconcile Role names a TalosConfigTemplate the Job does not apply ($shape pool)."
112+
echo " Role resourceNames: $r"
113+
echo " nodegroup.yaml references: $md"
114+
echo "The Job's server-side apply is a PATCH on the template's name even when"
115+
echo "it creates it, so it is Forbidden, and the pool never gets a template"
116+
echo "to scale onto. Name the rule after the same content-hash name."
117+
return 1
118+
fi
119+
done
120+
}
121+
122+
@test "Job and MachineDeployment agree on the TalosConfigTemplate name" {
123+
TMP=$(mktemp -d)
124+
render_pool "$TMP/r.yaml"
125+
assert_agreement "$TMP" "default"
126+
rm -rf "$TMP"
127+
}
128+
129+
@test "they still agree once the hashed spec changes (GPU pool)" {
130+
# A GPU pool renders extra nodeLabels into the machine config, so the hash --
131+
# and therefore both names -- must move together. Pinning the agreement on a
132+
# second shape is what catches a call site that reads a subset of the group:
133+
# the default shape renders gpus [], kubelet {} and instanceType "" as falsy,
134+
# so a consumer missing those keys is indistinguishable from one carrying them.
135+
TMP=$(mktemp -d)
136+
render_pool "$TMP/r.yaml" --set 'gpus[0].name=nvidia.com/GH200'
137+
assert_agreement "$TMP" "GPU"
138+
rm -rf "$TMP"
139+
}
140+
141+
@test "they still agree with an explicit kernelModules list" {
142+
# kernelModules has no values.yaml default, so the default and GPU shapes both
143+
# leave it unset and cannot tell a call site that carries it from one that
144+
# does not. Only a set value moves the hash on one side alone.
145+
TMP=$(mktemp -d)
146+
render_pool "$TMP/r.yaml" --set 'kernelModules[0].name=br_netfilter'
147+
assert_agreement "$TMP" "kernelModules"
148+
rm -rf "$TMP"
149+
}
150+
151+
@test "the two shapes hash differently" {
152+
# Guards the guard: if both renders collapsed to the same name, the agreement
153+
# assertions above would hold trivially and stop proving anything.
154+
TMP=$(mktemp -d)
155+
render_pool "$TMP/plain.yaml"
156+
render_pool "$TMP/gpu.yaml" --set 'gpus[0].name=nvidia.com/GH200'
157+
plain=$(job_tct_name "$TMP/plain.yaml")
158+
gpu=$(job_tct_name "$TMP/gpu.yaml")
159+
if [ "$plain" = "$gpu" ]; then
160+
echo "FAIL: the plain and GPU pools hashed to the same name ($plain)."
161+
echo "The content hash is supposed to track the rendered spec, and a GPU pool"
162+
echo "renders different nodeLabels. Equal names mean the hash input no longer"
163+
echo "covers the field that changed."
164+
false
165+
fi
166+
rm -rf "$TMP"
167+
}

‎packages/apps/kubernetes-nodes/Makefile‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,17 @@ generate:
1010
../../../hack/update-crd.sh
1111

1212
# Keep the worker-image inputs (Talos<->Kubernetes support matrix and the
13-
# kubectl.tag used by the talos-reconcile Job) and the worker machine-config
14-
# template in lockstep with the parent kubernetes chart. They are copied, not
15-
# symlinked, so `helm package` bundles them; run this after the parent chart's
16-
# `make update`.
13+
# kubectl.tag used by the talos-reconcile Job) in lockstep with the parent
14+
# kubernetes chart. They are copied, not symlinked, so `helm package` bundles
15+
# them; run this after the parent chart's `make update`.
1716
#
18-
# _talosconfigtemplate.tpl must stay byte-identical to the parent's copy: the
19-
# TalosConfigTemplate it renders is content-hash named, so any divergence would
20-
# point this pool's MachineDeployment at a different bootstrap template than the
21-
# parent rendered for the same group and roll every live worker. tests/render-
22-
# parity.sh compares the MachineDeployment across both charts and fails on it.
17+
# _talosconfigtemplate.tpl is not copied from anywhere: the parent chart no
18+
# longer carries worker node pools, so this chart owns the worker machine
19+
# config outright. It stays here because the TalosConfigTemplate it renders is
20+
# content-hash named and both consumers in this chart - nodegroup.yaml, which
21+
# writes bootstrap.configRef, and talos-reconcile-job.yaml, which creates the
22+
# object - have to hash the same bytes.
2323
update:
2424
cp ../kubernetes/files/versions.yaml files/versions.yaml
2525
cp ../kubernetes/images/kubectl.tag images/kubectl.tag
26-
cp ../kubernetes/templates/talos/_talosconfigtemplate.tpl templates/_talosconfigtemplate.tpl
2726
$(MAKE) generate

0 commit comments

Comments
 (0)