Skip to content

Commit 32c4d60

Browse files
committed
fix(kubernetes): run the Talos reconcile Job in the main install phase
The talos-reconcile Job is the only thing that creates the TalosConfigTemplate the worker MachineDeployment's bootstrap.configRef points to; CAPI blocks MachineSet creation until it exists, so tenant workers cannot scale until the Job has run. The Job was a post-install/post-upgrade Helm hook, which runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release emits in-tenant addon HelmReleases — notably cilium — whose readiness needs worker nodes, and those nodes need the TalosConfigTemplate this Job produces. So when the main-resource wait is in effect the install deadlocks: it waits for cilium to be Ready, cilium waits for workers, the workers wait for the template, and the template waits for this hook — which never runs because the install never finishes the wait that precedes the hook phase. The install times out and the MachineDeployment never scales. The release sets helm-install-disable-wait to skip that wait, but its effect is unreliable, which makes the failure bimodal: the template lands in ~46s when the wait is skipped, and the install is stranded when it is not. Render the Job and its RBAC/ServiceAccount/egress policy as ordinary main-phase resources. The Job is then created during the apply itself, not after the wait, and runs on the management cluster where nodes exist (unlike the tenant cilium-operator), so it produces the template, the MachineDeployment scales, workers join, cilium goes Ready, and the wait (if any) closes within the install budget — independent of whether helm-install-disable-wait takes effect. The Job is immutable, so it carries a content-hash name suffix over its rendered spec (mirrors the KubevirtMachineTemplate idiom in cluster.yaml): an identical render keeps the name and Helm no-ops it, while any spec change yields a new name and a fresh Job, so no spec change ever lands on the same name and triggers an immutable-field patch error. The applied TalosConfigTemplate keeps its stable <release>-<group> name, so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 4404c5c commit 32c4d60

10 files changed

Lines changed: 459 additions & 188 deletions

File tree

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
exist on first Helm render, so the gate fails, the TCT is
3737
skipped, and Helm's release-storage three-way merge then refuses
3838
to upgrade because no values changed. Both objects are now
39-
produced by the post-install/post-upgrade Helm hook Job in
39+
produced by the main-phase Job in
4040
templates/talos/talos-reconcile-job.yaml, which blocks until the
4141
four dependencies exist and then `kubectl apply`s deterministic
4242
results. The Job owns the TCT through an ownerReference to the
@@ -321,10 +321,9 @@ spec:
321321
protocol: TCP
322322
{{- /* Initial render lists only the DNS SANs because the apiserver
323323
Service ClusterIP is not yet allocated when Helm executes the
324-
template. The talos-reconcile post-install/post-upgrade hook
325-
re-patches certSANs with the live ClusterIP once Kamaji emits
326-
the Service, and Kamaji then re-issues the apiserver TLS cert
327-
to cover it. */}}
324+
template. The talos-reconcile main-phase Job re-patches certSANs
325+
with the live ClusterIP once Kamaji emits the Service, and
326+
Kamaji then re-issues the apiserver TLS cert to cover it. */}}
328327
certSANs:
329328
- {{ printf "%s.%s.svc" .Release.Name .Release.Namespace }}
330329
- {{ printf "%s.%s.svc.%s" .Release.Name .Release.Namespace (include "kubernetes.tenantClusterDomain" .) }}
@@ -578,14 +577,15 @@ metadata:
578577
produce a valid Talos worker machineconfig at Helm template time
579578
because three of the inputs (apiserver Service ClusterIP, Talos
580579
CA from cert-manager, Kamaji-issued tenant CA) are created
581-
asynchronously after Helm applies this manifest. The
582-
talos-reconcile-job hook (templates/talos/talos-reconcile-job.yaml)
583-
waits for those inputs and produces the TalosConfigTemplate with
584-
a KamajiControlPlane ownerReference for natural GC. The
580+
asynchronously after Helm applies this manifest. The main-phase
581+
Job in templates/talos/talos-reconcile-job.yaml waits for those
582+
inputs and produces the TalosConfigTemplate with a
583+
KamajiControlPlane ownerReference for natural GC. That Job runs in
584+
the main install phase, so it applies the template as soon as the
585+
inputs appear — in parallel with the rest of the install. The
585586
MachineDeployment below still references the TCT name; CAPI's
586587
Machine controller blocks on its absence and unblocks the moment
587-
the hook applies it, which happens in the same post-install/
588-
post-upgrade cycle. */}}
588+
the Job applies it. */}}
589589
---
590590
{{- $context := deepCopy $ }}
591591
{{- $_ := set $context "group" $group }}

‎packages/apps/kubernetes/templates/helmreleases/cilium.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{{- define "cozystack.defaultCiliumValues" -}}
22
{{- /* Cilium runs hostNetwork on the Talos worker nodes and resolves the
33
apiserver via the worker's /etc/hosts, which the talos-reconcile
4-
post-install hook seeds with the Kamaji Service ClusterIP under
4+
main-phase Job seeds with the Kamaji Service ClusterIP under
55
<release>.<ns>.svc through machine.network.extraHostEntries. So a
66
plain DNS host works here without a lookup-and-reuse race: the
77
cert covers the same DNS name from chart render time, and the

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 202 additions & 138 deletions
Large diffs are not rendered by default.

‎packages/apps/kubernetes/tests/cluster_test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ values:
88

99
# With one nodeGroup (md0) the cluster.yaml template renders in this
1010
# order. The TalosConfigTemplate is no longer rendered here — it is
11-
# produced at runtime by the post-install hook in
11+
# produced at runtime by the main-phase Job in
1212
# templates/talos/talos-reconcile-job.yaml, which writes the TCT YAML
1313
# from a string literal in the Job's command.
1414
# 0: Cluster

‎packages/apps/kubernetes/tests/gpu_node_labels_test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
suite: GPU node-group Talos worker nodeLabels tests
22

33
# The TalosConfigTemplate that defines worker nodeLabels is not rendered
4-
# directly by helm — the kubernetes chart emits a post-install Job
4+
# directly by helm — the kubernetes chart emits a main-phase Job
55
# (templates/talos/talos-reconcile-job.yaml) whose container command
6-
# carries the TCT YAML as a string literal that the hook applies at
6+
# carries the TCT YAML as a string literal that the Job applies at
77
# runtime. helm-unittest therefore asserts on the rendered Job's script
88
# via matchRegex / notMatchRegex, looking for the machine.nodeLabels
99
# block GPU node-groups must inject so HAMi's hami-device-plugin

‎packages/apps/kubernetes/tests/kubelet_reservation_test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ suite: kubelet reservation validation tests
33
# These tests cover the failure-path guards in cluster.yaml's per-nodeGroup
44
# validation block (memory/CPU format checks, total-reserved vs effective
55
# invariants, mixed-unit eviction rejection, etc.). The TalosConfigTemplate
6-
# is no longer rendered by cluster.yaml — the post-install hook in
6+
# is no longer rendered by cluster.yaml — the main-phase Job in
77
# templates/talos/talos-reconcile-job.yaml produces it at runtime — so the
88
# test file only asserts on the kubeadm-era validation paths that survived
99
# the Talos rollover. The kubelet-reservation math used to apply
1010
# machine.kubelet.extraConfig is exercised separately by helm-unittest on
11-
# the reconcile hook template (see tests/gpu_node_labels_test.yaml for the
11+
# the reconcile Job template (see tests/gpu_node_labels_test.yaml for the
1212
# matchRegex pattern against the Job's command literal).
1313

1414
templates:

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 217 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,26 @@
1-
suite: talos hook + secrets template render tests
1+
suite: talos main-phase Job + secrets template render tests
22

3-
# Minimal render-coverage for the four new talos/ templates introduced in
4-
# PR #2610. The TalosConfigTemplate embedded in talos-reconcile-job.yaml is
5-
# asserted separately by gpu_node_labels_test.yaml (matchRegex against the
6-
# Job's command literal); this suite covers the surrounding RBAC/Job/Secret
7-
# scaffolding so a future refactor breaking ServiceAccount/Role wiring is
8-
# caught locally before chart push. helm-unittest sees documents in source-
9-
# file order (not helm template's normalized render order), so the
10-
# documentIndex constants below follow the original template file
3+
# Render-coverage for the talos/ templates. The TalosConfigTemplate embedded in
4+
# talos-reconcile-job.yaml is asserted separately by gpu_node_labels_test.yaml
5+
# (matchRegex against the Job's command literal); this suite covers the
6+
# surrounding RBAC/Job/Secret scaffolding so a future refactor breaking
7+
# ServiceAccount/Role wiring is caught locally before chart push. helm-unittest
8+
# sees documents in source-file order (not helm template's normalized render
9+
# order), so the documentIndex constants below follow the original template file
1110
# layout — update if a top-level object is added/removed/reordered.
11+
#
12+
# The talos-reconcile resources run in the MAIN install phase (no helm.sh/hook
13+
# annotations) so the Job applies the TalosConfigTemplate at t0, in parallel
14+
# with the MachineDeployment, instead of after the install nearly finishes. The
15+
# Job is immutable, so it carries a content-hash name suffix (mirrors the
16+
# KubevirtMachineTemplate idiom in cluster.yaml) to stay churn-free on upgrade.
17+
# The assertions below pin both properties.
1218

1319
values:
1420
- values/common.yaml
1521

1622
tests:
17-
- it: talos-reconcile-job renders the full hook RBAC stack + Job
23+
- it: talos-reconcile-job renders the full RBAC stack + Job
1824
templates:
1925
- templates/talos/talos-reconcile-job.yaml
2026
release:
@@ -34,6 +40,31 @@ tests:
3440
path: metadata.name
3541
value: test-k8s-talos-reconcile
3642
documentIndex: 0
43+
# Main-phase, not a hook: NONE of the talos-reconcile resources may carry
44+
# helm.sh/hook. The whole set must be main-phase together — a re-hooked
45+
# Role/RoleBinding (hook-delete-policy: hook-succeeded) would be deleted
46+
# after the hook ran, leaving the main-phase Job permanently without
47+
# permissions. Pin the absence on every document (0 SA, 1 Role,
48+
# 2 CiliumNetworkPolicy, 3 RoleBinding, 4 ClusterRole, 5 ClusterRoleBinding,
49+
# 6 Job).
50+
- notExists:
51+
path: metadata.annotations["helm.sh/hook"]
52+
documentIndex: 0
53+
- notExists:
54+
path: metadata.annotations["helm.sh/hook"]
55+
documentIndex: 1
56+
- notExists:
57+
path: metadata.annotations["helm.sh/hook"]
58+
documentIndex: 2
59+
- notExists:
60+
path: metadata.annotations["helm.sh/hook"]
61+
documentIndex: 3
62+
- notExists:
63+
path: metadata.annotations["helm.sh/hook"]
64+
documentIndex: 4
65+
- notExists:
66+
path: metadata.annotations["helm.sh/hook"]
67+
documentIndex: 5
3768
- isKind:
3869
of: ClusterRole
3970
documentIndex: 4
@@ -55,6 +86,181 @@ tests:
5586
- isKind:
5687
of: Job
5788
documentIndex: 6
89+
# Main-phase Job: no hook annotations, so it is applied at t0 alongside
90+
# the MachineDeployment instead of after the install nearly finishes.
91+
- notExists:
92+
path: metadata.annotations["helm.sh/hook"]
93+
documentIndex: 6
94+
# Content-hash name suffix keeps the immutable Job churn-free on upgrade.
95+
- matchRegex:
96+
path: metadata.name
97+
pattern: ^test-k8s-talos-reconcile-md0-[0-9a-f]{6}$
98+
documentIndex: 6
99+
# TTL clears a Complete or Failed Job so a later reconcile can recreate
100+
# and retry it (the release disables Helm's wait, so a failed run is
101+
# invisible to Flux and would otherwise stay Failed forever).
102+
- equal:
103+
path: spec.ttlSecondsAfterFinished
104+
value: 600
105+
documentIndex: 6
106+
# The Job still applies the TalosConfigTemplate with a KamajiControlPlane
107+
# ownerReference (for GC) and the extraHostEntries that route workers to
108+
# the apiserver Service by DNS name.
109+
- matchRegex:
110+
path: spec.template.spec.containers[0].command[2]
111+
pattern: 'kind: TalosConfigTemplate'
112+
documentIndex: 6
113+
- matchRegex:
114+
path: spec.template.spec.containers[0].command[2]
115+
pattern: 'kind: KamajiControlPlane'
116+
documentIndex: 6
117+
- matchRegex:
118+
path: spec.template.spec.containers[0].command[2]
119+
pattern: 'extraHostEntries:'
120+
documentIndex: 6
121+
122+
# The two cases below pin the content-hash contract: a TCT-affecting change
123+
# rotates the immutable Job's name (so Helm creates a fresh Job instead of
124+
# failing on an immutable-field patch), while an unchanged render keeps the
125+
# same name (so Helm no-ops). The two renders differ ONLY in nodeGroups.md0.gpus
126+
# — everything else (release, namespace, resources, versions) is identical — so
127+
# the differing suffix proves the hash tracks the rendered spec. These exact
128+
# suffixes are expected to change whenever the Job spec changes; that is the
129+
# signal that the Job will roll, and the fixtures are updated in lockstep.
130+
- it: content-hash Job name is derived from the rendered spec (gpus unset)
131+
templates:
132+
- templates/talos/talos-reconcile-job.yaml
133+
release:
134+
name: test-k8s
135+
namespace: tenant-test
136+
set:
137+
_cluster:
138+
cluster-domain: cozy.local
139+
_namespace:
140+
etcd: cozy-etcd
141+
ingress: nginx
142+
host: example.com
143+
nodeGroups:
144+
md0:
145+
minReplicas: 0
146+
maxReplicas: 10
147+
instanceType: ""
148+
diskSize: 20Gi
149+
roles:
150+
- worker
151+
resources:
152+
cpu: 8
153+
memory: 16Gi
154+
kubelet: {}
155+
gpus: []
156+
asserts:
157+
- equal:
158+
path: metadata.name
159+
value: test-k8s-talos-reconcile-md0-6ddd7d
160+
documentSelector:
161+
path: kind
162+
value: Job
163+
164+
- it: content-hash Job name rotates when a TCT-affecting value changes (gpus)
165+
templates:
166+
- templates/talos/talos-reconcile-job.yaml
167+
release:
168+
name: test-k8s
169+
namespace: tenant-test
170+
set:
171+
_cluster:
172+
cluster-domain: cozy.local
173+
_namespace:
174+
etcd: cozy-etcd
175+
ingress: nginx
176+
host: example.com
177+
nodeGroups:
178+
md0:
179+
minReplicas: 0
180+
maxReplicas: 10
181+
instanceType: ""
182+
diskSize: 20Gi
183+
roles:
184+
- worker
185+
resources:
186+
cpu: 8
187+
memory: 16Gi
188+
kubelet: {}
189+
gpus:
190+
- name: nvidia.com/GH200
191+
asserts:
192+
# Only gpus changed vs the case above, yet the suffix differs (ceb45e vs
193+
# 6ddd7d) — the content hash tracks the rendered spec, and the TCT the Job
194+
# applies is what changed (GPU nodeLabels).
195+
- equal:
196+
path: metadata.name
197+
value: test-k8s-talos-reconcile-md0-ceb45e
198+
documentSelector:
199+
path: kind
200+
value: Job
201+
202+
- it: content-hash Job name is truncated to stay within the 63-char DNS-label limit
203+
templates:
204+
- templates/talos/talos-reconcile-job.yaml
205+
release:
206+
# release + group chosen so "<release>-talos-reconcile-<group>" alone is 61
207+
# chars; adding the "-<6hexhash>" suffix would push it to 68 without the
208+
# trunc-56 prefix guard, and the API server would reject the over-limit
209+
# Job name.
210+
name: kubernetes-pretty-long-tenant-clustername
211+
namespace: tenant-test
212+
set:
213+
_cluster:
214+
cluster-domain: cozy.local
215+
_namespace:
216+
etcd: cozy-etcd
217+
ingress: nginx
218+
host: example.com
219+
nodeGroups:
220+
md0:
221+
minReplicas: 0
222+
maxReplicas: 10
223+
instanceType: ""
224+
diskSize: 20Gi
225+
roles:
226+
- worker
227+
resources:
228+
cpu: 2
229+
memory: 4Gi
230+
kubelet: {}
231+
gpus: []
232+
asserts:
233+
# (a) within the 63-char metadata.name / job-name label limit.
234+
- matchRegex:
235+
path: metadata.name
236+
pattern: ^.{1,63}$
237+
documentSelector:
238+
path: kind
239+
value: Job
240+
# (b) the 6-char content hash is never truncated — it is always the suffix,
241+
# so a changed spec still rotates the name.
242+
- matchRegex:
243+
path: metadata.name
244+
pattern: -[0-9a-f]{6}$
245+
documentSelector:
246+
path: kind
247+
value: Job
248+
# (c) trimSuffix "-" guarantees no trailing/double dash before the hash even
249+
# when trunc lands on a dash.
250+
- matchRegex:
251+
path: metadata.name
252+
pattern: '[^-]-[0-9a-f]{6}$'
253+
documentSelector:
254+
path: kind
255+
value: Job
256+
# (d) the prefix is genuinely truncated: the full "talos-reconcile-md0" infix
257+
# would be present if trunc-56 had not fired.
258+
- notMatchRegex:
259+
path: metadata.name
260+
pattern: talos-reconcile-md0
261+
documentSelector:
262+
path: kind
263+
value: Job
58264

59265
- it: bootstrap-token tenant Job renders ServiceAccount + Job
60266
templates:
@@ -130,7 +336,7 @@ tests:
130336
value: test-k8s-talos-tls-cert
131337
documentIndex: 3
132338

133-
- it: talos-secrets Secret carries the four token-component keys the reconcile hook env binds from
339+
- it: talos-secrets Secret carries the four token-component keys the reconcile Job env binds from
134340
templates:
135341
- templates/talos/talos-secrets.yaml
136342
release:

‎packages/system/kubernetes-rd/cozyrds/kubernetes.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,14 @@ metadata:
1717
# annotation is raised, update that init deadline correspondingly.
1818
release.cozystack.io/helm-install-timeout: "20m"
1919
# The kubernetes chart emits in-tenant addon HelmReleases (cilium, coredns,
20-
# csi, ingress-nginx, metrics-server, ouroboros) and ships a talos-reconcile
21-
# post-install hook that produces the TalosConfigTemplate the worker
20+
# csi, ingress-nginx, metrics-server, ouroboros) and a main-phase
21+
# talos-reconcile Job that produces the TalosConfigTemplate the worker
2222
# MachineSets clone from. Without DisableWait the helm-controller waits for
23-
# the addon HelmReleases to be Ready before running the hook, but those
23+
# the addon HelmReleases to be Ready before the release settles, but those
2424
# cannot be Ready until worker nodes exist, and the workers cannot be
25-
# created without the TalosConfigTemplate the hook is supposed to produce.
26-
# Skip the readiness wait so the hook fires and the chicken-and-egg
27-
# resolves.
25+
# created without the TalosConfigTemplate the Job produces. Skip the
26+
# readiness wait so the release settles and the chicken-and-egg resolves;
27+
# the talos-reconcile Job runs to completion asynchronously.
2828
release.cozystack.io/helm-install-disable-wait: "true"
2929
spec:
3030
application:

‎pkg/config/config.go‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,14 +39,15 @@ const HelmUpgradeTimeoutAnnotation = "release.cozystack.io/helm-upgrade-timeout"
3939
// HelmInstallDisableWaitAnnotation is the ApplicationDefinition metadata
4040
// annotation key that sets HelmReleaseSpec.Install.DisableWait and
4141
// Upgrade.DisableWait to true for a given Application kind. Use when the
42-
// parent chart emits child HelmReleases whose readiness depends on
43-
// post-install hooks of the parent (e.g. the Kubernetes Application's
44-
// talos-reconcile hook produces the TalosConfigTemplate that worker
45-
// MachineSets clone from; without DisableWait the helm-controller waits
46-
// for the emitted in-tenant addon HelmReleases to be Ready before
47-
// running the hook, but those HelmReleases can never be Ready because
48-
// they have no worker nodes to schedule on yet, and the hook never
49-
// runs).
42+
// parent chart emits child HelmReleases that cannot become Ready during
43+
// the parent's own install (e.g. the Kubernetes Application emits
44+
// in-tenant addon HelmReleases that have no worker nodes to schedule on
45+
// until the worker MachineSets come up, plus a main-phase talos-reconcile
46+
// Job that produces the TalosConfigTemplate those MachineSets clone from).
47+
// Without DisableWait the helm-controller blocks on the addon HelmReleases
48+
// becoming Ready, which cannot happen during install, so the release never
49+
// settles. DisableWait lets it settle while the addon HelmReleases and the
50+
// reconcile Job converge asynchronously.
5051
const HelmInstallDisableWaitAnnotation = "release.cozystack.io/helm-install-disable-wait"
5152

5253
// helmTimeoutPattern mirrors the CRD validation pattern used by Flux

0 commit comments

Comments
 (0)