Skip to content

Commit d7f2799

Browse files
authored
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#3887)
## What this PR does Break the tenant `bootstrap-token-tenant` Job out of its `helm.sh/hook: post-install,post-upgrade` lifecycle and move it into the main install phase, so the `bootstrap-token-<id>` Secret lands in the tenant `kube-system` before `MachineDeployment` can converge. ### Why it deadlocks today - Job wired as `post-install,post-upgrade` fires only after Helm's wait phase resolves - Wait phase blocks on `MachineDeployment` Ready - MD Ready blocks on nodes registering - Node registration blocks on the kubelet CSR path, which needs the very `bootstrap-token-<id>` Secret this Job produces - Result: any fresh install, or any HR reconcile that finds MD in a non-Ready state, sits in a deadlock; `HR.status.upgradeFailures` climbs without the post-upgrade hook ever getting its turn `pre-install,pre-upgrade` is not a fix either — those fire before non-hook templates apply, so `<release>-admin-kubeconfig` (materialised by Kamaji after `KamajiControlPlane` is applied) is not yet available. ### The shape that works Same shape `talos-reconcile` already uses in this chart: main-phase resources. Helm applies the SA and Job together with `KamajiControlPlane` and `talos-secrets`. Job's Pod stays pending until Kamaji publishes the admin kubeconfig Secret (`backoffLimit` covers the retry window). Job completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state — it doesn't gate it on MD. ### Verification - Reproduced the deadlock on two independent long-running tenant clusters (v1.6.x), documented in #3875 - Reproduced the manual unblock (creating the `bootstrap-token-<id>` Secret by hand in the tenant `kube-system`) on both — nodes register within seconds, MD converges within minutes, HR clears - Extended `packages/apps/kubernetes/tests/talos_templates_test.yaml` to pin the absence of `helm.sh/hook` on both rendered documents; `helm unittest` passes 9/9 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md`. The diff touches `packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml` and its test — no app add/remove, no `values.schema.json`, no core/platform values, no `hack/`, no CRD, no `ApplicationDefinition`, no telemetry metric. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): tenant `bootstrap-token-<id>` Secret is now created in the main install phase instead of a post-install/post-upgrade hook, unblocking fresh installs and any HR reconcile that finds MachineDeployment in a non-Ready state. ``` Closes #3875 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved bootstrap-token job scheduling by running it as a standard chart resource instead of a Helm lifecycle hook. * Jobs are automatically recreated when relevant configuration changes, ensuring updates are applied reliably. * Added startup handling to wait for the admin kubeconfig when needed. * Improved retry and cleanup behavior so deployments are not unnecessarily blocked. * Ensured related resources are created consistently during chart deployment. * Improved job naming stability while respecting Kubernetes naming limits. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents e665cfe + 577ebfb commit d7f2799

3 files changed

Lines changed: 182 additions & 34 deletions

File tree

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 108 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,55 @@
88
kube-system, which the management cluster can only reach via the
99
Kamaji-issued admin kubeconfig stored as <release>-admin-kubeconfig.
1010

11-
The Job runs as a Helm post-install/post-upgrade hook so it executes
12-
after the KamajiControlPlane has reconciled and the admin kubeconfig
13-
Secret exists. It is idempotent (kubectl apply) and re-runs harmlessly
14-
on every upgrade.
11+
Why a main-phase resource and not a Helm post-install/post-upgrade hook
12+
(the earlier shape): the hook is unreliable on the very failure it was
13+
meant to cover. This release carries
14+
release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api
15+
sets it on both Install.DisableWait and Upgrade.DisableWait), so there
16+
is NO main-resource wait phase in front of the hooks — a post-install
17+
hook is not gated on MachineDeployment Ready and does fire on a fresh
18+
install, so the "hook waits on MD, MD waits on the token" deadlock
19+
cannot happen here. What actually breaks is upgrades: a post-UPGRADE
20+
hook only runs after a successful upgrade, and a release whose upgrade
21+
keeps failing (these tenants were observed at thousands of
22+
HR.status.upgradeFailures) never reaches its hook — so the token Secret
23+
never lands and the workers stay system:anonymous. pre-* hooks are not
24+
usable either: they fire before non-hook templates apply, so
25+
<release>-admin-kubeconfig (materialised by Kamaji after the
26+
KamajiControlPlane is applied) is not yet available.
27+
28+
As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml)
29+
the Job is applied on every install and every reconcile regardless of
30+
hook phase. It lands together with the KamajiControlPlane and
31+
talos-secrets; a wait-for-kubeconfig init container holds the Pod until
32+
Kamaji publishes the admin kubeconfig Secret (the Secret volume is
33+
optional so kubelet does not FailedMount in the meantime), then the
34+
kubectl container runs and completes. Under disable-wait Helm does not
35+
gate the release on the Job, so it self-heals through the content-hash
36+
+ TTL mechanism below rather than by failing the release.
37+
38+
Churn-free across upgrades. A Job's spec.template is immutable and the
39+
generated HelmRelease sets no Force, so a static Job name would fail any
40+
upgrade that lands while the Job is live (or inside
41+
ttlSecondsAfterFinished) with `spec.template: field is immutable`, and
42+
RetryOnFailure would loop on it. The Job therefore carries a
43+
content-hash name suffix over its whole rendered spec (mirrors
44+
talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in
45+
cluster.yaml): an identical render keeps the name and Helm no-ops it;
46+
any change that flows into the spec — notably .Values.images.kubectl —
47+
yields a new name so a fresh Job runs and Helm prunes the old one. The
48+
hash also folds in the looked-up bootstrap token — both halves,
49+
bootstrapTokenId and bootstrapTokenSecret — because the token reaches
50+
the Job only through secretKeyRef and so never changes the render on
51+
its own: without this, a rotated token (a recreated talos-secrets
52+
Secret) would leave Helm no-oping a completed Job and the tenant stuck
53+
on the stale bootstrap-token. talos-secrets.yaml regenerates the two
54+
halves independently (each guarded by its own empty check), so a change
55+
to either half must rotate the name; folding both in guarantees that.
56+
Folding the token in rotates the name on rotation so the new token is
57+
applied and the old Job is pruned. (One idempotent re-run happens on
58+
the first reconcile after a fresh install, when the looked-up token
59+
flips from empty to its value; the kubectl apply is a no-op.)
1560
*/}}
1661
---
1762
apiVersion: v1
@@ -21,30 +66,23 @@ metadata:
2166
namespace: {{ .Release.Namespace }}
2267
labels:
2368
{{- include "kubernetes.labels" . | nindent 4 }}
24-
annotations:
25-
"helm.sh/hook": post-install,post-upgrade
26-
"helm.sh/hook-weight": "0"
27-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
28-
---
29-
apiVersion: batch/v1
30-
kind: Job
31-
metadata:
32-
name: {{ .Release.Name }}-bootstrap-token-tenant
33-
namespace: {{ .Release.Namespace }}
34-
labels:
35-
{{- include "kubernetes.labels" . | nindent 4 }}
36-
annotations:
37-
"helm.sh/hook": post-install,post-upgrade
38-
"helm.sh/hook-weight": "10"
39-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
69+
{{- /* Job spec rendered through a named template so the immutable Job's name
70+
can carry a content hash of the whole spec (see the header comment).
71+
Invoked with the root context, so .Release / .Values / .Files resolve
72+
as usual. */}}
73+
{{- define "kubernetes.bootstrap-token-tenant-job-spec" -}}
4074
spec:
41-
# backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at
42-
# the tenant apiserver with exponential backoff (capped at 6 min between
43-
# restarts). Worst case ~15 min, well inside the chart's HR install timeout
44-
# (release.cozystack.io/helm-install-timeout: 20m on the cozyrds entry) so
45-
# the Job either succeeds within the HR window or fails the chart upgrade
46-
# loudly, instead of letting Helm mark the install "successful" while the
47-
# bootstrap-token Secret never lands in the tenant kube-system.
75+
# restartPolicy: OnFailure + backoffLimit gives the kubectl container up to
76+
# 11 attempts at the tenant apiserver with exponential backoff, covering the
77+
# transient unreadiness window after Kamaji bootstraps the konnectivity plane.
78+
# It does NOT fail the release loudly: under helm-install-disable-wait Helm
79+
# does not wait on this Job (see the header and talos-reconcile-job.yaml), so
80+
# an exhausted backoffLimit leaves the release green with no token Secret.
81+
# ttlSecondsAfterFinished then clears the Job so a later reconcile recreates
82+
# it (unchanged render => same hash => same name) and retries. The
83+
# wait-for-kubeconfig init container — not backoffLimit — covers the
84+
# admin-kubeconfig-not-yet-provisioned window, because a Pod blocked in an
85+
# init wait never counts as a failed attempt.
4886
backoffLimit: 10
4987
ttlSecondsAfterFinished: 600
5088
template:
@@ -55,12 +93,17 @@ spec:
5593
# so the management ServiceAccount token is not needed. Drop it.
5694
automountServiceAccountToken: false
5795
serviceAccountName: {{ .Release.Name }}-bootstrap-token-tenant
96+
# Hold the Pod until Kamaji has published <release>-admin-kubeconfig; the
97+
# kubeconfig volume is optional so kubelet does not FailedMount while the
98+
# Secret is still absent. Shared with kccm/csi/cluster-autoscaler.
99+
initContainers:
100+
{{- include "kubernetes.waitForAdminKubeconfig" . | nindent 6 }}
58101
containers:
59102
- name: kubectl
60103
image: "{{ default (.Files.Get "images/kubectl.tag" | trim) .Values.images.kubectl }}"
61104
env:
62105
- name: KUBECONFIG
63-
value: /tenant-admin/super-admin.svc
106+
value: /etc/kubernetes/kubeconfig/super-admin.svc
64107
- name: TOKEN_ID
65108
valueFrom:
66109
secretKeyRef:
@@ -72,8 +115,8 @@ spec:
72115
name: {{ .Release.Name }}-talos-secrets
73116
key: bootstrapTokenSecret
74117
volumeMounts:
75-
- name: tenant-admin
76-
mountPath: /tenant-admin
118+
- name: kubeconfig
119+
mountPath: /etc/kubernetes/kubeconfig
77120
readOnly: true
78121
resources:
79122
requests:
@@ -168,6 +211,40 @@ spec:
168211
169212
echo "[$(date -u +%H:%M:%S)] talos:node-apiserver-endpoint-discovery RBAC ensured"
170213
volumes:
171-
- name: tenant-admin
214+
- name: kubeconfig
172215
secret:
173216
secretName: {{ .Release.Name }}-admin-kubeconfig
217+
optional: true
218+
{{- end }}
219+
{{- /* Render the spec, then hash it (plus the looked-up bootstrap token, both
220+
halves) into a 6-char name suffix so the immutable Job is churn-free
221+
across upgrades yet still rotates when its image or the bootstrap token
222+
changes. See the header comment. lookup returns empty during `helm
223+
template`/unittest, so the token contribution is stable there and the
224+
render stays testable. */}}
225+
{{- $jobSpec := include "kubernetes.bootstrap-token-tenant-job-spec" . }}
226+
{{- $btId := "" }}
227+
{{- $btSecret := "" }}
228+
{{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-talos-secrets" .Release.Name) }}
229+
{{- if and $existing $existing.data }}
230+
{{- if hasKey $existing.data "bootstrapTokenId" }}
231+
{{- $btId = index $existing.data "bootstrapTokenId" | b64dec }}
232+
{{- end }}
233+
{{- if hasKey $existing.data "bootstrapTokenSecret" }}
234+
{{- $btSecret = index $existing.data "bootstrapTokenSecret" | b64dec }}
235+
{{- end }}
236+
{{- end }}
237+
{{- $jobHash := printf "%s\n%s\n%s" $jobSpec $btId $btSecret | sha256sum | trunc 6 }}
238+
{{- /* Truncate the name prefix, never the hash, so the name stays within the
239+
63-char DNS-label limit while the 6-char content hash is always the
240+
suffix. A no-op for normal-length release names. */}}
241+
{{- $jobName := printf "%s-bootstrap-token-tenant" .Release.Name | trunc 56 | trimSuffix "-" }}
242+
---
243+
apiVersion: batch/v1
244+
kind: Job
245+
metadata:
246+
name: {{ $jobName }}-{{ $jobHash }}
247+
namespace: {{ .Release.Namespace }}
248+
labels:
249+
{{- include "kubernetes.labels" . | nindent 4 }}
250+
{{ $jobSpec }}

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,8 @@
6363
RBAC, ServiceAccount and egress CiliumNetworkPolicy are main-phase too;
6464
the wait loop tolerates its dependencies (and its own network policy)
6565
not yet being in place, so Helm's kind-ordered apply needs no explicit
66-
sequencing. The bootstrap-token Job stays a post-install hook.
66+
sequencing. The bootstrap-token Job is main-phase too, for the same
67+
reasons (see bootstrap-token-tenant-job.yaml).
6768

6869
Churn-free across upgrades. The Job is immutable (spec.template cannot
6970
be patched), so it carries a content-hash name suffix — mirrors the

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 72 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -351,17 +351,87 @@ tests:
351351
- isKind:
352352
of: ServiceAccount
353353
documentIndex: 0
354+
# The ServiceAccount name is stable (not hashed) — the Job references
355+
# it by this name and only the immutable Job carries the hash suffix.
354356
- equal:
355357
path: metadata.name
356358
value: test-k8s-bootstrap-token-tenant
357359
documentIndex: 0
358360
- isKind:
359361
of: Job
360362
documentIndex: 1
361-
- equal:
363+
# The immutable Job carries a 6-char content-hash suffix (see the
364+
# dedicated content-hash cases below and the template header).
365+
- matchRegex:
362366
path: metadata.name
363-
value: test-k8s-bootstrap-token-tenant
367+
pattern: ^test-k8s-bootstrap-token-tenant-[0-9a-f]{6}$
368+
documentIndex: 1
369+
# The admin-kubeconfig volume must be optional and gated by the shared
370+
# wait-for-kubeconfig init container, so the Pod waits in init (uncounted
371+
# by backoffLimit) instead of FailedMount at t0.
372+
- equal:
373+
path: spec.template.spec.initContainers[0].name
374+
value: wait-for-kubeconfig
364375
documentIndex: 1
376+
- equal:
377+
path: spec.template.spec.volumes[0].secret.optional
378+
value: true
379+
documentIndex: 1
380+
# Main-phase, not a hook: NONE of the bootstrap-token resources may
381+
# carry helm.sh/hook. A post-install hook is not gated on
382+
# MachineDeployment here (the release sets helm-install-disable-wait,
383+
# so there is no main-resource wait in front of the hooks); the reason
384+
# to leave the hook is that a post-UPGRADE hook never runs on a release
385+
# whose upgrade keeps failing, so the token Secret never lands. Pin the
386+
# absence on both documents (0 SA, 1 Job).
387+
- notExists:
388+
path: metadata.annotations["helm.sh/hook"]
389+
documentIndex: 0
390+
- notExists:
391+
path: metadata.annotations["helm.sh/hook"]
392+
documentIndex: 1
393+
394+
# The two cases below pin the bootstrap-token Job's content-hash contract:
395+
# an image change rotates the immutable Job's name (so Helm creates a fresh
396+
# Job instead of failing on an immutable-field patch), while an unchanged
397+
# render keeps the same name (so Helm no-ops). The exact suffix is expected
398+
# to change whenever the Job spec changes; that is the signal that the Job
399+
# will roll, and the fixture is updated in lockstep.
400+
- it: bootstrap-token Job name is derived from the rendered spec
401+
templates:
402+
- templates/talos/bootstrap-token-tenant-job.yaml
403+
release:
404+
name: test-k8s
405+
namespace: tenant-test
406+
asserts:
407+
- equal:
408+
path: metadata.name
409+
value: test-k8s-bootstrap-token-tenant-fc314f
410+
documentSelector:
411+
path: kind
412+
value: Job
413+
414+
- it: bootstrap-token Job name rotates when images.kubectl changes
415+
templates:
416+
- templates/talos/bootstrap-token-tenant-job.yaml
417+
release:
418+
name: test-k8s
419+
namespace: tenant-test
420+
set:
421+
images:
422+
kubectl: example.com/kubectl:v9.9.9
423+
asserts:
424+
# A different exact hash from the default-image render above (fc314f),
425+
# proving the hash tracks the rendered spec (the kubectl image is in it),
426+
# so an image bump lands a fresh Job rather than an immutable-field patch.
427+
# Pinned exactly (like the talos-reconcile sibling cases) rather than by
428+
# regex, so a hash drift on either render is caught, not silently masked.
429+
- equal:
430+
path: metadata.name
431+
value: test-k8s-bootstrap-token-tenant-dbf53d
432+
documentSelector:
433+
path: kind
434+
value: Job
365435

366436
- it: talos-pki emits two cert-manager Certificates and two Issuers
367437
templates:

0 commit comments

Comments
 (0)