Skip to content

Commit ec603be

Browse files
committed
fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job. The deadlock the header claimed cannot occur on this release: the kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api maps to Install.DisableWait and Upgrade.DisableWait, so there is no main-resource wait phase in front of the hooks. Rewrite the rationale around the real reason to leave the hook shape: a post-upgrade hook never runs on a release whose upgrade keeps failing, so the token Secret never lands. Fix the stale 'stays a post-install hook' note in talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment, which is untrue once Helm no longer waits on the Job. Give the immutable Job a content-hash name suffix over its whole rendered spec, mirroring talos-reconcile: a static name would fail an upgrade that lands while the Job is live (or inside its TTL) with 'spec.template: field is immutable'. Fold the looked-up bootstrapTokenId into the hash so a rotated token also rolls the Job instead of Helm no-oping a completed one. Make the admin-kubeconfig volume optional and gate the Pod on the shared wait-for-kubeconfig init container, so it waits in init (uncounted by backoffLimit) instead of FailedMount at t0. Extend the template tests for the hashed name, its rotation on an image change, and the optional volume + init container. Signed-off-by: IvanHunters <[email protected]>
1 parent e2a5275 commit ec603be

3 files changed

Lines changed: 170 additions & 45 deletions

File tree

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 98 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -8,21 +8,52 @@
88
kube-system, which the management cluster can only reach via the
99
Kamaji-issued admin kubeconfig stored as <release>-admin-kubeconfig.
1010

11-
Why not a Helm post-install/post-upgrade hook (the earlier shape):
12-
post-* hooks execute only after Helm's wait phase resolves. On the
13-
tenant `kubernetes` release the wait blocks on MachineDeployment Ready,
14-
which blocks on nodes registering, which blocks on kubelet CSR
15-
authentication, which needs the bootstrap-token this Job produces —
16-
a hard deadlock on every fresh install and on every HR reconcile that
17-
finds MD in a non-Ready state. pre-* hooks are not usable either: they
18-
fire before non-hook templates apply, so <release>-admin-kubeconfig
19-
(materialised by Kamaji after KamajiControlPlane is applied) is not
20-
yet available. The workable shape is a regular resource in the chart:
21-
it lands together with the KamajiControlPlane and the talos-secrets,
22-
its Pod stays pending until Kamaji publishes the admin kubeconfig
23-
Secret (backoffLimit covers the retry window), then runs and completes
24-
before MachineDeployment can converge. Helm's wait treats the Job as
25-
another resource to reach terminal state and does not gate it on MD.
11+
Why a main-phase resource and not a Helm post-install/post-upgrade hook
12+
(the earlier shape): the hook is unreliable on the very failure it was
13+
meant to cover. This release carries
14+
release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api
15+
sets it on both Install.DisableWait and Upgrade.DisableWait), so there
16+
is NO main-resource wait phase in front of the hooks — a post-install
17+
hook is not gated on MachineDeployment Ready and does fire on a fresh
18+
install, so the "hook waits on MD, MD waits on the token" deadlock
19+
cannot happen here. What actually breaks is upgrades: a post-UPGRADE
20+
hook only runs after a successful upgrade, and a release whose upgrade
21+
keeps failing (these tenants were observed at thousands of
22+
HR.status.upgradeFailures) never reaches its hook — so the token Secret
23+
never lands and the workers stay system:anonymous. pre-* hooks are not
24+
usable either: they fire before non-hook templates apply, so
25+
<release>-admin-kubeconfig (materialised by Kamaji after the
26+
KamajiControlPlane is applied) is not yet available.
27+
28+
As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml)
29+
the Job is applied on every install and every reconcile regardless of
30+
hook phase. It lands together with the KamajiControlPlane and
31+
talos-secrets; a wait-for-kubeconfig init container holds the Pod until
32+
Kamaji publishes the admin kubeconfig Secret (the Secret volume is
33+
optional so kubelet does not FailedMount in the meantime), then the
34+
kubectl container runs and completes. Under disable-wait Helm does not
35+
gate the release on the Job, so it self-heals through the content-hash
36+
+ TTL mechanism below rather than by failing the release.
37+
38+
Churn-free across upgrades. A Job's spec.template is immutable and the
39+
generated HelmRelease sets no Force, so a static Job name would fail any
40+
upgrade that lands while the Job is live (or inside
41+
ttlSecondsAfterFinished) with `spec.template: field is immutable`, and
42+
RetryOnFailure would loop on it. The Job therefore carries a
43+
content-hash name suffix over its whole rendered spec (mirrors
44+
talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in
45+
cluster.yaml): an identical render keeps the name and Helm no-ops it;
46+
any change that flows into the spec — notably .Values.images.kubectl —
47+
yields a new name so a fresh Job runs and Helm prunes the old one. The
48+
hash also folds in the looked-up bootstrapTokenId, because the token
49+
reaches the Job only through secretKeyRef and so never changes the
50+
render on its own: without this, a rotated token (a recreated
51+
talos-secrets Secret) would leave Helm no-oping a completed Job and the
52+
tenant stuck on the stale bootstrap-token. Folding the id in rotates
53+
the name on rotation so the new token is applied and the old Job is
54+
pruned. (One idempotent re-run happens on the first reconcile after a
55+
fresh install, when the looked-up id flips from empty to its value; the
56+
kubectl apply is a no-op.)
2657
*/}}
2758
---
2859
apiVersion: v1
@@ -32,22 +63,23 @@ metadata:
3263
namespace: {{ .Release.Namespace }}
3364
labels:
3465
{{- include "kubernetes.labels" . | nindent 4 }}
35-
---
36-
apiVersion: batch/v1
37-
kind: Job
38-
metadata:
39-
name: {{ .Release.Name }}-bootstrap-token-tenant
40-
namespace: {{ .Release.Namespace }}
41-
labels:
42-
{{- include "kubernetes.labels" . | nindent 4 }}
66+
{{- /* Job spec rendered through a named template so the immutable Job's name
67+
can carry a content hash of the whole spec (see the header comment).
68+
Invoked with the root context, so .Release / .Values / .Files resolve
69+
as usual. */}}
70+
{{- define "kubernetes.bootstrap-token-tenant-job-spec" -}}
4371
spec:
44-
# backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at
45-
# the tenant apiserver with exponential backoff (capped at 6 min between
46-
# restarts). Worst case ~15 min, well inside the chart's HR install timeout
47-
# (release.cozystack.io/helm-install-timeout: 20m on the cozyrds entry) so
48-
# the Job either succeeds within the HR window or fails the chart upgrade
49-
# loudly, instead of letting Helm mark the install "successful" while the
50-
# bootstrap-token Secret never lands in the tenant kube-system.
72+
# restartPolicy: OnFailure + backoffLimit gives the kubectl container up to
73+
# 11 attempts at the tenant apiserver with exponential backoff, covering the
74+
# transient unreadiness window after Kamaji bootstraps the konnectivity plane.
75+
# It does NOT fail the release loudly: under helm-install-disable-wait Helm
76+
# does not wait on this Job (see the header and talos-reconcile-job.yaml), so
77+
# an exhausted backoffLimit leaves the release green with no token Secret.
78+
# ttlSecondsAfterFinished then clears the Job so a later reconcile recreates
79+
# it (unchanged render => same hash => same name) and retries. The
80+
# wait-for-kubeconfig init container — not backoffLimit — covers the
81+
# admin-kubeconfig-not-yet-provisioned window, because a Pod blocked in an
82+
# init wait never counts as a failed attempt.
5183
backoffLimit: 10
5284
ttlSecondsAfterFinished: 600
5385
template:
@@ -58,12 +90,17 @@ spec:
5890
# so the management ServiceAccount token is not needed. Drop it.
5991
automountServiceAccountToken: false
6092
serviceAccountName: {{ .Release.Name }}-bootstrap-token-tenant
93+
# Hold the Pod until Kamaji has published <release>-admin-kubeconfig; the
94+
# kubeconfig volume is optional so kubelet does not FailedMount while the
95+
# Secret is still absent. Shared with kccm/csi/cluster-autoscaler.
96+
initContainers:
97+
{{- include "kubernetes.waitForAdminKubeconfig" . | nindent 6 }}
6198
containers:
6299
- name: kubectl
63100
image: "{{ default (.Files.Get "images/kubectl.tag" | trim) .Values.images.kubectl }}"
64101
env:
65102
- name: KUBECONFIG
66-
value: /tenant-admin/super-admin.svc
103+
value: /etc/kubernetes/kubeconfig/super-admin.svc
67104
- name: TOKEN_ID
68105
valueFrom:
69106
secretKeyRef:
@@ -75,8 +112,8 @@ spec:
75112
name: {{ .Release.Name }}-talos-secrets
76113
key: bootstrapTokenSecret
77114
volumeMounts:
78-
- name: tenant-admin
79-
mountPath: /tenant-admin
115+
- name: kubeconfig
116+
mountPath: /etc/kubernetes/kubeconfig
80117
readOnly: true
81118
resources:
82119
requests:
@@ -171,6 +208,33 @@ spec:
171208
172209
echo "[$(date -u +%H:%M:%S)] talos:node-apiserver-endpoint-discovery RBAC ensured"
173210
volumes:
174-
- name: tenant-admin
211+
- name: kubeconfig
175212
secret:
176213
secretName: {{ .Release.Name }}-admin-kubeconfig
214+
optional: true
215+
{{- end }}
216+
{{- /* Render the spec, then hash it (plus the looked-up bootstrapTokenId) into
217+
a 6-char name suffix so the immutable Job is churn-free across upgrades
218+
yet still rotates when its image or the bootstrap token changes. See the
219+
header comment. lookup returns empty during `helm template`/unittest, so
220+
the token contribution is stable there and the render stays testable. */}}
221+
{{- $jobSpec := include "kubernetes.bootstrap-token-tenant-job-spec" . }}
222+
{{- $btId := "" }}
223+
{{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-talos-secrets" .Release.Name) }}
224+
{{- if and $existing $existing.data (hasKey $existing.data "bootstrapTokenId") }}
225+
{{- $btId = index $existing.data "bootstrapTokenId" | b64dec }}
226+
{{- end }}
227+
{{- $jobHash := printf "%s\n%s" $jobSpec $btId | sha256sum | trunc 6 }}
228+
{{- /* Truncate the name prefix, never the hash, so the name stays within the
229+
63-char DNS-label limit while the 6-char content hash is always the
230+
suffix. A no-op for normal-length release names. */}}
231+
{{- $jobName := printf "%s-bootstrap-token-tenant" .Release.Name | trunc 56 | trimSuffix "-" }}
232+
---
233+
apiVersion: batch/v1
234+
kind: Job
235+
metadata:
236+
name: {{ $jobName }}-{{ $jobHash }}
237+
namespace: {{ .Release.Namespace }}
238+
labels:
239+
{{- include "kubernetes.labels" . | nindent 4 }}
240+
{{ $jobSpec }}

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,8 @@
6363
RBAC, ServiceAccount and egress CiliumNetworkPolicy are main-phase too;
6464
the wait loop tolerates its dependencies (and its own network policy)
6565
not yet being in place, so Helm's kind-ordered apply needs no explicit
66-
sequencing. The bootstrap-token Job stays a post-install hook.
66+
sequencing. The bootstrap-token Job is main-phase too, for the same
67+
reasons (see bootstrap-token-tenant-job.yaml).
6768

6869
Churn-free across upgrades. The Job is immutable (spec.template cannot
6970
be patched), so it carries a content-hash name suffix — mirrors the

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 70 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -351,33 +351,93 @@ tests:
351351
- isKind:
352352
of: ServiceAccount
353353
documentIndex: 0
354+
# The ServiceAccount name is stable (not hashed) — the Job references
355+
# it by this name and only the immutable Job carries the hash suffix.
354356
- equal:
355357
path: metadata.name
356358
value: test-k8s-bootstrap-token-tenant
357359
documentIndex: 0
358360
- isKind:
359361
of: Job
360362
documentIndex: 1
361-
- equal:
363+
# The immutable Job carries a 6-char content-hash suffix (see the
364+
# dedicated content-hash cases below and the template header).
365+
- matchRegex:
362366
path: metadata.name
363-
value: test-k8s-bootstrap-token-tenant
367+
pattern: ^test-k8s-bootstrap-token-tenant-[0-9a-f]{6}$
368+
documentIndex: 1
369+
# Point 4 of the review: the admin-kubeconfig volume must be optional and
370+
# gated by the shared wait-for-kubeconfig init container, so the Pod waits
371+
# in init (uncounted by backoffLimit) instead of FailedMount at t0.
372+
- equal:
373+
path: spec.template.spec.initContainers[0].name
374+
value: wait-for-kubeconfig
375+
documentIndex: 1
376+
- equal:
377+
path: spec.template.spec.volumes[0].secret.optional
378+
value: true
364379
documentIndex: 1
365380
# Main-phase, not a hook: NONE of the bootstrap-token resources may
366-
# carry helm.sh/hook. The Job must run at t0 alongside the tenant
367-
# KamajiControlPlane so that the bootstrap-token-<id> Secret lands
368-
# in the tenant kube-system before MachineDeployment starts
369-
# scaling. post-install/post-upgrade hooks would gate this on
370-
# Helm's wait phase, which itself waits on MachineDeployment
371-
# Ready, which waits on nodes to register, which waits on the
372-
# very token this Job produces — a deadlock. Pin the absence on
373-
# both documents (0 SA, 1 Job).
381+
# carry helm.sh/hook. A post-install hook is not gated on
382+
# MachineDeployment here (the release sets helm-install-disable-wait,
383+
# so there is no main-resource wait in front of the hooks); the reason
384+
# to leave the hook is that a post-UPGRADE hook never runs on a release
385+
# whose upgrade keeps failing, so the token Secret never lands. Pin the
386+
# absence on both documents (0 SA, 1 Job).
374387
- notExists:
375388
path: metadata.annotations["helm.sh/hook"]
376389
documentIndex: 0
377390
- notExists:
378391
path: metadata.annotations["helm.sh/hook"]
379392
documentIndex: 1
380393

394+
# The two cases below pin the bootstrap-token Job's content-hash contract:
395+
# an image change rotates the immutable Job's name (so Helm creates a fresh
396+
# Job instead of failing on an immutable-field patch), while an unchanged
397+
# render keeps the same name (so Helm no-ops). The exact suffix is expected
398+
# to change whenever the Job spec changes; that is the signal that the Job
399+
# will roll, and the fixture is updated in lockstep.
400+
- it: bootstrap-token Job name is derived from the rendered spec
401+
templates:
402+
- templates/talos/bootstrap-token-tenant-job.yaml
403+
release:
404+
name: test-k8s
405+
namespace: tenant-test
406+
asserts:
407+
- equal:
408+
path: metadata.name
409+
value: test-k8s-bootstrap-token-tenant-ddf388
410+
documentSelector:
411+
path: kind
412+
value: Job
413+
414+
- it: bootstrap-token Job name rotates when images.kubectl changes
415+
templates:
416+
- templates/talos/bootstrap-token-tenant-job.yaml
417+
release:
418+
name: test-k8s
419+
namespace: tenant-test
420+
set:
421+
images:
422+
kubectl: example.com/kubectl:v9.9.9
423+
asserts:
424+
# Still a valid hashed name ...
425+
- matchRegex:
426+
path: metadata.name
427+
pattern: ^test-k8s-bootstrap-token-tenant-[0-9a-f]{6}$
428+
documentSelector:
429+
path: kind
430+
value: Job
431+
# ... but a different one from the default-image render above (e8c6ea),
432+
# proving the hash tracks the rendered spec (the kubectl image is in it),
433+
# so an image bump lands a fresh Job rather than an immutable-field patch.
434+
- notMatchRegex:
435+
path: metadata.name
436+
pattern: -ddf388$
437+
documentSelector:
438+
path: kind
439+
value: Job
440+
381441
- it: talos-pki emits two cert-manager Certificates and two Issuers
382442
templates:
383443
- templates/talos/talos-pki.yaml

0 commit comments

Comments
 (0)