88 kube-system, which the management cluster can only reach via the
99 Kamaji-issued admin kubeconfig stored as <release>-admin-kubeconfig.
1010
11- Why not a Helm post-install/post-upgrade hook (the earlier shape) :
12- post-* hooks execute only after Helm's wait phase resolves. On the
13- tenant `kubernetes` release the wait blocks on MachineDeployment Ready,
14- which blocks on nodes registering, which blocks on kubelet CSR
15- authentication, which needs the bootstrap-token this Job produces —
16- a hard deadlock on every fresh install and on every HR reconcile that
17- finds MD in a non-Ready state. pre-* hooks are not usable either : they
18- fire before non-hook templates apply, so <release>-admin-kubeconfig
19- (materialised by Kamaji after KamajiControlPlane is applied) is not
20- yet available. The workable shape is a regular resource in the chart :
21- it lands together with the KamajiControlPlane and the talos-secrets,
22- its Pod stays pending until Kamaji publishes the admin kubeconfig
23- Secret (backoffLimit covers the retry window), then runs and completes
24- before MachineDeployment can converge. Helm's wait treats the Job as
25- another resource to reach terminal state and does not gate it on MD.
11+ Why a main-phase resource and not a Helm post-install/post-upgrade hook
12+ (the earlier shape) : the hook is unreliable on the very failure it was
13+ meant to cover. This release carries
14+ release.cozystack.io/helm-install-disable-wait : " true" (cozystack-api
15+ sets it on both Install.DisableWait and Upgrade.DisableWait), so there
16+ is NO main-resource wait phase in front of the hooks — a post-install
17+ hook is not gated on MachineDeployment Ready and does fire on a fresh
18+ install, so the "hook waits on MD, MD waits on the token" deadlock
19+ cannot happen here. What actually breaks is upgrades : a post-UPGRADE
20+ hook only runs after a successful upgrade, and a release whose upgrade
21+ keeps failing (these tenants were observed at thousands of
22+ HR.status.upgradeFailures) never reaches its hook — so the token Secret
23+ never lands and the workers stay system:anonymous. pre-* hooks are not
24+ usable either : they fire before non-hook templates apply, so
25+ <release>-admin-kubeconfig (materialised by Kamaji after the
26+ KamajiControlPlane is applied) is not yet available.
27+
28+ As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml)
29+ the Job is applied on every install and every reconcile regardless of
30+ hook phase. It lands together with the KamajiControlPlane and
31+ talos-secrets; a wait-for-kubeconfig init container holds the Pod until
32+ Kamaji publishes the admin kubeconfig Secret (the Secret volume is
33+ optional so kubelet does not FailedMount in the meantime), then the
34+ kubectl container runs and completes. Under disable-wait Helm does not
35+ gate the release on the Job, so it self-heals through the content-hash
36+ + TTL mechanism below rather than by failing the release.
37+
38+ Churn-free across upgrades. A Job's spec.template is immutable and the
39+ generated HelmRelease sets no Force, so a static Job name would fail any
40+ upgrade that lands while the Job is live (or inside
41+ ttlSecondsAfterFinished) with `spec.template : field is immutable`, and
42+ RetryOnFailure would loop on it. The Job therefore carries a
43+ content-hash name suffix over its whole rendered spec (mirrors
44+ talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in
45+ cluster.yaml) : an identical render keeps the name and Helm no-ops it;
46+ any change that flows into the spec — notably .Values.images.kubectl —
47+ yields a new name so a fresh Job runs and Helm prunes the old one. The
48+ hash also folds in the looked-up bootstrapTokenId, because the token
49+ reaches the Job only through secretKeyRef and so never changes the
50+ render on its own : without this, a rotated token (a recreated
51+ talos-secrets Secret) would leave Helm no-oping a completed Job and the
52+ tenant stuck on the stale bootstrap-token. Folding the id in rotates
53+ the name on rotation so the new token is applied and the old Job is
54+ pruned. (One idempotent re-run happens on the first reconcile after a
55+ fresh install, when the looked-up id flips from empty to its value; the
56+ kubectl apply is a no-op.)
2657*/}}
2758---
2859apiVersion : v1
@@ -32,22 +63,23 @@ metadata:
3263 namespace : {{ .Release.Namespace }}
3364 labels :
3465 {{- include "kubernetes.labels" . | nindent 4 }}
35- ---
36- apiVersion : batch/v1
37- kind : Job
38- metadata :
39- name : {{ .Release.Name }}-bootstrap-token-tenant
40- namespace : {{ .Release.Namespace }}
41- labels :
42- {{- include "kubernetes.labels" . | nindent 4 }}
66+ {{- /* Job spec rendered through a named template so the immutable Job's name
67+ can carry a content hash of the whole spec (see the header comment).
68+ Invoked with the root context, so .Release / .Values / .Files resolve
69+ as usual. */}}
70+ {{- define "kubernetes.bootstrap-token-tenant-job-spec" -}}
4371spec :
44- # backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at
45- # the tenant apiserver with exponential backoff (capped at 6 min between
46- # restarts). Worst case ~15 min, well inside the chart's HR install timeout
47- # (release.cozystack.io/helm-install-timeout: 20m on the cozyrds entry) so
48- # the Job either succeeds within the HR window or fails the chart upgrade
49- # loudly, instead of letting Helm mark the install "successful" while the
50- # bootstrap-token Secret never lands in the tenant kube-system.
72+ # restartPolicy: OnFailure + backoffLimit gives the kubectl container up to
73+ # 11 attempts at the tenant apiserver with exponential backoff, covering the
74+ # transient unreadiness window after Kamaji bootstraps the konnectivity plane.
75+ # It does NOT fail the release loudly: under helm-install-disable-wait Helm
76+ # does not wait on this Job (see the header and talos-reconcile-job.yaml), so
77+ # an exhausted backoffLimit leaves the release green with no token Secret.
78+ # ttlSecondsAfterFinished then clears the Job so a later reconcile recreates
79+ # it (unchanged render => same hash => same name) and retries. The
80+ # wait-for-kubeconfig init container — not backoffLimit — covers the
81+ # admin-kubeconfig-not-yet-provisioned window, because a Pod blocked in an
82+ # init wait never counts as a failed attempt.
5183 backoffLimit : 10
5284 ttlSecondsAfterFinished : 600
5385 template :
@@ -58,12 +90,17 @@ spec:
5890 # so the management ServiceAccount token is not needed. Drop it.
5991 automountServiceAccountToken : false
6092 serviceAccountName : {{ .Release.Name }}-bootstrap-token-tenant
93+ # Hold the Pod until Kamaji has published <release>-admin-kubeconfig; the
94+ # kubeconfig volume is optional so kubelet does not FailedMount while the
95+ # Secret is still absent. Shared with kccm/csi/cluster-autoscaler.
96+ initContainers :
97+ {{- include "kubernetes.waitForAdminKubeconfig" . | nindent 6 }}
6198 containers :
6299 - name : kubectl
63100 image : " {{ default (.Files.Get " images/kubectl.tag" | trim) .Values.images.kubectl }}"
64101 env :
65102 - name : KUBECONFIG
66- value : /tenant-admin /super-admin.svc
103+ value : /etc/kubernetes/kubeconfig /super-admin.svc
67104 - name : TOKEN_ID
68105 valueFrom :
69106 secretKeyRef :
75112 name : {{ .Release.Name }}-talos-secrets
76113 key : bootstrapTokenSecret
77114 volumeMounts :
78- - name : tenant-admin
79- mountPath : /tenant-admin
115+ - name : kubeconfig
116+ mountPath : /etc/kubernetes/kubeconfig
80117 readOnly : true
81118 resources :
82119 requests :
@@ -171,6 +208,33 @@ spec:
171208
172209 echo "[$(date -u +%H:%M:%S)] talos:node-apiserver-endpoint-discovery RBAC ensured"
173210 volumes :
174- - name : tenant-admin
211+ - name : kubeconfig
175212 secret :
176213 secretName : {{ .Release.Name }}-admin-kubeconfig
214+ optional : true
215+ {{- end }}
216+ {{- /* Render the spec, then hash it (plus the looked-up bootstrapTokenId) into
217+ a 6-char name suffix so the immutable Job is churn-free across upgrades
218+ yet still rotates when its image or the bootstrap token changes. See the
219+ header comment. lookup returns empty during `helm template`/unittest, so
220+ the token contribution is stable there and the render stays testable. */}}
221+ {{- $jobSpec := include "kubernetes.bootstrap-token-tenant-job-spec" . }}
222+ {{- $btId := "" }}
223+ {{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-talos-secrets" .Release.Name) }}
224+ {{- if and $existing $existing.data (hasKey $existing.data "bootstrapTokenId") }}
225+ {{- $btId = index $existing.data "bootstrapTokenId" | b64dec }}
226+ {{- end }}
227+ {{- $jobHash := printf "%s\n%s" $jobSpec $btId | sha256sum | trunc 6 }}
228+ {{- /* Truncate the name prefix, never the hash, so the name stays within the
229+ 63-char DNS-label limit while the 6-char content hash is always the
230+ suffix. A no-op for normal-length release names. */}}
231+ {{- $jobName := printf "%s-bootstrap-token-tenant" .Release.Name | trunc 56 | trimSuffix "-" }}
232+ ---
233+ apiVersion : batch/v1
234+ kind : Job
235+ metadata :
236+ name : {{ $jobName }}-{{ $jobHash }}
237+ namespace : {{ .Release.Namespace }}
238+ labels :
239+ {{- include "kubernetes.labels" . | nindent 4 }}
240+ {{ $jobSpec }}
0 commit comments