Skip to content

Commit 2b90e8c

Browse files
[Backport release-1.6] fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#4541)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Backports #3887 to `release-1.6`. The fix reached `main` only: `v1.6.3` and `v1.6.4` both still ship the bootstrap-token Job as a `post-install,post-upgrade` Helm hook, and `git tag --contains` on the merge commit returns `v1.7.0-alpha.1` and nothing from the 1.6 line. Tracked as #4539. #3875 describes the deadlock this removes: the chart upgrade waits for `MachineDeployment` and the child HelmReleases, workers cannot register without the `bootstrap-token-<id>` Secret in the guest cluster, and the hook that creates that Secret only runs once the wait has already succeeded. The wait cannot succeed without nodes, so the upgrade times out and Flux retries the same cycle indefinitely. It is deterministic rather than flaky, and it takes out every tenant Kubernetes cluster on the platform at once. What makes the backport worth filing separately: anyone hitting this on 1.6.3 finds #3875 closed in the tracker, upgrades to 1.6.4 expecting the fix, and lands in exactly the same state. Observed on a production cluster before the manual workaround: machines stuck in `Provisioned` with no `NODENAME`, VMIs `Running` with IPs assigned, guest clusters reporting `No resources found` for nodes, and no bootstrap-token Secret anywhere in the guest `kube-system`. Creating the Secret by hand brought nodes up within 90 seconds on each affected tenant. `git cherry-pick --mainline 1` of the merge commit applied cleanly with no conflicts, and nothing was changed on top: the diff here is identical to what merged on `main`. Validation, `helm unittest` in `packages/apps/kubernetes` including `tests/talos_templates_test.yaml` which this change extends: ``` Charts: 1 passed, 1 total Test Suites: 19 passed, 19 total Tests: 189 passed, 189 total ``` ### Screenshots Not applicable, no UI changes. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): create the tenant bootstrap-token Secret outside the Helm hook, so Talos workers can register during a chart upgrade instead of deadlocking against the wait that needs them ```
2 parents 234dfda + 844bd33 commit 2b90e8c

3 files changed

Lines changed: 182 additions & 34 deletions

File tree

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 108 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,55 @@
88
kube-system, which the management cluster can only reach via the
99
Kamaji-issued admin kubeconfig stored as <release>-admin-kubeconfig.
1010

11-
The Job runs as a Helm post-install/post-upgrade hook so it executes
12-
after the KamajiControlPlane has reconciled and the admin kubeconfig
13-
Secret exists. It is idempotent (kubectl apply) and re-runs harmlessly
14-
on every upgrade.
11+
Why a main-phase resource and not a Helm post-install/post-upgrade hook
12+
(the earlier shape): the hook is unreliable on the very failure it was
13+
meant to cover. This release carries
14+
release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api
15+
sets it on both Install.DisableWait and Upgrade.DisableWait), so there
16+
is NO main-resource wait phase in front of the hooks — a post-install
17+
hook is not gated on MachineDeployment Ready and does fire on a fresh
18+
install, so the "hook waits on MD, MD waits on the token" deadlock
19+
cannot happen here. What actually breaks is upgrades: a post-UPGRADE
20+
hook only runs after a successful upgrade, and a release whose upgrade
21+
keeps failing (these tenants were observed at thousands of
22+
HR.status.upgradeFailures) never reaches its hook — so the token Secret
23+
never lands and the workers stay system:anonymous. pre-* hooks are not
24+
usable either: they fire before non-hook templates apply, so
25+
<release>-admin-kubeconfig (materialised by Kamaji after the
26+
KamajiControlPlane is applied) is not yet available.
27+
28+
As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml)
29+
the Job is applied on every install and every reconcile regardless of
30+
hook phase. It lands together with the KamajiControlPlane and
31+
talos-secrets; a wait-for-kubeconfig init container holds the Pod until
32+
Kamaji publishes the admin kubeconfig Secret (the Secret volume is
33+
optional so kubelet does not FailedMount in the meantime), then the
34+
kubectl container runs and completes. Under disable-wait Helm does not
35+
gate the release on the Job, so it self-heals through the content-hash
36+
+ TTL mechanism below rather than by failing the release.
37+
38+
Churn-free across upgrades. A Job's spec.template is immutable and the
39+
generated HelmRelease sets no Force, so a static Job name would fail any
40+
upgrade that lands while the Job is live (or inside
41+
ttlSecondsAfterFinished) with `spec.template: field is immutable`, and
42+
RetryOnFailure would loop on it. The Job therefore carries a
43+
content-hash name suffix over its whole rendered spec (mirrors
44+
talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in
45+
cluster.yaml): an identical render keeps the name and Helm no-ops it;
46+
any change that flows into the spec — notably .Values.images.kubectl —
47+
yields a new name so a fresh Job runs and Helm prunes the old one. The
48+
hash also folds in the looked-up bootstrap token — both halves,
49+
bootstrapTokenId and bootstrapTokenSecret — because the token reaches
50+
the Job only through secretKeyRef and so never changes the render on
51+
its own: without this, a rotated token (a recreated talos-secrets
52+
Secret) would leave Helm no-oping a completed Job and the tenant stuck
53+
on the stale bootstrap-token. talos-secrets.yaml regenerates the two
54+
halves independently (each guarded by its own empty check), so a change
55+
to either half must rotate the name; folding both in guarantees that.
56+
Folding the token in rotates the name on rotation so the new token is
57+
applied and the old Job is pruned. (One idempotent re-run happens on
58+
the first reconcile after a fresh install, when the looked-up token
59+
flips from empty to its value; the kubectl apply is a no-op.)
1560
*/}}
1661
---
1762
apiVersion: v1
@@ -21,30 +66,23 @@ metadata:
2166
namespace: {{ .Release.Namespace }}
2267
labels:
2368
{{- include "kubernetes.labels" . | nindent 4 }}
24-
annotations:
25-
"helm.sh/hook": post-install,post-upgrade
26-
"helm.sh/hook-weight": "0"
27-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
28-
---
29-
apiVersion: batch/v1
30-
kind: Job
31-
metadata:
32-
name: {{ .Release.Name }}-bootstrap-token-tenant
33-
namespace: {{ .Release.Namespace }}
34-
labels:
35-
{{- include "kubernetes.labels" . | nindent 4 }}
36-
annotations:
37-
"helm.sh/hook": post-install,post-upgrade
38-
"helm.sh/hook-weight": "10"
39-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
69+
{{- /* Job spec rendered through a named template so the immutable Job's name
70+
can carry a content hash of the whole spec (see the header comment).
71+
Invoked with the root context, so .Release / .Values / .Files resolve
72+
as usual. */}}
73+
{{- define "kubernetes.bootstrap-token-tenant-job-spec" -}}
4074
spec:
41-
# backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at
42-
# the tenant apiserver with exponential backoff (capped at 6 min between
43-
# restarts). Worst case ~15 min, well inside the chart's HR install timeout
44-
# (release.cozystack.io/helm-install-timeout: 20m on the cozyrds entry) so
45-
# the Job either succeeds within the HR window or fails the chart upgrade
46-
# loudly, instead of letting Helm mark the install "successful" while the
47-
# bootstrap-token Secret never lands in the tenant kube-system.
75+
# restartPolicy: OnFailure + backoffLimit gives the kubectl container up to
76+
# 11 attempts at the tenant apiserver with exponential backoff, covering the
77+
# transient unreadiness window after Kamaji bootstraps the konnectivity plane.
78+
# It does NOT fail the release loudly: under helm-install-disable-wait Helm
79+
# does not wait on this Job (see the header and talos-reconcile-job.yaml), so
80+
# an exhausted backoffLimit leaves the release green with no token Secret.
81+
# ttlSecondsAfterFinished then clears the Job so a later reconcile recreates
82+
# it (unchanged render => same hash => same name) and retries. The
83+
# wait-for-kubeconfig init container — not backoffLimit — covers the
84+
# admin-kubeconfig-not-yet-provisioned window, because a Pod blocked in an
85+
# init wait never counts as a failed attempt.
4886
backoffLimit: 10
4987
ttlSecondsAfterFinished: 600
5088
template:
@@ -55,12 +93,17 @@ spec:
5593
# so the management ServiceAccount token is not needed. Drop it.
5694
automountServiceAccountToken: false
5795
serviceAccountName: {{ .Release.Name }}-bootstrap-token-tenant
96+
# Hold the Pod until Kamaji has published <release>-admin-kubeconfig; the
97+
# kubeconfig volume is optional so kubelet does not FailedMount while the
98+
# Secret is still absent. Shared with kccm/csi/cluster-autoscaler.
99+
initContainers:
100+
{{- include "kubernetes.waitForAdminKubeconfig" . | nindent 6 }}
58101
containers:
59102
- name: kubectl
60103
image: "{{ default (.Files.Get "images/kubectl.tag" | trim) .Values.images.kubectl }}"
61104
env:
62105
- name: KUBECONFIG
63-
value: /tenant-admin/super-admin.svc
106+
value: /etc/kubernetes/kubeconfig/super-admin.svc
64107
- name: TOKEN_ID
65108
valueFrom:
66109
secretKeyRef:
@@ -72,8 +115,8 @@ spec:
72115
name: {{ .Release.Name }}-talos-secrets
73116
key: bootstrapTokenSecret
74117
volumeMounts:
75-
- name: tenant-admin
76-
mountPath: /tenant-admin
118+
- name: kubeconfig
119+
mountPath: /etc/kubernetes/kubeconfig
77120
readOnly: true
78121
resources:
79122
requests:
@@ -168,6 +211,40 @@ spec:
168211
169212
echo "[$(date -u +%H:%M:%S)] talos:node-apiserver-endpoint-discovery RBAC ensured"
170213
volumes:
171-
- name: tenant-admin
214+
- name: kubeconfig
172215
secret:
173216
secretName: {{ .Release.Name }}-admin-kubeconfig
217+
optional: true
218+
{{- end }}
219+
{{- /* Render the spec, then hash it (plus the looked-up bootstrap token, both
220+
halves) into a 6-char name suffix so the immutable Job is churn-free
221+
across upgrades yet still rotates when its image or the bootstrap token
222+
changes. See the header comment. lookup returns empty during `helm
223+
template`/unittest, so the token contribution is stable there and the
224+
render stays testable. */}}
225+
{{- $jobSpec := include "kubernetes.bootstrap-token-tenant-job-spec" . }}
226+
{{- $btId := "" }}
227+
{{- $btSecret := "" }}
228+
{{- $existing := lookup "v1" "Secret" .Release.Namespace (printf "%s-talos-secrets" .Release.Name) }}
229+
{{- if and $existing $existing.data }}
230+
{{- if hasKey $existing.data "bootstrapTokenId" }}
231+
{{- $btId = index $existing.data "bootstrapTokenId" | b64dec }}
232+
{{- end }}
233+
{{- if hasKey $existing.data "bootstrapTokenSecret" }}
234+
{{- $btSecret = index $existing.data "bootstrapTokenSecret" | b64dec }}
235+
{{- end }}
236+
{{- end }}
237+
{{- $jobHash := printf "%s\n%s\n%s" $jobSpec $btId $btSecret | sha256sum | trunc 6 }}
238+
{{- /* Truncate the name prefix, never the hash, so the name stays within the
239+
63-char DNS-label limit while the 6-char content hash is always the
240+
suffix. A no-op for normal-length release names. */}}
241+
{{- $jobName := printf "%s-bootstrap-token-tenant" .Release.Name | trunc 56 | trimSuffix "-" }}
242+
---
243+
apiVersion: batch/v1
244+
kind: Job
245+
metadata:
246+
name: {{ $jobName }}-{{ $jobHash }}
247+
namespace: {{ .Release.Namespace }}
248+
labels:
249+
{{- include "kubernetes.labels" . | nindent 4 }}
250+
{{ $jobSpec }}

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,8 @@
6363
RBAC, ServiceAccount and egress CiliumNetworkPolicy are main-phase too;
6464
the wait loop tolerates its dependencies (and its own network policy)
6565
not yet being in place, so Helm's kind-ordered apply needs no explicit
66-
sequencing. The bootstrap-token Job stays a post-install hook.
66+
sequencing. The bootstrap-token Job is main-phase too, for the same
67+
reasons (see bootstrap-token-tenant-job.yaml).
6768

6869
Churn-free across upgrades. The Job is immutable (spec.template cannot
6970
be patched), so it carries a content-hash name suffix — mirrors the

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 72 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,17 +277,87 @@ tests:
277277
- isKind:
278278
of: ServiceAccount
279279
documentIndex: 0
280+
# The ServiceAccount name is stable (not hashed) — the Job references
281+
# it by this name and only the immutable Job carries the hash suffix.
280282
- equal:
281283
path: metadata.name
282284
value: test-k8s-bootstrap-token-tenant
283285
documentIndex: 0
284286
- isKind:
285287
of: Job
286288
documentIndex: 1
287-
- equal:
289+
# The immutable Job carries a 6-char content-hash suffix (see the
290+
# dedicated content-hash cases below and the template header).
291+
- matchRegex:
288292
path: metadata.name
289-
value: test-k8s-bootstrap-token-tenant
293+
pattern: ^test-k8s-bootstrap-token-tenant-[0-9a-f]{6}$
294+
documentIndex: 1
295+
# The admin-kubeconfig volume must be optional and gated by the shared
296+
# wait-for-kubeconfig init container, so the Pod waits in init (uncounted
297+
# by backoffLimit) instead of FailedMount at t0.
298+
- equal:
299+
path: spec.template.spec.initContainers[0].name
300+
value: wait-for-kubeconfig
290301
documentIndex: 1
302+
- equal:
303+
path: spec.template.spec.volumes[0].secret.optional
304+
value: true
305+
documentIndex: 1
306+
# Main-phase, not a hook: NONE of the bootstrap-token resources may
307+
# carry helm.sh/hook. A post-install hook is not gated on
308+
# MachineDeployment here (the release sets helm-install-disable-wait,
309+
# so there is no main-resource wait in front of the hooks); the reason
310+
# to leave the hook is that a post-UPGRADE hook never runs on a release
311+
# whose upgrade keeps failing, so the token Secret never lands. Pin the
312+
# absence on both documents (0 SA, 1 Job).
313+
- notExists:
314+
path: metadata.annotations["helm.sh/hook"]
315+
documentIndex: 0
316+
- notExists:
317+
path: metadata.annotations["helm.sh/hook"]
318+
documentIndex: 1
319+
320+
# The two cases below pin the bootstrap-token Job's content-hash contract:
321+
# an image change rotates the immutable Job's name (so Helm creates a fresh
322+
# Job instead of failing on an immutable-field patch), while an unchanged
323+
# render keeps the same name (so Helm no-ops). The exact suffix is expected
324+
# to change whenever the Job spec changes; that is the signal that the Job
325+
# will roll, and the fixture is updated in lockstep.
326+
- it: bootstrap-token Job name is derived from the rendered spec
327+
templates:
328+
- templates/talos/bootstrap-token-tenant-job.yaml
329+
release:
330+
name: test-k8s
331+
namespace: tenant-test
332+
asserts:
333+
- equal:
334+
path: metadata.name
335+
value: test-k8s-bootstrap-token-tenant-fc314f
336+
documentSelector:
337+
path: kind
338+
value: Job
339+
340+
- it: bootstrap-token Job name rotates when images.kubectl changes
341+
templates:
342+
- templates/talos/bootstrap-token-tenant-job.yaml
343+
release:
344+
name: test-k8s
345+
namespace: tenant-test
346+
set:
347+
images:
348+
kubectl: example.com/kubectl:v9.9.9
349+
asserts:
350+
# A different exact hash from the default-image render above (fc314f),
351+
# proving the hash tracks the rendered spec (the kubectl image is in it),
352+
# so an image bump lands a fresh Job rather than an immutable-field patch.
353+
# Pinned exactly (like the talos-reconcile sibling cases) rather than by
354+
# regex, so a hash drift on either render is caught, not silently masked.
355+
- equal:
356+
path: metadata.name
357+
value: test-k8s-bootstrap-token-tenant-dbf53d
358+
documentSelector:
359+
path: kind
360+
value: Job
291361

292362
- it: talos-pki emits two cert-manager Certificates and two Issuers
293363
templates:

0 commit comments

Comments
 (0)