Repository navigation
Commit d7f2799
authored
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#3887)
## What this PR does
Break the tenant `bootstrap-token-tenant` Job out of its `helm.sh/hook:
post-install,post-upgrade` lifecycle and move it into the main install
phase, so the `bootstrap-token-<id>` Secret lands in the tenant
`kube-system` before `MachineDeployment` can converge.
### Why it deadlocks today
- Job wired as `post-install,post-upgrade` fires only after Helm's wait
phase resolves
- Wait phase blocks on `MachineDeployment` Ready
- MD Ready blocks on nodes registering
- Node registration blocks on the kubelet CSR path, which needs the very
`bootstrap-token-<id>` Secret this Job produces
- Result: any fresh install, or any HR reconcile that finds MD in a
non-Ready state, sits in a deadlock; `HR.status.upgradeFailures` climbs
without the post-upgrade hook ever getting its turn
`pre-install,pre-upgrade` is not a fix either — those fire before
non-hook templates apply, so `<release>-admin-kubeconfig` (materialised
by Kamaji after `KamajiControlPlane` is applied) is not yet available.
### The shape that works
Same shape `talos-reconcile` already uses in this chart: main-phase
resources. Helm applies the SA and Job together with
`KamajiControlPlane` and `talos-secrets`. Job's Pod stays pending until
Kamaji publishes the admin kubeconfig Secret (`backoffLimit` covers the
retry window). Job completes before MachineDeployment can converge.
Helm's wait treats the Job as another resource to reach terminal state —
it doesn't gate it on MD.
### Verification
- Reproduced the deadlock on two independent long-running tenant
clusters (v1.6.x), documented in #3875
- Reproduced the manual unblock (creating the `bootstrap-token-<id>`
Secret by hand in the tenant `kube-system`) on both — nodes register
within seconds, MD converges within minutes, HR clears
- Extended `packages/apps/kubernetes/tests/talos_templates_test.yaml` to
pin the absence of `helm.sh/hook` on both rendered documents; `helm
unittest` passes 9/9
### Screenshots
Not a UI change.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md`. The diff
touches
`packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml`
and its test — no app add/remove, no `values.schema.json`, no
core/platform values, no `hack/`, no CRD, no `ApplicationDefinition`, no
telemetry metric.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(kubernetes): tenant `bootstrap-token-<id>` Secret is now created in the main install phase instead of a post-install/post-upgrade hook, unblocking fresh installs and any HR reconcile that finds MachineDeployment in a non-Ready state.
```
Closes #3875
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved bootstrap-token job scheduling by running it as a standard
chart resource instead of a Helm lifecycle hook.
* Jobs are automatically recreated when relevant configuration changes,
ensuring updates are applied reliably.
* Added startup handling to wait for the admin kubeconfig when needed.
* Improved retry and cleanup behavior so deployments are not
unnecessarily blocked.
* Ensured related resources are created consistently during chart
deployment.
* Improved job naming stability while respecting Kubernetes naming
limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->3 files changed
Lines changed: 182 additions & 34 deletions
File tree
- packages/apps/kubernetes
- templates/talos
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
15 | 60 | | |
16 | 61 | | |
17 | 62 | | |
| |||
21 | 66 | | |
22 | 67 | | |
23 | 68 | | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
40 | 74 | | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
48 | 86 | | |
49 | 87 | | |
50 | 88 | | |
| |||
55 | 93 | | |
56 | 94 | | |
57 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
58 | 101 | | |
59 | 102 | | |
60 | 103 | | |
61 | 104 | | |
62 | 105 | | |
63 | | - | |
| 106 | + | |
64 | 107 | | |
65 | 108 | | |
66 | 109 | | |
| |||
72 | 115 | | |
73 | 116 | | |
74 | 117 | | |
75 | | - | |
76 | | - | |
| 118 | + | |
| 119 | + | |
77 | 120 | | |
78 | 121 | | |
79 | 122 | | |
| |||
168 | 211 | | |
169 | 212 | | |
170 | 213 | | |
171 | | - | |
| 214 | + | |
172 | 215 | | |
173 | 216 | | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
66 | | - | |
| 66 | + | |
| 67 | + | |
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
351 | 351 | | |
352 | 352 | | |
353 | 353 | | |
| 354 | + | |
| 355 | + | |
354 | 356 | | |
355 | 357 | | |
356 | 358 | | |
357 | 359 | | |
358 | 360 | | |
359 | 361 | | |
360 | 362 | | |
361 | | - | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
362 | 366 | | |
363 | | - | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
364 | 375 | | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
365 | 435 | | |
366 | 436 | | |
367 | 437 | | |
| |||
0 commit comments