Skip to content

Commit 2f8c953

Browse files
fix(kubernetes): render the talos-reconcile Job for the default md0 group (#3535)
## What this PR does Fixes #3504. The chart has two loops over worker node groups and they read different sources. `templates/cluster.yaml` iterates the effective set through the `kubernetes.nodeGroups` helper, whose else-branch supplies the built-in `md0` group when the user declares none — that is how #2936 made `md0` removable without a Helm values merge re-adding it. `templates/talos/talos-reconcile-job.yaml` iterated the raw `.Values.nodeGroups` map instead, which is empty in exactly that case. The consequence is a dangling reference. The chart cannot render the `TalosConfigTemplate` itself — it needs the apiserver Service ClusterIP, the Talos CA and the Kubernetes CA, none of which exist while Helm is executing the template — so `cluster.yaml` deliberately stopped rendering it and the Job became its only producer, while the `MachineDeployment` still names it in `spec.template.spec.bootstrap.configRef`. On a cluster with the default `nodeGroups`, `MachineDeployment/<release>-md0` renders and zero Jobs render, so `TalosConfigTemplate/<release>-md0` is never created and CAPI blocks every Machine that ever joins the group. The same Job also patches `KamajiControlPlane.spec.network.certSANs` with the live Service ClusterIP, so that is skipped too. Nothing fails at install time, because the built-in group carries `minReplicas: 0`. The failure waits for the first scale-up, which is the documented path rather than a corner: `values.yaml` tells operators that enabling the ingress-nginx addon on a cluster with default `nodeGroups` means waiting for the cluster-autoscaler to bring `md0` up in response to the controller Pods becoming Pending, and the autoscaler is deployed unconditionally whenever the tenant has an etcd DataStore. The fix is to range over the helper so the Job set tracks the MachineDeployment set exactly. A cluster that declares its own groups renders byte-identically — which is why the pinned content-hash fixtures in `tests/talos_templates_test.yaml` are untouched — and `md0` stays removable, because the fix defers to the helper rather than merging `md0` in unconditionally. On upgrade it is additive: the Job appears where there was none, applies the `TalosConfigTemplate` the MachineDeployment already expects, and the content-hash name suffix means Helm creates a fresh Job rather than attempting to patch an immutable one. ### How this survived Every helm-unittest fixture and every e2e suite declares `md0` explicitly. `tests/nodegroups_default_test.yaml` does render the empty-`nodeGroups` case, but lists only `templates/cluster.yaml`, so it never looked at a Job. The two OIDC chainsaw lanes do install with an empty map, but assert that the HelmRelease exists rather than that it becomes Ready, so a worker that never boots is invisible to them. Offline `helm template` cannot reach this path at all: `cluster.yaml`'s instanceType validator needs a live `lookup` for `u1.medium`. `tests/talos_reconcile_nodegroups_test.yaml` closes the gap and pins both halves of the helper contract — an empty map yields exactly one `md0` Job carrying `GROUP_NAME=md0`, and a two-group map yields exactly two Jobs with no `md0` among them, so a future fix that merges `md0` in unconditionally fails here. Reverting the template change fails the first case on the document count, so it guards the behaviour rather than restating it. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff: it is one Helm template loop and one new helm-unittest suite inside `packages/apps/kubernetes`. No package is added, renamed or removed; no `values.yaml`, `values.schema.json`, `ApplicationDefinition`, version enum or default changes; nothing under `hack/`, no namespace, variant, label, annotation or metric renamed. The behaviour now matches what `values.yaml` already documents, so no reference page drifts. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): a tenant Kubernetes cluster created without an explicit `nodeGroups` map now gets a talos-reconcile Job for the built-in `md0` node group. Previously that Job rendered only for explicitly declared groups, so the default `md0` MachineDeployment referenced a TalosConfigTemplate nothing ever created: every worker the cluster-autoscaler added to `md0` stayed stuck waiting for its bootstrap config, and the KamajiControlPlane certSANs patch the same Job performs never ran. Clusters that declare their own node groups were unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Ensured the built-in `md0` node group receives its Talos reconciliation Job when no node groups are configured. * Prevented duplicate reconciliation Jobs when custom node groups are specified. * Preserved correct group-specific settings and control-plane configuration in generated Jobs. * **Tests** * Added regression coverage for default and custom node-group scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3f967ec + 4e9c5ce commit 2f8c953

2 files changed

Lines changed: 139 additions & 1 deletion

File tree

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -485,7 +485,14 @@ spec:
485485
*management* cluster-domain, sourced from .Values._cluster
486486
(populated by cozystack-controller from the platform config). */}}
487487
{{- $mgmtClusterDomain := (index .Values._cluster "cluster-domain") | default "cozy.local" }}
488-
{{- range $groupName, $group := .Values.nodeGroups }}
488+
{{- /* Iterate the EFFECTIVE node groups (the kubernetes.nodeGroups helper),
489+
never the raw .Values.nodeGroups map. The built-in md0 group lives in
490+
the helper's else-branch, so a cluster that supplies no nodeGroups has
491+
an empty raw map: ranging over it renders zero Jobs, leaving the md0
492+
MachineDeployment below pointing at a TalosConfigTemplate that only
493+
this Job ever creates. CAPI then blocks every Machine the autoscaler
494+
adds. Must stay in lockstep with the loop in cluster.yaml. */}}
495+
{{- range $groupName, $group := (include "kubernetes.nodeGroups" $ | fromYaml) }}
489496
{{- /* Mirror the kubelet-reservation computation from cluster.yaml so the
490497
worker machineconfig the Job applies carries the same numbers the
491498
MachineDeployment validator was happy with. Auto-computed
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
suite: talos-reconcile Job covers every effective node group
2+
3+
# Regression guard for cozystack/cozystack#3504. The Job in
4+
# templates/talos/talos-reconcile-job.yaml is the only producer of the
5+
# TalosConfigTemplate that each worker MachineDeployment names in
6+
# `spec.template.spec.bootstrap.configRef` (cluster.yaml renders the reference
7+
# but not the object). Its loop must therefore iterate the same effective group
8+
# set as the MachineDeployment loop — the kubernetes.nodeGroups helper — not the
9+
# raw .Values.nodeGroups map. When the raw map was read, a cluster that supplied
10+
# no nodeGroups got the helper's built-in md0 MachineDeployment and zero Jobs, so
11+
# every Machine the autoscaler added to md0 blocked forever on a
12+
# TalosConfigTemplate nothing would ever create, and the KamajiControlPlane
13+
# certSANs patch the same Job performs never ran either.
14+
#
15+
# The two cases below pin both halves of the helper contract: the default group
16+
# gets a Job (the bug), and a user-supplied group set stays authoritative — md0
17+
# must NOT reappear alongside it, which is the removability #2936 introduced.
18+
#
19+
# Suite-level `templates` is load-bearing: it limits the render to this one
20+
# template. Declaring `templates` per test does not filter the render, and the
21+
# unrelated `lookup` in templates/helmreleases/csi.yaml then fails the suite.
22+
23+
templates:
24+
- templates/talos/talos-reconcile-job.yaml
25+
26+
tests:
27+
- it: renders a Job for the built-in md0 group when nodeGroups is empty
28+
release:
29+
name: test-k8s
30+
namespace: tenant-test
31+
set:
32+
_namespace:
33+
etcd: etcd
34+
ingress: nginx
35+
host: example.com
36+
_cluster:
37+
cluster-domain: cozy.local
38+
# nodeGroups intentionally left at the chart default ({}) so the helper
39+
# emits the built-in md0 — the case that rendered no Job at all.
40+
asserts:
41+
# 6 scaffolding documents (ServiceAccount, Role, CiliumNetworkPolicy,
42+
# RoleBinding, ClusterRole, ClusterRoleBinding) plus one Job for md0.
43+
# Before the fix this was 6: the Job was missing entirely.
44+
- hasDocuments:
45+
count: 7
46+
- isKind:
47+
of: Job
48+
documentIndex: 6
49+
- matchRegex:
50+
path: metadata.name
51+
pattern: ^test-k8s-talos-reconcile-md0-[0-9a-f]{6}$
52+
documentIndex: 6
53+
# The Job applies TalosConfigTemplate ${RELEASE}-${GROUP_NAME}, so
54+
# GROUP_NAME is what ties it to the MachineDeployment's configRef name
55+
# (test-k8s-md0). A Job for the wrong group name would satisfy the
56+
# document count above while leaving the MachineDeployment stuck.
57+
- contains:
58+
path: spec.template.spec.containers[0].env
59+
content:
60+
name: GROUP_NAME
61+
value: md0
62+
documentIndex: 6
63+
- matchRegex:
64+
path: spec.template.spec.containers[0].command[2]
65+
pattern: 'kind: TalosConfigTemplate'
66+
documentIndex: 6
67+
# Secondary effect of the missing Job: the KamajiControlPlane certSANs
68+
# patch that adds the live apiserver Service ClusterIP never ran.
69+
- matchRegex:
70+
path: spec.template.spec.containers[0].command[2]
71+
pattern: 'patch kamajicontrolplane'
72+
documentIndex: 6
73+
74+
- it: user-supplied node groups stay authoritative — one Job each, no md0
75+
release:
76+
name: test-k8s
77+
namespace: tenant-test
78+
set:
79+
_namespace:
80+
etcd: etcd
81+
ingress: nginx
82+
host: example.com
83+
_cluster:
84+
cluster-domain: cozy.local
85+
nodeGroups:
86+
worker0:
87+
minReplicas: 1
88+
maxReplicas: 3
89+
instanceType: u1.medium
90+
diskSize: 20Gi
91+
storageClass: ""
92+
roles:
93+
- ingress-nginx
94+
resources: {}
95+
gpus: []
96+
kubelet: {}
97+
worker1:
98+
minReplicas: 0
99+
maxReplicas: 5
100+
instanceType: u1.medium
101+
diskSize: 20Gi
102+
storageClass: ""
103+
roles: []
104+
resources: {}
105+
gpus: []
106+
kubelet: {}
107+
asserts:
108+
# 6 scaffolding documents + exactly two Jobs. A fix that unconditionally
109+
# merged md0 into the loop would render three and fail here.
110+
- hasDocuments:
111+
count: 8
112+
- matchRegex:
113+
path: metadata.name
114+
pattern: ^test-k8s-talos-reconcile-worker0-[0-9a-f]{6}$
115+
documentIndex: 6
116+
- matchRegex:
117+
path: metadata.name
118+
pattern: ^test-k8s-talos-reconcile-worker1-[0-9a-f]{6}$
119+
documentIndex: 7
120+
- contains:
121+
path: spec.template.spec.containers[0].env
122+
content:
123+
name: GROUP_NAME
124+
value: worker0
125+
documentIndex: 6
126+
- contains:
127+
path: spec.template.spec.containers[0].env
128+
content:
129+
name: GROUP_NAME
130+
value: worker1
131+
documentIndex: 7

0 commit comments

Comments
 (0)