Skip to content

Commit 479d966

Browse files
fix(kubernetes): raise kubectl Job memory limits above the OOMKill floor
The chart's kubectl-apply containers were capped at 256Mi and got OOMKilled (exit 137) part-way through their scripts. Under restartPolicy: OnFailure the pod respawned while the Job-level status stayed active, which made the failure invisible apart from a growing restart count. kubectl apply loads full discovery plus the aggregated OpenAPI document of the target apiserver before it can compute a patch, and on a Cozystack management cluster that spans several hundred CRDs. 256Mi is below the working set. Raise the floor to 512Mi on the three affected containers: the talos-reconcile Job in kubernetes-nodes, which is the one #3519 reports, plus bootstrap-token-tenant and oidc-bootstrap in the parent chart. The talos-reconcile Job moved to the kubernetes-nodes chart with the worker node pools after this branch opened, and the move carried the 256Mi limit with it, so the reported failure is still reachable on main. The knob this change first carried, talos.reconcileJobMemoryLimit, is not reintroduced: 512Mi is the floor, and a value that has to be declared in two charts to cover one Job is worth adding only once a deployment is shown to need it. The bootstrap-token Job is named by a content hash of its rendered spec, so the new limit rotates the name and the pinned fixtures move with it. Fixes #3519 Signed-off-by: Mattia Eleuteri <[email protected]> Assisted-by: LLM
1 parent a1b32f2 commit 479d966

5 files changed

Lines changed: 86 additions & 8 deletions

File tree

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -365,13 +365,22 @@ spec:
365365
disabled: true
366366
EOF
367367
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} applied"
368+
{{- /* 512Mi, not 256Mi: this container runs `kubectl apply` against the
369+
management cluster, which loads discovery plus the aggregated
370+
OpenAPI document (several hundred CRDs on Cozystack) into memory
371+
before it can compute a patch. At 256Mi it is OOMKilled at exit
372+
137 part-way through the script, after the KamajiControlPlane
373+
certSANs patch and before the TalosConfigTemplate apply, and
374+
under restartPolicy: OnFailure with backoffLimit: 30 it retries
375+
invisibly while the Job status stays `active`
376+
(cozystack/cozystack#3519). Request stays at 64Mi. */}}
368377
resources:
369378
requests:
370379
cpu: 10m
371380
memory: 64Mi
372381
limits:
373382
cpu: 100m
374-
memory: 256Mi
383+
memory: 512Mi
375384
securityContext:
376385
allowPrivilegeEscalation: false
377386
capabilities:
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
suite: talos-reconcile Job memory floor
2+
3+
# The kubectl container of this Job runs `kubectl apply` against the MANAGEMENT
4+
# cluster, not the tenant one, so it loads that apiserver's full discovery plus
5+
# its aggregated OpenAPI document before it can compute a patch — several hundred
6+
# CRDs on Cozystack. At 256Mi it was OOMKilled at exit 137 part-way through the
7+
# script, after the KamajiControlPlane certSANs patch and before the
8+
# TalosConfigTemplate apply, so machine-config changes never propagated. The
9+
# failure was silent: restartPolicy OnFailure with backoffLimit 30 respawns the
10+
# container while the Job-level status stays `active`, so the only symptom is a
11+
# growing restart count. See cozystack/cozystack#3519.
12+
#
13+
# Pinned so a later edit cannot quietly lower it back under the floor. The
14+
# request is pinned with it: raising the limit alone would be free in scheduling
15+
# terms, and a future change that lifts the request to match would silently make
16+
# every pool's Job harder to schedule.
17+
18+
templates:
19+
- templates/talos-reconcile-job.yaml
20+
release:
21+
name: kubernetes-nodes-myk8s-md0
22+
namespace: tenant-test
23+
set:
24+
cluster: myk8s
25+
_cluster:
26+
cluster-domain: cozy.local
27+
version: "v1.35"
28+
minReplicas: 0
29+
maxReplicas: 3
30+
instanceType: ""
31+
diskSize: 20Gi
32+
storageClass: replicated
33+
roles:
34+
- ingress-nginx
35+
resources:
36+
cpu: "2"
37+
memory: 4Gi
38+
39+
tests:
40+
- it: pins the kubectl container above the OOMKill floor
41+
asserts:
42+
- equal:
43+
path: spec.template.spec.containers[0].resources.limits.memory
44+
value: 512Mi
45+
documentSelector:
46+
path: kind
47+
value: Job
48+
- equal:
49+
path: spec.template.spec.containers[0].resources.requests.memory
50+
value: 64Mi
51+
documentSelector:
52+
path: kind
53+
value: Job

‎packages/apps/kubernetes/templates/oidc-rbac-job.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,13 +168,20 @@ spec:
168168
value: {{ $issuer | quote }}
169169
- name: OIDC_SERVER_URL
170170
value: {{ printf "https://%s:443" $externalHost | quote }}
171+
{{- /* 512Mi, not 256Mi: this container runs `kubectl apply` (tenant
172+
ClusterRoleBindings) and `kubectl patch` against two apiservers,
173+
and kubectl loads discovery plus the aggregated OpenAPI document
174+
of each into memory before it can compute a patch. At 256Mi it
175+
is OOMKilled at exit 137 and, under restartPolicy: OnFailure,
176+
retries invisibly while the Job status stays `active`. Request
177+
stays at 64Mi. */}}
171178
resources:
172179
requests:
173180
cpu: 10m
174181
memory: 64Mi
175182
limits:
176183
cpu: 200m
177-
memory: 256Mi
184+
memory: 512Mi
178185
command:
179186
- /bin/sh
180187
- -c

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,13 +118,22 @@ spec:
118118
- name: kubeconfig
119119
mountPath: /etc/kubernetes/kubeconfig
120120
readOnly: true
121+
{{- /* 512Mi, not 256Mi: this container shells out to `kubectl apply`,
122+
which loads discovery plus the aggregated OpenAPI document of
123+
the target apiserver into memory before it can compute a patch.
124+
The tenant apiserver starts small but grows a CRD at a time as
125+
addons land (cert-manager, cilium, flux, velero), and this Job
126+
re-runs on every upgrade — so the footprint here tracks tenant
127+
growth. At 256Mi it is OOMKilled at exit 137 and, under
128+
restartPolicy: OnFailure, retries invisibly while the Job
129+
status stays `active`. Request stays at 64Mi. */}}
121130
resources:
122131
requests:
123132
cpu: 10m
124133
memory: 64Mi
125134
limits:
126135
cpu: 100m
127-
memory: 256Mi
136+
memory: 512Mi
128137
securityContext:
129138
allowPrivilegeEscalation: false
130139
capabilities:

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ tests:
8383
asserts:
8484
- equal:
8585
path: metadata.name
86-
value: test-k8s-bootstrap-token-tenant-fc314f
86+
value: test-k8s-bootstrap-token-tenant-24544a
8787
documentSelector:
8888
path: kind
8989
value: Job
@@ -98,14 +98,14 @@ tests:
9898
images:
9999
kubectl: example.com/kubectl:v9.9.9
100100
asserts:
101-
# A different exact hash from the default-image render above (fc314f),
101+
# A different exact hash from the default-image render above (24544a),
102102
# proving the hash tracks the rendered spec (the kubectl image is in it),
103103
# so an image bump lands a fresh Job rather than an immutable-field patch.
104-
# Pinned exactly (like the talos-reconcile sibling cases) rather than by
105-
# regex, so a hash drift on either render is caught, not silently masked.
104+
# Pinned exactly rather than by regex, so a hash drift on either render
105+
# is caught, not silently masked.
106106
- equal:
107107
path: metadata.name
108-
value: test-k8s-bootstrap-token-tenant-dbf53d
108+
value: test-k8s-bootstrap-token-tenant-45dfa1
109109
documentSelector:
110110
path: kind
111111
value: Job

0 commit comments

Comments
 (0)