Skip to content

Commit 6ee7e20

Browse files
fix(kubernetes): raise kubectl Job memory limits above the OOMKill floor
The chart's kubectl-apply containers were capped at 256Mi and got OOMKilled (exit 137) part-way through their scripts. Under restartPolicy: OnFailure the pod respawned while the Job-level status stayed active, which made the failure invisible apart from a growing restart count. kubectl apply loads full discovery plus the aggregated OpenAPI document of the target apiserver before it can compute a patch, and on a Cozystack management cluster that spans several hundred CRDs. 256Mi is below the working set. Raise the floor to 512Mi on bootstrap-token-tenant and oidc-bootstrap. The talos-reconcile Job, where this was first observed, no longer exists in the chart: it was removed with the worker node pools, which also retires the talos.reconcileJobMemoryLimit knob this change first carried. The bootstrap-token Job is named by a content hash of its rendered spec, so the new limit rotates the name and the pinned fixtures move with it. Fixes #3519 Signed-off-by: Mattia Eleuteri <[email protected]> Assisted-by: LLM
1 parent a1b32f2 commit 6ee7e20

3 files changed

Lines changed: 23 additions & 7 deletions

File tree

‎packages/apps/kubernetes/templates/oidc-rbac-job.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,13 +168,20 @@ spec:
168168
value: {{ $issuer | quote }}
169169
- name: OIDC_SERVER_URL
170170
value: {{ printf "https://%s:443" $externalHost | quote }}
171+
{{- /* 512Mi, not 256Mi: this container runs `kubectl apply` (tenant
172+
ClusterRoleBindings) and `kubectl patch` against two apiservers,
173+
and kubectl loads discovery plus the aggregated OpenAPI document
174+
of each into memory before it can compute a patch. At 256Mi it
175+
is OOMKilled at exit 137 and, under restartPolicy: OnFailure,
176+
retries invisibly while the Job status stays `active`. Request
177+
stays at 64Mi. */}}
171178
resources:
172179
requests:
173180
cpu: 10m
174181
memory: 64Mi
175182
limits:
176183
cpu: 200m
177-
memory: 256Mi
184+
memory: 512Mi
178185
command:
179186
- /bin/sh
180187
- -c

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,13 +118,22 @@ spec:
118118
- name: kubeconfig
119119
mountPath: /etc/kubernetes/kubeconfig
120120
readOnly: true
121+
{{- /* 512Mi, not 256Mi: this container shells out to `kubectl apply`,
122+
which loads discovery plus the aggregated OpenAPI document of
123+
the target apiserver into memory before it can compute a patch.
124+
The tenant apiserver starts small but grows a CRD at a time as
125+
addons land (cert-manager, cilium, flux, velero), and this Job
126+
re-runs on every upgrade — so the footprint here tracks tenant
127+
growth. At 256Mi it is OOMKilled at exit 137 and, under
128+
restartPolicy: OnFailure, retries invisibly while the Job
129+
status stays `active`. Request stays at 64Mi. */}}
121130
resources:
122131
requests:
123132
cpu: 10m
124133
memory: 64Mi
125134
limits:
126135
cpu: 100m
127-
memory: 256Mi
136+
memory: 512Mi
128137
securityContext:
129138
allowPrivilegeEscalation: false
130139
capabilities:

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ tests:
8383
asserts:
8484
- equal:
8585
path: metadata.name
86-
value: test-k8s-bootstrap-token-tenant-fc314f
86+
value: test-k8s-bootstrap-token-tenant-24544a
8787
documentSelector:
8888
path: kind
8989
value: Job
@@ -98,14 +98,14 @@ tests:
9898
images:
9999
kubectl: example.com/kubectl:v9.9.9
100100
asserts:
101-
# A different exact hash from the default-image render above (fc314f),
101+
# A different exact hash from the default-image render above (24544a),
102102
# proving the hash tracks the rendered spec (the kubectl image is in it),
103103
# so an image bump lands a fresh Job rather than an immutable-field patch.
104-
# Pinned exactly (like the talos-reconcile sibling cases) rather than by
105-
# regex, so a hash drift on either render is caught, not silently masked.
104+
# Pinned exactly rather than by regex, so a hash drift on either render
105+
# is caught, not silently masked.
106106
- equal:
107107
path: metadata.name
108-
value: test-k8s-bootstrap-token-tenant-dbf53d
108+
value: test-k8s-bootstrap-token-tenant-45dfa1
109109
documentSelector:
110110
path: kind
111111
value: Job

0 commit comments

Comments
 (0)