Skip to content

Commit baf5500

Browse files
fix(kubernetes): raise kubectl Job memory limits above the OOMKill floor
The talos-reconcile Job's kubectl container was capped at 256Mi and got OOMKilled (exit 137) part-way through its script: it reached the KamajiControlPlane certSANs patch and died before the TalosConfigTemplate apply, so nothing downstream of that point ever ran. Under restartPolicy: OnFailure with backoffLimit: 30 the pod respawned while the Job-level status stayed active, which made the failure invisible apart from a growing restart count. kubectl apply loads full discovery plus the aggregated OpenAPI document of the target apiserver before it can compute a patch, and on a Cozystack management cluster that spans several hundred CRDs. 256Mi is below the working set. Raise the floor to 512Mi and expose it as talos.reconcileJobMemoryLimit for clusters with an even larger API surface. The two sibling kubectl-apply containers in this chart (bootstrap-token-tenant, oidc-bootstrap) get the same floor. Fixes #3519 Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent f1f3836 commit baf5500

10 files changed

Lines changed: 98 additions & 17 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 2 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes/README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -214,13 +214,14 @@ See the reference for components utilized in this service:
214214

215215
### Talos Worker Image
216216

217-
| Name | Description | Type | Value |
218-
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
219-
| `talos` | Talos worker image configuration. | `object` | `{}` |
220-
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
221-
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
222-
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
223-
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
217+
| Name | Description | Type | Value |
218+
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------ |
219+
| `talos` | Talos worker image configuration. | `object` | `{}` |
220+
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
221+
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
222+
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
223+
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
224+
| `talos.reconcileJobMemoryLimit` | Memory limit for the kubectl container of the chart-internal `talos-reconcile` Job. That Job runs `kubectl apply` against the management cluster, which loads full discovery plus the aggregated OpenAPI document (several hundred CRDs on Cozystack) into memory; too low a limit gets the container OOMKilled mid-run, and because it uses `restartPolicy: OnFailure` the Job status stays `active` while the pod silently loops. Raise on management clusters with an unusually large API surface. Changing this value rotates the Job's content-hash name, so the next reconcile creates a fresh Job that re-applies a byte-identical TalosConfigTemplate — a CAPI no-op that does not roll existing workers. | `quantity` | `512Mi` |
224225
225226
226227
### Node Health Check

‎packages/apps/kubernetes/templates/oidc-rbac-job.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,13 +168,20 @@ spec:
168168
value: {{ $issuer | quote }}
169169
- name: OIDC_SERVER_URL
170170
value: {{ printf "https://%s:443" $externalHost | quote }}
171+
{{- /* 512Mi, not 256Mi: this container runs `kubectl apply` (tenant
172+
ClusterRoleBindings) and `kubectl patch` against two apiservers,
173+
and kubectl loads discovery plus the aggregated OpenAPI document
174+
of each into memory before it can compute a patch. Same failure
175+
mode the talos-reconcile Job hit at 256Mi (OOMKill at exit 137,
176+
silently retried under restartPolicy: OnFailure). Request stays
177+
at 64Mi. */}}
171178
resources:
172179
requests:
173180
cpu: 10m
174181
memory: 64Mi
175182
limits:
176183
cpu: 200m
177-
memory: 256Mi
184+
memory: 512Mi
178185
command:
179186
- /bin/sh
180187
- -c

‎packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,13 +75,22 @@ spec:
7575
- name: tenant-admin
7676
mountPath: /tenant-admin
7777
readOnly: true
78+
{{- /* 512Mi, not 256Mi: this container shells out to `kubectl apply`,
79+
which loads discovery plus the aggregated OpenAPI document of
80+
the target apiserver into memory before it can compute a patch.
81+
The tenant apiserver starts small but grows a CRD at a time as
82+
addons land (cert-manager, cilium, flux, velero), and this Job
83+
re-runs on every upgrade — so the footprint here tracks tenant
84+
growth. Same failure mode the talos-reconcile Job hit at 256Mi
85+
(OOMKill at exit 137, silently retried under
86+
restartPolicy: OnFailure). Request stays at 64Mi. */}}
7887
resources:
7988
requests:
8089
cpu: 10m
8190
memory: 64Mi
8291
limits:
8392
cpu: 100m
84-
memory: 256Mi
93+
memory: 512Mi
8594
securityContext:
8695
allowPrivilegeEscalation: false
8796
capabilities:

‎packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -447,13 +447,31 @@ spec:
447447
disabled: true
448448
EOF
449449
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} applied"
450+
{{- /* Memory: `kubectl apply` builds an in-memory model of the target
451+
API surface before it can compute a patch — full discovery plus
452+
the aggregated OpenAPI document, which on a Cozystack management
453+
cluster spans several hundred CRDs. At the former 256Mi limit
454+
this container was OOMKilled (exit 137) part-way through the
455+
script: it got as far as the KamajiControlPlane certSANs patch
456+
and died before the TalosConfigTemplate apply, so nothing
457+
downstream of that point ever ran. And the failure was silent —
458+
restartPolicy: OnFailure with backoffLimit: 30 keeps respawning
459+
the container while the Job-level status stays `active`, so the
460+
only symptom is a pod with a growing restart count. The request
461+
stays at 64Mi (the container spends most of its life idle in the
462+
input-wait loop above, and the limit is only touched during the
463+
apply), so the headroom costs nothing in scheduling terms; it
464+
does count against a tenant ResourceQuota on limits.memory, but
465+
only for the tens of seconds the Job lives. Operators whose
466+
management cluster carries an even larger API surface raise
467+
talos.reconcileJobMemoryLimit. */}}
450468
resources:
451469
requests:
452470
cpu: 10m
453471
memory: 64Mi
454472
limits:
455473
cpu: 100m
456-
memory: 256Mi
474+
memory: {{ $.Values.talos.reconcileJobMemoryLimit | default "512Mi" | quote }}
457475
securityContext:
458476
allowPrivilegeEscalation: false
459477
capabilities:

‎packages/apps/kubernetes/tests/talos_templates_test.yaml‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,32 @@ tests:
118118
path: spec.template.spec.containers[0].command[2]
119119
pattern: 'extraHostEntries:'
120120
documentIndex: 6
121+
# Memory floor. `kubectl apply` against the management cluster's API
122+
# surface (several hundred CRDs, large aggregated OpenAPI document) does
123+
# not fit in 256Mi: the container was OOMKilled mid-script and, under
124+
# restartPolicy: OnFailure with the Job status stuck on `active`, retried
125+
# invisibly forever. Pin the default so a future edit cannot silently
126+
# lower it back below the floor.
127+
- equal:
128+
path: spec.template.spec.containers[0].resources.limits.memory
129+
value: 512Mi
130+
documentIndex: 6
131+
132+
- it: talos-reconcile Job memory limit is operator-overridable
133+
templates:
134+
- templates/talos/talos-reconcile-job.yaml
135+
release:
136+
name: test-k8s
137+
namespace: tenant-test
138+
set:
139+
talos.reconcileJobMemoryLimit: 1Gi
140+
asserts:
141+
- equal:
142+
path: spec.template.spec.containers[0].resources.limits.memory
143+
value: 1Gi
144+
documentSelector:
145+
path: kind
146+
value: Job
121147

122148
# The two cases below pin the content-hash contract: a TCT-affecting change
123149
# rotates the immutable Job's name (so Helm creates a fresh Job instead of
@@ -156,7 +182,7 @@ tests:
156182
asserts:
157183
- equal:
158184
path: metadata.name
159-
value: test-k8s-talos-reconcile-md0-16c1c9
185+
value: test-k8s-talos-reconcile-md0-47e921
160186
documentSelector:
161187
path: kind
162188
value: Job
@@ -189,12 +215,12 @@ tests:
189215
gpus:
190216
- name: nvidia.com/GH200
191217
asserts:
192-
# Only gpus changed vs the case above, yet the suffix differs (a9dc0c vs
193-
# 16c1c9) — the content hash tracks the rendered spec, and the TCT the Job
218+
# Only gpus changed vs the case above, yet the suffix differs (e39ca2 vs
219+
# 47e921) — the content hash tracks the rendered spec, and the TCT the Job
194220
# applies is what changed (GPU nodeLabels).
195221
- equal:
196222
path: metadata.name
197-
value: test-k8s-talos-reconcile-md0-a9dc0c
223+
value: test-k8s-talos-reconcile-md0-e39ca2
198224
documentSelector:
199225
path: kind
200226
value: Job

‎packages/apps/kubernetes/values.schema.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -954,6 +954,20 @@
954954
"type": "string",
955955
"default": "factory.talos.dev/installer"
956956
},
957+
"reconcileJobMemoryLimit": {
958+
"description": "Memory limit for the kubectl container of the chart-internal `talos-reconcile` Job. That Job runs `kubectl apply` against the management cluster, which loads full discovery plus the aggregated OpenAPI document (several hundred CRDs on Cozystack) into memory; too low a limit gets the container OOMKilled mid-run, and because it uses `restartPolicy: OnFailure` the Job status stays `active` while the pod silently loops. Raise on management clusters with an unusually large API surface. Changing this value rotates the Job's content-hash name, so the next reconcile creates a fresh Job that re-applies a byte-identical TalosConfigTemplate — a CAPI no-op that does not roll existing workers.",
959+
"default": "512Mi",
960+
"pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$",
961+
"anyOf": [
962+
{
963+
"type": "integer"
964+
},
965+
{
966+
"type": "string"
967+
}
968+
],
969+
"x-kubernetes-int-or-string": true
970+
},
957971
"schematicID": {
958972
"description": "Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.",
959973
"type": "string",

0 commit comments

Comments
 (0)