Skip to content

Commit 1728b49

Browse files
mattia-eleuterilexfrei
authored andcommitted
fix(kubernetes): apply worker TalosConfigTemplate server-side
The talos-reconcile Job runs under a 256Mi limit and applies its TalosConfigTemplate client-side. Building the merge patch means pulling the cluster's full OpenAPI schema down, which on a management cluster carrying a few hundred CRDs costs around 290Mi of RSS, so the kernel OOMKills kubectl before the Job can finish. Only the merge path pays that cost. The first run creates the object and stays cheap, so a fresh install succeeds and the Job then CrashLoopBackOffs on every reconcile afterwards, which means on every chart upgrade. Measured against a 438-CRD cluster with the Job's own kubectl (alpine/k8s:1.33.4), cold discovery cache: get -o jsonpath 52Mi apply, object absent (create) 52Mi apply, object present (merge) 292Mi <- OOMKilled at 256Mi apply --server-side 56Mi --validate=false changes nothing, so the cost is the merge strategy lookup rather than validation. Server-side apply leaves the merge to the apiserver and the client never needs the schema. --force-conflicts keeps the Job authoritative over an object an earlier client-side apply created. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 717d091 commit 1728b49

2 files changed

Lines changed: 56 additions & 1 deletion

File tree

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,15 @@ spec:
253253
# Apply the TalosConfigTemplate. ownerReference back to the parent
254254
# KamajiControlPlane makes the standard Kubernetes garbage collector
255255
# clean it up when the parent cluster is deleted.
256-
cat <<EOF | kubectl apply -f -
256+
#
257+
# Server-side apply keeps the merge in the apiserver. A client-side
258+
# apply pulls the full OpenAPI schema down to build the merge patch,
259+
# ~290Mi RSS against a management cluster carrying a few hundred
260+
# CRDs, over this container's 256Mi limit. Only the merge path pays
261+
# that, so a create survives where an update is OOMKilled.
262+
# --force-conflicts keeps this Job authoritative over an object an
263+
# earlier client-side apply created.
264+
cat <<EOF | kubectl apply --server-side --force-conflicts -f -
257265
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
258266
kind: TalosConfigTemplate
259267
metadata:
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
suite: worker TalosConfigTemplate is applied server-side
2+
3+
# The reconcile Job runs under a 256Mi limit. A client-side apply builds its merge
4+
# patch from the cluster's full OpenAPI schema, which costs ~290Mi RSS against a
5+
# management cluster carrying a few hundred CRDs, so the Job is OOMKilled on every
6+
# reconcile that updates an existing TalosConfigTemplate -- that is, on every chart
7+
# upgrade, while the create on first install stays cheap and hides the fault.
8+
9+
templates:
10+
- templates/talos-reconcile-job.yaml
11+
release:
12+
name: kubernetes-nodes-myk8s-md0
13+
namespace: tenant-test
14+
set:
15+
cluster: myk8s
16+
_cluster:
17+
cluster-domain: cozy.local
18+
version: "v1.35"
19+
minReplicas: 0
20+
maxReplicas: 3
21+
instanceType: ""
22+
diskSize: 20Gi
23+
storageClass: replicated
24+
roles:
25+
- ingress-nginx
26+
resources:
27+
cpu: "2"
28+
memory: 4Gi
29+
30+
tests:
31+
- it: applies the TalosConfigTemplate server-side
32+
asserts:
33+
- matchRegex:
34+
path: spec.template.spec.containers[0].command[2]
35+
pattern: 'kubectl apply --server-side --force-conflicts -f -'
36+
documentSelector:
37+
path: kind
38+
value: Job
39+
40+
- it: never falls back to a client-side apply
41+
asserts:
42+
- notMatchRegex:
43+
path: spec.template.spec.containers[0].command[2]
44+
pattern: 'kubectl apply -f -'
45+
documentSelector:
46+
path: kind
47+
value: Job

0 commit comments

Comments
 (0)