Skip to content

Commit 20545d0

Browse files
yankawaiyankawai
committed
fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits
Without a global section HAProxy 3.4 sizes itself from the node. maxconn comes from the nofile limit, cut to the built-in fd-hard-limit of 1048576, so about 524k, and the thread count is the number of CPUs the process may run on, whatever the CPU limit says. With one thread that alone idles at about 85 MiB, and every thread adds about 0.6 MiB, so a t1.nano replica with its 128Mi limit runs on a small node and is OOMKilled at startup on a node with many CPUs. fd-hard-limit is now 32 per MiB of the memory limit, from which HAProxy derives about 16 connections per MiB, 2035 for the t1.nano defaults. That leaves 64 KiB per connection against the roughly 35 KiB HAProxy budgets per connection itself when it is given a memory limit with -m, and the rest covers the process and the kernel socket buffers charged to the same cgroup. It caps the derived value instead of setting maxconn, because an explicit maxconn above half the hard nofile limit stops HAProxy at startup (strict-limits is on by default), while fd-hard-limit still follows a lower system limit. thread-hard-limit is the CPU limit rounded up, at least 1. It only caps the detected thread count, so a node with fewer CPUs is not forced to run more threads than it has. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent 1f92bcd commit 20545d0

2 files changed

Lines changed: 43 additions & 0 deletions

File tree

‎packages/apps/tcp-balancer/templates/configmap.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ default-server observe layer4 error-limit 10 on-error mark-down check send-proxy
1111
{{- fail (printf "mode %s is not supported" .) }}
1212
{{- end }}
1313
{{- end }}
14+
{{- $resources := include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | fromYaml }}
15+
{{- $memoryMi := div (include "cozy-lib.resources.toFloat" $resources.limits.memory | float64 | int64) 1048576 }}
16+
{{- $cpu := include "cozy-lib.resources.toFloat" $resources.limits.cpu | float64 }}
1417

1518
apiVersion: v1
1619
kind: ConfigMap
@@ -21,6 +24,10 @@ metadata:
2124
app.kubernetes.io/managed-by: {{ .Release.Service }}
2225
data:
2326
haproxy.cfg: |
27+
global
28+
fd-hard-limit {{ mul $memoryMi 32 }}
29+
thread-hard-limit {{ max (ceil $cpu | int64) 1 }}
30+
2431
defaults
2532
mode tcp
2633
option dontlognull

‎packages/apps/tcp-balancer/tests/haproxy-config_test.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,3 +130,39 @@ tests:
130130
- matchRegex:
131131
path: data["haproxy.cfg"]
132132
pattern: '(?m)^frontend talos$\n(?:[ \t]+.*\n)*?[ \t]+acl whitelist src 10\.0\.0\.0/8\n[ \t]+acl all src 0\.0\.0\.0\n[ \t]+tcp-request content accept if whitelist\n[ \t]+tcp-request content reject$'
133+
134+
- it: caps the automatic fd and thread counts by the default preset
135+
asserts:
136+
- matchRegex:
137+
path: data["haproxy.cfg"]
138+
pattern: '\Aglobal\n[ \t]+fd-hard-limit 4096\n[ \t]+thread-hard-limit 1\n\n'
139+
- notMatchRegex:
140+
path: data["haproxy.cfg"]
141+
pattern: '\Aglobal\n(?:[ \t]+.*\n)*?[ \t]+(?:maxconn|nbthread)\b'
142+
143+
- it: scales the caps with the preset
144+
set:
145+
resourcesPreset: t1.medium
146+
asserts:
147+
- matchRegex:
148+
path: data["haproxy.cfg"]
149+
pattern: '\Aglobal\n[ \t]+fd-hard-limit 32768\n[ \t]+thread-hard-limit 2\n\n'
150+
151+
- it: takes the caps from explicit resources and rounds the CPU limit up
152+
set:
153+
resources:
154+
cpu: 2500m
155+
memory: 768Mi
156+
asserts:
157+
- matchRegex:
158+
path: data["haproxy.cfg"]
159+
pattern: '\Aglobal\n[ \t]+fd-hard-limit 24576\n[ \t]+thread-hard-limit 3\n\n'
160+
161+
- it: never renders a thread cap of zero, which HAProxy refuses to parse
162+
set:
163+
resources:
164+
cpu: "0"
165+
asserts:
166+
- matchRegex:
167+
path: data["haproxy.cfg"]
168+
pattern: '(?m)^[ \t]+thread-hard-limit 1$'

0 commit comments

Comments
 (0)