Skip to content

Commit 45a273e

Browse files
committed
fix(monitoring-agents): relax fluent-bit probes for loaded installs
The fluent-bit DaemonSet crash-loops on a loaded install, timing out its HelmRelease and cascading into velero and backupstrategy-controller DependencyNotReady failures. The subchart default liveness/readiness probes (initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP server on :2020, but that server does not bind until AFTER the kubernetes filter's 'testing connectivity with API server' step. Under a concurrent install the kube-API can be unreachable for ~10s, delaying the :2020 bind, while the single-threaded HTTP server can miss a 1s timeout when the engine is busy. The probes then fire into a connection-refused port and liveness-kill fluent-bit into a restart loop. Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5, failureThreshold 6) so the API-gated late bind and busy-engine blips are tolerated. The httpGet target is unchanged (path / on port http = :2020). Add a helm-unittest pinning the relaxed timing and the preserved target. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 29bf56f commit 45a273e

2 files changed

Lines changed: 72 additions & 0 deletions

File tree

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
suite: fluent-bit liveness/readiness probes carry load headroom
2+
3+
# Regression guard for the fluent-bit crash loop on a loaded install. The
4+
# subchart default probes (initialDelaySeconds 0, timeoutSeconds 1,
5+
# failureThreshold 3) fire into a connection-refused :2020 before fluent-bit's
6+
# HTTP server binds — that bind happens only AFTER the kubernetes filter's
7+
# "testing connectivity with API server" step, which the kube-API can gate for
8+
# ~10s under a concurrent install. The aggressive defaults then liveness-kill
9+
# fluent-bit into a restart loop, timing out the HelmRelease. The umbrella
10+
# values relax both probes. Pin the relaxed timing AND the unchanged target
11+
# (httpGet path / on port http = :2020) — a wrong path/port would make the
12+
# probe always fail, strictly worse than the bug it fixes.
13+
release:
14+
name: monitoring-agents
15+
namespace: cozy-monitoring
16+
tests:
17+
- it: liveness probe keeps :2020 target and gains load headroom
18+
template: charts/fluent-bit/templates/daemonset.yaml
19+
asserts:
20+
- equal:
21+
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
22+
value: /
23+
- equal:
24+
path: spec.template.spec.containers[0].livenessProbe.httpGet.port
25+
value: http
26+
- equal:
27+
path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
28+
value: 15
29+
- equal:
30+
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
31+
value: 10
32+
- equal:
33+
path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
34+
value: 5
35+
- equal:
36+
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
37+
value: 6
38+
39+
- it: readiness probe keeps :2020 target and gains load headroom
40+
template: charts/fluent-bit/templates/daemonset.yaml
41+
asserts:
42+
- equal:
43+
path: spec.template.spec.containers[0].readinessProbe.httpGet.path
44+
value: /
45+
- equal:
46+
path: spec.template.spec.containers[0].readinessProbe.httpGet.port
47+
value: http
48+
- equal:
49+
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
50+
value: 5
51+
- equal:
52+
path: spec.template.spec.containers[0].readinessProbe.failureThreshold
53+
value: 6

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,9 +287,28 @@ vmagent:
287287
fluent-bit:
288288
rbac:
289289
eventsAccess: true
290+
# Relax probe timing to survive a loaded install. Fluent-bit's HTTP server on
291+
# :2020 (the probe target) does not bind until AFTER the kubernetes filter's
292+
# "testing connectivity with API server" step; under concurrent install the
293+
# kube-API can be unreachable for ~10s, delaying the :2020 bind, and the
294+
# single-threaded HTTP server can miss a 1s timeout while the engine is busy.
295+
# The subchart defaults (initialDelaySeconds 0, timeoutSeconds 1,
296+
# failureThreshold 3) then liveness-kill fluent-bit into a crash loop. httpGet
297+
# stays on path / port http (:2020) — same target as the subchart defaults.
298+
livenessProbe:
299+
httpGet:
300+
path: /
301+
port: http
302+
initialDelaySeconds: 15
303+
periodSeconds: 10
304+
timeoutSeconds: 5
305+
failureThreshold: 6
290306
readinessProbe:
291307
httpGet:
292308
path: /
309+
port: http
310+
timeoutSeconds: 5
311+
failureThreshold: 6
293312
daemonSetVolumes:
294313
- name: varlog
295314
hostPath:

0 commit comments

Comments
 (0)