Skip to content

Commit 40949c8

Browse files
fix(grafana-operator): add startupProbe to survive slow startup under load
Install Cozystack failed because grafana-operator pod looped in CrashLoopBackOff under concurrent helm-install load (Flux v2.8 helm-controller dispatching every system HR in parallel + kstatus walking all applied resources). The chain that fell over: grafana-operator pod liveness probe :8081/healthz fails 3x in 30s -> monitoring-rd dependency 'cozy-grafana-operator/grafana-operator' -> tenant-rd dependency 'cozy-system/monitoring-rd' -> cozystack-basics dependency 'cozy-system/tenant-rd' (cozystack-basics creates the cozy-public ns) -> kubevirt-cdi RoleBinding cdi-clone-dv: namespace "cozy-public" not found -> vm-disk-rd, dashboard, flux-plunger all wait, install times out Pod log shows the manager starts at T+0, controllers spin up workers, folder/dashboard/datasource sync completes by T+10s — then SIGTERM at T+27s. Liveness probe is the upstream chart default (delay=0s, timeout=1s, period=10s, failureThreshold=3) which gives ~30s grace, not enough on a fresh cluster where controller-runtime's HTTP probe server is still binding while kubelet starts polling immediately. Same shape as the kubevirt-cdi-operator commit already in this branch (37b06e2). Mirror the fix: - Edit deployment.yaml on the vendored chart to add a startupProbe (30 x 5s = 150s grace, /healthz on :8081). Once it passes once, liveness/readiness keep the existing strict config. - Stash the change as patches/startupProbe.diff and re-apply it from the Makefile update target after the curl|tar upstream fetch, so the next chart bump does not silently regress. The patch uses package-relative paths (a/charts/...) so `patch -p1` from the package dir resolves correctly — repo-relative paths would need patch's fuzzy fallback which is unreliable across distros. Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 394b1b2 commit 40949c8

3 files changed

Lines changed: 25 additions & 0 deletions

File tree

‎packages/system/grafana-operator/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ update:
99
mkdir -p charts
1010
curl -sSL https://github.com/grafana-operator/grafana-operator/archive/refs/heads/master.tar.gz | \
1111
tar xzvf - --strip 3 -C charts grafana-operator-master/deploy/helm/grafana-operator
12+
patch --no-backup-if-mismatch -p1 < patches/startupProbe.diff
1213

1314
image:
1415
docker buildx build --file images/grafana-dashboards/Dockerfile ../../.. \

‎packages/system/grafana-operator/charts/grafana-operator/templates/deployment.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,13 @@ spec:
7979
httpGet:
8080
path: /readyz
8181
port: 8081
82+
startupProbe:
83+
httpGet:
84+
path: /healthz
85+
port: 8081
86+
periodSeconds: 5
87+
failureThreshold: 30
88+
timeoutSeconds: 10
8289
{{- with .Values.resources }}
8390
resources:
8491
{{- toYaml . | nindent 12 }}
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
diff --git a/charts/grafana-operator/templates/deployment.yaml b/charts/grafana-operator/templates/deployment.yaml
2+
--- a/charts/grafana-operator/templates/deployment.yaml
3+
+++ b/charts/grafana-operator/templates/deployment.yaml
4+
@@ -79,6 +79,13 @@ spec:
5+
httpGet:
6+
path: /readyz
7+
port: 8081
8+
+ startupProbe:
9+
+ httpGet:
10+
+ path: /healthz
11+
+ port: 8081
12+
+ periodSeconds: 5
13+
+ failureThreshold: 30
14+
+ timeoutSeconds: 10
15+
{{- with .Values.resources }}
16+
resources:
17+
{{- toYaml . | nindent 12 }}

0 commit comments

Comments
 (0)