Repository navigation
Commit 9118cb3
committed
fix(grafana-operator): add startupProbe to survive slow startup under load
Install Cozystack failed because grafana-operator pod looped in
CrashLoopBackOff under concurrent helm-install load (Flux v2.8
helm-controller dispatching every system HR in parallel + kstatus
walking all applied resources). The chain that fell over:
grafana-operator pod liveness probe :8081/healthz fails 3x in 30s
-> monitoring-rd dependency 'cozy-grafana-operator/grafana-operator'
-> tenant-rd dependency 'cozy-system/monitoring-rd'
-> cozystack-basics dependency 'cozy-system/tenant-rd'
(cozystack-basics creates the cozy-public ns)
-> kubevirt-cdi RoleBinding cdi-clone-dv: namespace
"cozy-public" not found
-> vm-disk-rd, dashboard, flux-plunger all wait, install times out
Pod log shows the manager starts at T+0, controllers spin up workers,
folder/dashboard/datasource sync completes by T+10s — then SIGTERM at
T+27s. Liveness probe is the upstream chart default (delay=0s,
timeout=1s, period=10s, failureThreshold=3) which gives ~30s grace,
not enough on a fresh cluster where controller-runtime's HTTP probe
server is still binding while kubelet starts polling immediately.
Same shape as the kubevirt-cdi-operator commit already in this branch
(37b06e2). Mirror the fix:
- Edit deployment.yaml on the vendored chart to add a startupProbe
(30 x 5s = 150s grace, /healthz on :8081). Once it passes once,
liveness/readiness keep the existing strict config.
- Stash the change as patches/startupProbe.diff and re-apply it from
the Makefile update target after the curl|tar upstream fetch, so
the next chart bump does not silently regress.
The patch uses package-relative paths (a/charts/...) so `patch -p1`
from the package dir resolves correctly — repo-relative paths would
need patch's fuzzy fallback which is unreliable across distros.
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit 40949c8)1 parent 43548d9 commit 9118cb3
3 files changed
Lines changed: 25 additions & 0 deletions
File tree
- packages/system/grafana-operator
- charts/grafana-operator/templates
- patches
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
82 | 89 | | |
83 | 90 | | |
84 | 91 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
0 commit comments