Skip to content

Commit b25a132

Browse files
authored
fix(velero): add startupProbe so slow startup does not crashloop the install gate (#3138)
## What Add a `startupProbe` to the velero server container (and supply it through the package values), so a slow server start under a heavy parallel install no longer trips the liveness probe. Pin the probe with a helm-unittest regression guard. ## Why Velero binds its metrics/health endpoint (`:8085`, `http-monitoring`) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe starts checking `/metrics` at `initialDelaySeconds=10` and kills the container after `failureThreshold=5` (~130s total) — before `:8085` is bound. The kubelet SIGTERMs velero (exit code 0 / Completed — a graceful kill, not a crash), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because `backupstrategy-controller` hard-depends on velero, this blocks the install gate. ## How While a `startupProbe` is failing, the kubelet holds off the liveness and readiness probes entirely. So the startupProbe grants the server an explicit startup budget — `failureThreshold * periodSeconds = 30 * 10s = 300s` of grace — and then hands over to the unchanged, tight upstream liveness (`10s / 30s / 5`) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection for the lifetime of every pod. The upstream velero chart (12.0.3) does not render a `startupProbe`, so the package patches the deployment template via `patches/add-startup-probe.patch`, re-applied after re-vendor by the Makefile `update` target — the same vendored-patch convention used by `seaweedfs`, `ingress-nginx`, and the existing `startupProbe` patches in `grafana-operator` and `kubevirt-cdi-operator`. The `startupProbe` value itself is supplied through the package values override. ## Notes Surfaced as a recurring (not flaky) install failure under heavy parallel load. Does not close a specific issue. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable Velero `startupProbe` for the `/metrics` endpoint (`http-monitoring`) to better accommodate slow startup. * **Bug Fixes** * When metrics are disabled, the rendered container no longer includes `startupProbe`, `livenessProbe`, or `readinessProbe`. * **Tests** * Added invariant chart rendering tests to verify `startupProbe` behavior (including pinned timing fields) and confirm existing probe timing remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 9c18260 + f7271b8 commit b25a132

5 files changed

Lines changed: 114 additions & 0 deletions

File tree

‎packages/system/velero/Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,6 @@ update:
1212
helm repo add tanzu https://vmware-tanzu.github.io/helm-charts
1313
helm repo update tanzu
1414
helm pull tanzu/velero --untar --untardir charts --version 12.0.3
15+
# Render a startupProbe from .Values.startupProbe (upstream chart omits it),
16+
# so a slow server start under heavy parallel install does not trip liveness.
17+
patch --no-backup-if-mismatch -p4 < patches/add-startup-probe.patch

‎packages/system/velero/charts/velero/templates/deployment.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,9 @@ spec:
199199
{{- toYaml . | nindent 12 }}
200200
{{- end }}
201201
{{- if .Values.metrics.enabled }}
202+
{{- with .Values.startupProbe }}
203+
startupProbe: {{- toYaml . | nindent 12 }}
204+
{{- end }}
202205
{{- with .Values.livenessProbe }}
203206
livenessProbe: {{- toYaml . | nindent 12 }}
204207
{{- end }}
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
diff --git a/packages/system/velero/charts/velero/templates/deployment.yaml b/packages/system/velero/charts/velero/templates/deployment.yaml
2+
index d2a72639a..a46d73200 100644
3+
--- a/packages/system/velero/charts/velero/templates/deployment.yaml
4+
+++ b/packages/system/velero/charts/velero/templates/deployment.yaml
5+
@@ -199,6 +199,9 @@ spec:
6+
{{- toYaml . | nindent 12 }}
7+
{{- end }}
8+
{{- if .Values.metrics.enabled }}
9+
+ {{- with .Values.startupProbe }}
10+
+ startupProbe: {{- toYaml . | nindent 12 }}
11+
+ {{- end }}
12+
{{- with .Values.livenessProbe }}
13+
livenessProbe: {{- toYaml . | nindent 12 }}
14+
{{- end }}

‎packages/system/velero/tests/velero_test.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,73 @@ tests:
3636
asserts:
3737
- hasDocuments:
3838
count: 0
39+
40+
# Velero binds :8085 (http-monitoring) late under a heavy parallel install, so
41+
# the upstream liveness probe (initialDelaySeconds 10, ~130s total budget) would
42+
# kill it mid-startup and crashloop it out of the install-gate window. A
43+
# startupProbe (patched into the upstream chart, which omits one) holds liveness
44+
# and readiness off until the server is up, granting 30 * 10s = 300s of startup
45+
# grace while leaving steady-state liveness at the tight upstream default. Pin
46+
# the startupProbe budget AND that liveness was NOT loosened, so neither
47+
# silently regresses.
48+
- it: gates startup with a startupProbe and keeps liveness tight
49+
template: charts/velero/templates/deployment.yaml
50+
documentSelector:
51+
path: metadata.name
52+
value: velero
53+
asserts:
54+
- equal:
55+
path: spec.template.spec.containers[0].startupProbe.periodSeconds
56+
value: 10
57+
- equal:
58+
path: spec.template.spec.containers[0].startupProbe.failureThreshold
59+
value: 30
60+
- equal:
61+
path: spec.template.spec.containers[0].startupProbe.timeoutSeconds
62+
value: 5
63+
- equal:
64+
path: spec.template.spec.containers[0].startupProbe.successThreshold
65+
value: 1
66+
- equal:
67+
path: spec.template.spec.containers[0].startupProbe.httpGet.path
68+
value: /metrics
69+
- equal:
70+
path: spec.template.spec.containers[0].startupProbe.httpGet.port
71+
value: http-monitoring
72+
# No initialDelaySeconds: the first startup check fires immediately, so the
73+
# full 300s budget is spent probing. Pin its absence so a future change
74+
# cannot silently re-introduce a delay that eats into the grace window.
75+
- notExists:
76+
path: spec.template.spec.containers[0].startupProbe.initialDelaySeconds
77+
# Liveness stays at the tight upstream default — the startupProbe owns the
78+
# grace, so loosening liveness would only weaken steady-state crash
79+
# detection. Pin the full budget (initialDelay/period/failureThreshold) so a
80+
# future change cannot silently re-delay or loosen liveness.
81+
- equal:
82+
path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
83+
value: 10
84+
- equal:
85+
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
86+
value: 30
87+
- equal:
88+
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
89+
value: 5
90+
91+
# All three probes live behind {{- if .Values.metrics.enabled }} — :8085 only
92+
# exists when metrics are on. Pin that the startupProbe follows the same gate as
93+
# liveness/readiness, so disabling metrics drops all probes together and never
94+
# leaves a startupProbe pointed at an unbound port.
95+
- it: omits all probes (including the startupProbe) when metrics are disabled
96+
template: charts/velero/templates/deployment.yaml
97+
documentSelector:
98+
path: metadata.name
99+
value: velero
100+
set:
101+
velero.metrics.enabled: false
102+
asserts:
103+
- notExists:
104+
path: spec.template.spec.containers[0].startupProbe
105+
- notExists:
106+
path: spec.template.spec.containers[0].livenessProbe
107+
- notExists:
108+
path: spec.template.spec.containers[0].readinessProbe

‎packages/system/velero/values.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,27 @@ velero:
3232
# Increase timeout for item operations to 24 hours to prevent timeouts
3333
# during backups of very large volumes. The Velero default is 4 hours.
3434
defaultItemOperationTimeout: 24h
35+
36+
# Velero binds its metrics/health endpoint (:8085, http-monitoring) only after
37+
# the server finishes loading plugins and connecting to the API server. Under a
38+
# heavy parallel platform install that can take minutes, but the upstream
39+
# liveness probe starts checking /metrics at initialDelaySeconds=10 and kills
40+
# the container after failureThreshold=5 (~130s total) before :8085 is bound.
41+
# The kubelet SIGTERMs velero (exitCode 0/Completed), it BackOff-restarts, and
42+
# never escapes the loop inside the install-gate window, blocking the gate.
43+
#
44+
# Give the server an explicit startup budget. While the startupProbe is failing
45+
# the kubelet holds off the liveness and readiness probes entirely, so steady-
46+
# state liveness stays at the tight upstream default (10s / 30s / 5 failures)
47+
# once startup completes. failureThreshold * periodSeconds = 30 * 10s = 300s of
48+
# grace. The upstream chart does not render a startupProbe, so the velero
49+
# package patches it in (patches/add-startup-probe.patch, re-applied on update).
50+
startupProbe:
51+
httpGet:
52+
path: /metrics
53+
port: http-monitoring
54+
scheme: HTTP
55+
periodSeconds: 10
56+
timeoutSeconds: 5
57+
successThreshold: 1
58+
failureThreshold: 30

0 commit comments

Comments
 (0)