You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b25a132
Browse filesBrowse the repository at this point in the historyBrowse files
fix(velero): add startupProbe so slow startup does not crashloop the install gate (#3138)
## What
Add a `startupProbe` to the velero server container (and supply it
through the package values), so a slow server start under a heavy
parallel install no longer trips the liveness probe. Pin the probe with
a helm-unittest regression guard.
## Why
Velero binds its metrics/health endpoint (`:8085`, `http-monitoring`)
only after the server loads its plugins and connects to the API server.
Under a heavy parallel platform install that can take minutes, but the
upstream liveness probe starts checking `/metrics` at
`initialDelaySeconds=10` and kills the container after
`failureThreshold=5` (~130s total) — before `:8085` is bound. The
kubelet SIGTERMs velero (exit code 0 / Completed — a graceful kill, not
a crash), it BackOff-restarts, and never escapes the loop inside the
install-gate window. Because `backupstrategy-controller` hard-depends on
velero, this blocks the install gate.
## How
While a `startupProbe` is failing, the kubelet holds off the liveness
and readiness probes entirely. So the startupProbe grants the server an
explicit startup budget — `failureThreshold * periodSeconds = 30 * 10s =
300s` of grace — and then hands over to the unchanged, tight upstream
liveness (`10s / 30s / 5`) for steady-state crash detection. This is
strictly better than loosening liveness, which would also blunt
steady-state detection for the lifetime of every pod.
The upstream velero chart (12.0.3) does not render a `startupProbe`, so
the package patches the deployment template via
`patches/add-startup-probe.patch`, re-applied after re-vendor by the
Makefile `update` target — the same vendored-patch convention used by
`seaweedfs`, `ingress-nginx`, and the existing `startupProbe` patches in
`grafana-operator` and `kubevirt-cdi-operator`. The `startupProbe` value
itself is supplied through the package values override.
## Notes
Surfaced as a recurring (not flaky) install failure under heavy parallel
load. Does not close a specific issue.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a configurable Velero `startupProbe` for the `/metrics` endpoint
(`http-monitoring`) to better accommodate slow startup.
* **Bug Fixes**
* When metrics are disabled, the rendered container no longer includes
`startupProbe`, `livenessProbe`, or `readinessProbe`.
* **Tests**
* Added invariant chart rendering tests to verify `startupProbe`
behavior (including pinned timing fields) and confirm existing probe
timing remains unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
0 commit comments