Repository navigation
Commit 2cb1b4c
authored
fix(seaweedfs): raise master probe timeouts above the cluster-status backoff (#3068)
## What this PR does
The SeaweedFS master's `/cluster/status` HTTP handler calls
`topo.Leader()`, which backs off exponentially up to
`MaxElapsedTime=20s` while the node has no leader in view. A 3-master
cluster bootstrap is a race: a master that briefly becomes the initial
raft leader and is then demoted can sit with an empty leader-view for a
while, so its `/cluster/status` takes up to ~20s to return (it does
return 200 — just slowly). The master liveness and readiness probes both
`httpGet /cluster/status:9333` with `timeoutSeconds: 10`, which is
shorter than that worst case. Such a master is liveness-killed roughly
every ~137s into a permanent CrashLoop (the master uses an `emptyDir`
data dir, so each container restart re-reads the same raft state and
re-wedges), and its readiness never succeeds — so the master StatefulSet
never reaches full readiness and the `seaweedfs-system` HelmRelease
install times out.
This raises both master probe `timeoutSeconds` to 30 (above the 20s
backoff) via the overlay values, deep-merging onto the subchart defaults
so only the timeout changes. Both probes must be raised: leaving
readiness at 10s would keep the master NotReady on the same slow
endpoint even if liveness no longer kills it. The volume component in
this same chart already uses a 30s probe timeout, so this aligns master
with that precedent. This is purely probe tuning — no raft, protocol, or
storage change.
The underlying upstream behavior (`/cluster/status` slow for a master
with an unhealthy raft view) is tracked upstream and is unchanged across
current SeaweedFS releases, so a version bump does not address it.
Switching the master to the Hashicorp raft backend is a separate, larger
change (it pulls in bootstrap and storage considerations) and is
intentionally out of scope here.
Operational note: on chart update each master StatefulSet rolls one pod
at a time with the new timeouts (raft quorum is preserved); a platform
upgrade therefore performs a short graceful roll of master pods on each
instance — not a data outage.
Adds a helm-unittest pinning both master probe timeouts and guarding
that the subchart's other probe fields survive the deep-merge.
### Release note
```release-note
fix(seaweedfs): raise the master liveness/readiness probe timeouts above the cluster-status leader-lookup backoff so a master with a transiently empty raft view is not killed into a permanent CrashLoop, which could time out the seaweedfs install
```2 files changed
Lines changed: 77 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
35 | 48 | | |
36 | 49 | | |
37 | 50 | | |
| |||
0 commit comments