Skip to content

Commit 2cb1b4c

Browse files
authored
fix(seaweedfs): raise master probe timeouts above the cluster-status backoff (#3068)
## What this PR does The SeaweedFS master's `/cluster/status` HTTP handler calls `topo.Leader()`, which backs off exponentially up to `MaxElapsedTime=20s` while the node has no leader in view. A 3-master cluster bootstrap is a race: a master that briefly becomes the initial raft leader and is then demoted can sit with an empty leader-view for a while, so its `/cluster/status` takes up to ~20s to return (it does return 200 — just slowly). The master liveness and readiness probes both `httpGet /cluster/status:9333` with `timeoutSeconds: 10`, which is shorter than that worst case. Such a master is liveness-killed roughly every ~137s into a permanent CrashLoop (the master uses an `emptyDir` data dir, so each container restart re-reads the same raft state and re-wedges), and its readiness never succeeds — so the master StatefulSet never reaches full readiness and the `seaweedfs-system` HelmRelease install times out. This raises both master probe `timeoutSeconds` to 30 (above the 20s backoff) via the overlay values, deep-merging onto the subchart defaults so only the timeout changes. Both probes must be raised: leaving readiness at 10s would keep the master NotReady on the same slow endpoint even if liveness no longer kills it. The volume component in this same chart already uses a 30s probe timeout, so this aligns master with that precedent. This is purely probe tuning — no raft, protocol, or storage change. The underlying upstream behavior (`/cluster/status` slow for a master with an unhealthy raft view) is tracked upstream and is unchanged across current SeaweedFS releases, so a version bump does not address it. Switching the master to the Hashicorp raft backend is a separate, larger change (it pulls in bootstrap and storage considerations) and is intentionally out of scope here. Operational note: on chart update each master StatefulSet rolls one pod at a time with the new timeouts (raft quorum is preserved); a platform upgrade therefore performs a short graceful roll of master pods on each instance — not a data outage. Adds a helm-unittest pinning both master probe timeouts and guarding that the subchart's other probe fields survive the deep-merge. ### Release note ```release-note fix(seaweedfs): raise the master liveness/readiness probe timeouts above the cluster-status leader-lookup backoff so a master with a transiently empty raft view is not killed into a permanent CrashLoop, which could time out the seaweedfs install ```
2 parents cfd9ae3 + bfc1fbd commit 2cb1b4c

2 files changed

Lines changed: 77 additions & 0 deletions

File tree

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
suite: seaweedfs master probe timeouts
2+
3+
# Pins the master probe timeout overrides that keep master-0 alive during raft
4+
# bootstrap. The master's /cluster/status handler calls topo.Leader(), which
5+
# backs off up to ~20s (MaxElapsedTime=20s) while the node has no leader in
6+
# view — e.g. a master that was briefly the initial raft leader then was demoted.
7+
# The subchart default 10s probe timeout is shorter than that worst case, so such
8+
# a master is liveness-killed into a permanent CrashLoop and the master
9+
# StatefulSet never reaches full readiness. Both probes must allow more than the
10+
# 20s backoff. The overlay overrides only timeoutSeconds and relies on Helm's
11+
# deep-merge to keep every other subchart probe field, so this also guards that
12+
# the merge does not drop the surrounding probe configuration.
13+
14+
templates:
15+
- charts/seaweedfs/templates/master/master-statefulset.yaml
16+
17+
release:
18+
name: seaweedfs
19+
namespace: cozy-seaweedfs
20+
21+
tests:
22+
- it: raises both master probe timeouts above the cluster-status backoff
23+
asserts:
24+
- equal:
25+
path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
26+
value: 30
27+
- equal:
28+
path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
29+
value: 30
30+
31+
- it: preserves the other master probe fields through the deep-merge
32+
asserts:
33+
# liveness: only timeoutSeconds is overridden, the rest survive
34+
- equal:
35+
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
36+
value: /cluster/status
37+
- equal:
38+
path: spec.template.spec.containers[0].livenessProbe.failureThreshold
39+
value: 4
40+
- equal:
41+
path: spec.template.spec.containers[0].livenessProbe.successThreshold
42+
value: 1
43+
- equal:
44+
path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
45+
value: 20
46+
- equal:
47+
path: spec.template.spec.containers[0].livenessProbe.periodSeconds
48+
value: 30
49+
# readiness: only timeoutSeconds is overridden, the rest survive
50+
- equal:
51+
path: spec.template.spec.containers[0].readinessProbe.httpGet.path
52+
value: /cluster/status
53+
- equal:
54+
path: spec.template.spec.containers[0].readinessProbe.failureThreshold
55+
value: 100
56+
- equal:
57+
path: spec.template.spec.containers[0].readinessProbe.successThreshold
58+
value: 2
59+
- equal:
60+
path: spec.template.spec.containers[0].readinessProbe.initialDelaySeconds
61+
value: 10
62+
- equal:
63+
path: spec.template.spec.containers[0].readinessProbe.periodSeconds
64+
value: 45

‎packages/system/seaweedfs/values.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,19 @@ seaweedfs:
3232
type: "emptyDir"
3333
logs:
3434
type: ""
35+
# The master's /cluster/status handler calls topo.Leader(), which backs off
36+
# up to ~20s (MaxElapsedTime=20s) while the node has no leader in view — e.g.
37+
# a master that was briefly the initial raft leader then was demoted during
38+
# bootstrap. The default 10s probe timeout is shorter than that worst case, so
39+
# such a master is liveness-killed every ~137s into a permanent CrashLoop and
40+
# the master StatefulSet never reaches full readiness, timing out the install.
41+
# Both probes must allow more than the 20s backoff (the volume component in
42+
# this chart already uses 30s). Override only timeoutSeconds; Helm deep-merges
43+
# these onto the subchart defaults, keeping enabled/path/initialDelay/etc.
44+
livenessProbe:
45+
timeoutSeconds: 30
46+
readinessProbe:
47+
timeoutSeconds: 30
3548
volume:
3649
replicas: 2
3750
resizeHook:

0 commit comments

Comments
 (0)