Repository navigation
Description
Version
docker.io/chrislusf/seaweedfs:3.97
Problem Description
Volume servers in Kubernetes clusters are constantly failing liveness probes, causing pods to restart repeatedly (observed 934+ restarts). The /healthz endpoint returns HTTP 503 (Service Unavailable) when any volume with replication > 1 has issues checking remote replicas, which is too strict for a liveness probe.
Current Behavior
The healthzHandler in weed/server/volume_server_handlers_admin.go checks all volumes with replication > 1 and returns 503 if GetWritableRemoteReplications fails for any volume. This causes:
- Kubernetes liveness probe failures
- Pod restarts (observed 934+ restarts on some pods)
- Cascading failures during temporary network issues or cluster maintenance
- Service disruption even when the volume server itself is healthy
Kubernetes Events Example
Warning Unhealthy pod/seaweedfs-volume-gld-1 Liveness probe failed: HTTP probe failed with statuscode: 503
Normal Killing pod/seaweedfs-volume-gld-2 Container seaweedfs failed liveness probe, will be restarted
Liveness Probe Configuration
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 20
periodSeconds: 90
failureThreshold: 4
timeoutSeconds: 30Root Cause
The current implementation at weed/server/volume_server_handlers_admin.go:14-30:
func (vs *VolumeServer) healthzHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Server", "SeaweedFS Volume "+version.VERSION)
volumeInfos := vs.store.VolumeInfos()
for _, vinfo := range volumeInfos {
if len(vinfo.Collection) == 0 {
continue
}
if vinfo.ReplicaPlacement.GetCopyCount() > 1 {
_, err := topology.GetWritableRemoteReplications(vs.store, vs.grpcDialOption, vinfo.Id, vs.GetMaster)
if err != nil {
w.WriteHeader(http.StatusServiceUnavailable) // ❌ Too strict
return
}
}
}
w.WriteHeader(http.StatusOK)
}The GetWritableRemoteReplications function can fail due to:
- Temporary network issues with master nodes
- Temporary unavailability of remote replicas during maintenance
- Lookup failures in
operation.LookupVolumeId - Replica count mismatch (when
len(locations) < copyCount)
Expected Behavior
A liveness probe should check if the service itself is running and responsive, not whether all volumes have perfect replication status. The service should be considered "alive" even if some volumes have replication issues, as these are operational concerns that should be monitored separately.
Impact
- High: Causes unnecessary pod restarts and service disruption
- Affects clusters with volumes using replication > 1
- Particularly problematic during:
- Cluster maintenance
- Temporary network issues
- Partial node failures
- Initial cluster setup
Proposed Solution
Modify healthzHandler to:
- Verify the store is accessible (basic liveness check)
- Log replication issues for monitoring/debugging
- Return 200 OK even if some volumes have replication issues
- Only return 503 if the service itself is unavailable (e.g.,
vs.store == nil)
Implementation
func (vs *VolumeServer) healthzHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Server", "SeaweedFS Volume "+version.VERSION)
// Check basic service availability
if vs.store == nil {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
volumeInfos := vs.store.VolumeInfos()
if volumeInfos == nil || len(volumeInfos) == 0 {
// No volumes is OK - server might be starting up
w.WriteHeader(http.StatusOK)
return
}
// Log replication issues for monitoring, but don't fail the health check
replicationIssues := 0
for _, vinfo := range volumeInfos {
if len(vinfo.Collection) == 0 {
continue
}
if vinfo.ReplicaPlacement.GetCopyCount() > 1 {
_, err := topology.GetWritableRemoteReplications(vs.store, vs.grpcDialOption, vinfo.Id, vs.GetMaster)
if err != nil {
replicationIssues++
glog.V(1).Infof("healthz: volume %d has replication issue: %v", vinfo.Id, err)
}
}
}
if replicationIssues > 0 {
glog.V(0).Infof("healthz: %d volumes have replication issues, but service is healthy", replicationIssues)
}
w.WriteHeader(http.StatusOK)
}Alternative Considerations
If stricter health checking is desired for replication status:
- Create a separate
/readinessendpoint for replication checks - Use
/readinessfor Kubernetes readiness probes (not liveness) - Keep
/healthzsimple for liveness checks
Environment
- SeaweedFS Version: 3.97 (observed in logs)
- Deployment: Kubernetes (StatefulSets)
- Replication: Volumes with replication factor > 1 (replication=100 observed)
Steps to Reproduce
- Deploy SeaweedFS volume servers with volumes using replication > 1
- Configure Kubernetes liveness probe on
/healthzendpoint - Cause temporary network issues or partial replica unavailability
- Observe liveness probe failures and pod restarts
Related Code
weed/server/volume_server_handlers_admin.go- health check handlerweed/topology/store_replicate.go:209-GetWritableRemoteReplicationsfunctionweed/operation/lookup.go:57-LookupVolumeIdfunction
Activity
kvaps commentedon Dec 15, 2025
Probably fixed already by #7610
Update SeaweedFS v4.02 (#1725)
Update SeaweedFS v4.02 (#1725)
Update SeaweedFS v4.02 (#1725)
Update SeaweedFS v4.02 (#1725)