Skip to content

Volume servers fail liveness probe due to strict replication check in /healthz endpoint #7757

Closed

Description

@kvaps

Version

docker.io/chrislusf/seaweedfs:3.97

Problem Description

Volume servers in Kubernetes clusters are constantly failing liveness probes, causing pods to restart repeatedly (observed 934+ restarts). The /healthz endpoint returns HTTP 503 (Service Unavailable) when any volume with replication > 1 has issues checking remote replicas, which is too strict for a liveness probe.

Current Behavior

The healthzHandler in weed/server/volume_server_handlers_admin.go checks all volumes with replication > 1 and returns 503 if GetWritableRemoteReplications fails for any volume. This causes:

  1. Kubernetes liveness probe failures
  2. Pod restarts (observed 934+ restarts on some pods)
  3. Cascading failures during temporary network issues or cluster maintenance
  4. Service disruption even when the volume server itself is healthy

Kubernetes Events Example

Warning   Unhealthy                 pod/seaweedfs-volume-gld-1    Liveness probe failed: HTTP probe failed with statuscode: 503
Normal    Killing                   pod/seaweedfs-volume-gld-2    Container seaweedfs failed liveness probe, will be restarted

Liveness Probe Configuration

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 20
  periodSeconds: 90
  failureThreshold: 4
  timeoutSeconds: 30

Root Cause

The current implementation at weed/server/volume_server_handlers_admin.go:14-30:

func (vs *VolumeServer) healthzHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Server", "SeaweedFS Volume "+version.VERSION)
    volumeInfos := vs.store.VolumeInfos()
    for _, vinfo := range volumeInfos {
        if len(vinfo.Collection) == 0 {
            continue
        }
        if vinfo.ReplicaPlacement.GetCopyCount() > 1 {
            _, err := topology.GetWritableRemoteReplications(vs.store, vs.grpcDialOption, vinfo.Id, vs.GetMaster)
            if err != nil {
                w.WriteHeader(http.StatusServiceUnavailable)  // ❌ Too strict
                return
            }
        }
    }
    w.WriteHeader(http.StatusOK)
}

The GetWritableRemoteReplications function can fail due to:

  • Temporary network issues with master nodes
  • Temporary unavailability of remote replicas during maintenance
  • Lookup failures in operation.LookupVolumeId
  • Replica count mismatch (when len(locations) < copyCount)

Expected Behavior

A liveness probe should check if the service itself is running and responsive, not whether all volumes have perfect replication status. The service should be considered "alive" even if some volumes have replication issues, as these are operational concerns that should be monitored separately.

Impact

  • High: Causes unnecessary pod restarts and service disruption
  • Affects clusters with volumes using replication > 1
  • Particularly problematic during:
    • Cluster maintenance
    • Temporary network issues
    • Partial node failures
    • Initial cluster setup

Proposed Solution

Modify healthzHandler to:

  1. Verify the store is accessible (basic liveness check)
  2. Log replication issues for monitoring/debugging
  3. Return 200 OK even if some volumes have replication issues
  4. Only return 503 if the service itself is unavailable (e.g., vs.store == nil)

Implementation

func (vs *VolumeServer) healthzHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Server", "SeaweedFS Volume "+version.VERSION)
    
    // Check basic service availability
    if vs.store == nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    
    volumeInfos := vs.store.VolumeInfos()
    if volumeInfos == nil || len(volumeInfos) == 0 {
        // No volumes is OK - server might be starting up
        w.WriteHeader(http.StatusOK)
        return
    }
    
    // Log replication issues for monitoring, but don't fail the health check
    replicationIssues := 0
    for _, vinfo := range volumeInfos {
        if len(vinfo.Collection) == 0 {
            continue
        }
        if vinfo.ReplicaPlacement.GetCopyCount() > 1 {
            _, err := topology.GetWritableRemoteReplications(vs.store, vs.grpcDialOption, vinfo.Id, vs.GetMaster)
            if err != nil {
                replicationIssues++
                glog.V(1).Infof("healthz: volume %d has replication issue: %v", vinfo.Id, err)
            }
        }
    }
    
    if replicationIssues > 0 {
        glog.V(0).Infof("healthz: %d volumes have replication issues, but service is healthy", replicationIssues)
    }
    
    w.WriteHeader(http.StatusOK)
}

Alternative Considerations

If stricter health checking is desired for replication status:

  1. Create a separate /readiness endpoint for replication checks
  2. Use /readiness for Kubernetes readiness probes (not liveness)
  3. Keep /healthz simple for liveness checks

Environment

  • SeaweedFS Version: 3.97 (observed in logs)
  • Deployment: Kubernetes (StatefulSets)
  • Replication: Volumes with replication factor > 1 (replication=100 observed)

Steps to Reproduce

  1. Deploy SeaweedFS volume servers with volumes using replication > 1
  2. Configure Kubernetes liveness probe on /healthz endpoint
  3. Cause temporary network issues or partial replica unavailability
  4. Observe liveness probe failures and pod restarts

Related Code

  • weed/server/volume_server_handlers_admin.go - health check handler
  • weed/topology/store_replicate.go:209 - GetWritableRemoteReplications function
  • weed/operation/lookup.go:57 - LookupVolumeId function

Activity

  1. kvaps commented on Dec 15, 2025

    @kvaps
    ContributorAuthor

    Probably fixed already by #7610

  2. added a commit that references this issue on Dec 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions