SeaweedFS masters keep raft state on emptyDir and crash-loop on split-brain after quorum loss #4607

Describe the bug

SeaweedFS masters lose their raft state on every restart, and after a quorum loss they never recover on their own. All three masters end up exiting on SeaweedFS's split-brain check, and the tenant's S3 stays down until someone deletes all master pods by hand.

The masters run with -mdir=/data on an emptyDir. The value comes from packages/system/seaweedfs/values.yaml, which sets master.data.type: "emptyDir". The upstream chart default is hostPath.

One master restart is harmless: it comes back empty and catches up from the leader. Losing two masters at once is the problem, for example a node outage followed by drains. As far as I can tell from the logs, the fresh masters form a new raft cluster with a new TopologyId while the survivor keeps the old one. SeaweedFS then refuses to continue, and every master that becomes leader exits with glog.Fatalf in weed/topology/topology.go.

Environment

• Cozystack v1.6.4, SeaweedFS image chrislusf/seaweedfs:4.31, topology: Simple, 3 masters
• Provider: bare metal (Hetzner), Talos

To Reproduce

I did not reproduce this on purpose. It happened in one of three tenants after a night of node reboots. The other two tenants went through the same reboots and survived. Most likely steps:

1. Deploy a SeaweedFS app with master.replicas: 3.
2. Restart two master pods at the same time while the third keeps running (for example, lose one node and drain another).
3. Check the master logs once the pods are back.

Expected behaviour

Masters rejoin the existing cluster after any restart. At minimum, a quorum loss should not leave them with two different TopologyIds.

Actual behaviour

Every master crash-loops, and the filers and the S3 gateway stop serving. The tenant stayed like this for about 16 hours, until I deleted all three master pods in one command. podManagementPolicy is Parallel, so they came back together with empty raft and agreed on a new TopologyId. Nothing was lost: volume data sits on the volume servers' PVCs, and filer metadata is in the seaweedfs-db Postgres.

Logs

I master_server.go:248 [seaweedfs-master-0...:9333.19333] seaweedfs-master-0...:9333.19333 becomes leader.
I master_server.go:310 ensureTopologyId: sending barrier command
F topology.go:712 Split-brain detected! Current TopologyId is ca380c32-1a50-44bf-bdaf-96aed48b31de, but received ea3d2884-8c17-4eac-8d64-97dcfd108f04. Stopping to prevent data corruption.

Filers meanwhile:

masterClient failed to receive from seaweedfs-master-2...:9333: rpc error: code = Unknown desc = raft.Server: Not current leader

Additional context

The chart already supports master.data.type: persistentVolumeClaim. With a PVC a restarted master keeps its raft state, which should prevent the split. Existing tenants will need a migration note, since volumeClaimTemplates of a StatefulSet cannot be changed in place. Until then, the recovery is to delete all master pods at once.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs