Repository navigation
Description
Describe the bug
The master dies with an unrecoverable fatal error: concurrent map read and map write: VolumeLayout.SetVolumeCrowded inserts into vl.crowded while holding only accessLock.RLock(), and GetWritableVolumeCount (called from the Assign
hot path) reads the same map under RLock(). Two read-lock holders, one writing a map another is reading → the Go runtime aborts the process. The "becomes crowded" log line from the writer interleaves into the fatal output of the crashed reader goroutine.
Version:4.45
Stack trace (reader/crashed side) and the interleaved writer log
I0904 14:23:59.983556 volume_layout.go:1017 Volume 166143146 becomes crowded
I0904 14:24:02.922182 volume_layout.go:1009 Volume 166143146 becomes uncrowded
I0904 14:24:04.242155 volume_layout.go:1009 Volume 166143152 becomes uncrowded
fatal error: I0904 14:24:07.492703 volume_layout.go:1017 Volume 166143152 becomes crowded
concurrent map read and map write
goroutine 50484648 [running]:
internal/runtime/maps.fatal({0x61bb3f2?, 0xd32e77c?})
/usr/local/go/src/runtime/panic.go:1195 +0x18
github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(0x3da81e58a480)
/github/workspace/weed/topology/volume_layout.go:834 +0xc5
github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(0x3da81d0be540, {0xcdcfad0, 0x3da8256d24b0}, 0x3da82589c160)
/github/workspace/weed/server/master_grpc_server_assign.go:97 +0x2de
github.com/seaweedfs/seaweedfs/weed/pb/master_pb._Seaweed_Assign_Handler.func1({0xcdcfad0?, 0x3da8256d24b0?}, {0xb89f888?, 0x3da82589c160?})
/github/workspace/weed/pb/master_pb/master_grpc.pb.go:568 +0xcb
github.com/seaweedfs/seaweedfs/weed/pb.requestIDUnaryInterceptor.func1({0xcdcfad0, 0x3da8256d2420}, {0xb89f888, 0x3da82589c160}, 0x2a?, 0x3da81d498150)
/github/workspace/weed/pb/grpc_client_server.go:321 +0x22e
github.com/seaweedfs/seaweedfs/weed/pb/master_pb._Seaweed_Assign_Handler({0xbb3ff28, 0x3da81d0be540}, {0xcdcfad0, 0x3da8256d2420}, 0x3da81d498120, 0xce321f8)
/github/workspace/weed/pb/master_pb/master_grpc.pb.go:570 +0x153
google.golang.org/grpc.(*Server).wrapUnaryHandler.func1({0xbb3ff28, 0x3da81d0be540}, {0xcde1918, 0x3da82540be00})
/go/pkg/mod/google.golang.org/[email protected]/server.go:1274 +0xc3
google.golang.org/grpc.(*Server).processRPC(0x3da81daa1688, {0xcdcfad0, 0x3da8256d23c0}, 0x3da8244d01a0, 0x3da81d29d230, 0x3da81e012060, 0x0)
/go/pkg/mod/google.golang.org/[email protected]/server.go:1433 +0x11b4
google.golang.org/grpc.(*Server).handleStream(0x3da81daa1688, {0xcdd2118, 0x3da81f066d00}, 0x3da8244d01a0)
/go/pkg/mod/google.golang.org/[email protected]/server.go:1577 +0xa94
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/go/pkg/mod/google.golang.org/[email protected]/server.go:1086 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 438023
/go/pkg/mod/google.golang.org/[email protected]/server.go:1097 +0x127
Root cause
weed/topology/volume_layout.go:
func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
// since delete is guarded by accessLock.Lock(),
// ...
// RLock() should be safe enough <-- the bet
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
vl.setVolumeCrowded(vid) // == vl.crowded[vid] = struct{}{} :1015-1016
}Until #10522 (commit ef6a706, 2026-07-31) nothing read the crowded map by key under the lock, so the write-under-RLock "solved" the problem for the existing readers. #10522 added a per-vid read of vl.crowded inside GetWritableVolumeCount (volume_layout.go:833-838) — precisely on the Assign hot path — turning the assumption into a live race:
for _, vid := range vl.writables {
if _, ok := vl.crowded[vid]; ok { // reader under RLockRWMutex allows concurrent RLock holders, so one heartbeating cluster with any assign traffic and any volume flipping to crowded hits fatal: concurrent map read and map write at random and kills the master. Unlike a panic, a fatal error cannot be recovered.
The event path is asynchronous: volume-state scan -> chanCrowdedVolumes (node.go) -> Topology.SetVolumeCrowded (topology_event_handling.go) -> the RLock writer above.
Suggested fix
Give the event path the write lock; it is a low-frequency single consumer and every other mutation of writables/crowded is already done under Lock():
func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
- vl.accessLock.RLock()
- defer vl.accessLock.RUnlock()
+ vl.accessLock.Lock()
+ defer vl.accessLock.Unlock()
vl.setVolumeCrowded(vid)
}setVolumeCrowded takes no nested locks, so there is no deadlock path. The hot readers (GetWritableVolumeCount, CloneWritableVolumes) keep using RLock.
Reproducer: loop concurrent SetVolumeCrowded(vid) against GetWritableVolumeCount; a -race test flags it immediately, passes (plain and -race) with the lock change.
Impact
Random master process death in steady-state write traffic once any volume becomes crowded. No data corruption (topology is rebuilt after restart), but the master is unavailable until restarted, and fatal errors bypass every recover-based mitigation.
Activity
topology: fix fatal concurrent map read/write on VolumeLayout.crowded (…
fix(seaweedfs): stop the master crashing on the crowded volume map (#…
fix(seaweedfs): stop the master crashing on the crowded volume map (#…