Skip to content

[BUG REPORT] 4.45 master: fatal "concurrent map read and map write" on VolumeLayout.crowded — SetVolumeCrowded writes the map under a read lock, raced by GetWritableVolumeCount on the Assign hot path #11211

Closed

Description

@qqren

Describe the bug

The master dies with an unrecoverable fatal error: concurrent map read and map write: VolumeLayout.SetVolumeCrowded inserts into vl.crowded while holding only accessLock.RLock(), and GetWritableVolumeCount (called from the Assign
hot path) reads the same map under RLock(). Two read-lock holders, one writing a map another is reading → the Go runtime aborts the process. The "becomes crowded" log line from the writer interleaves into the fatal output of the crashed reader goroutine.

Version:4.45

Stack trace (reader/crashed side) and the interleaved writer log

I0904 14:23:59.983556 volume_layout.go:1017 Volume 166143146 becomes crowded
I0904 14:24:02.922182 volume_layout.go:1009 Volume 166143146 becomes uncrowded
I0904 14:24:04.242155 volume_layout.go:1009 Volume 166143152 becomes uncrowded
fatal error: I0904 14:24:07.492703 volume_layout.go:1017 Volume 166143152 becomes crowded
concurrent map read and map write

goroutine 50484648 [running]:
internal/runtime/maps.fatal({0x61bb3f2?, 0xd32e77c?})
	/usr/local/go/src/runtime/panic.go:1195 +0x18
github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(0x3da81e58a480)
	/github/workspace/weed/topology/volume_layout.go:834 +0xc5
github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(0x3da81d0be540, {0xcdcfad0, 0x3da8256d24b0}, 0x3da82589c160)
	/github/workspace/weed/server/master_grpc_server_assign.go:97 +0x2de
github.com/seaweedfs/seaweedfs/weed/pb/master_pb._Seaweed_Assign_Handler.func1({0xcdcfad0?, 0x3da8256d24b0?}, {0xb89f888?, 0x3da82589c160?})
	/github/workspace/weed/pb/master_pb/master_grpc.pb.go:568 +0xcb
github.com/seaweedfs/seaweedfs/weed/pb.requestIDUnaryInterceptor.func1({0xcdcfad0, 0x3da8256d2420}, {0xb89f888, 0x3da82589c160}, 0x2a?, 0x3da81d498150)
	/github/workspace/weed/pb/grpc_client_server.go:321 +0x22e
github.com/seaweedfs/seaweedfs/weed/pb/master_pb._Seaweed_Assign_Handler({0xbb3ff28, 0x3da81d0be540}, {0xcdcfad0, 0x3da8256d2420}, 0x3da81d498120, 0xce321f8)
	/github/workspace/weed/pb/master_pb/master_grpc.pb.go:570 +0x153
google.golang.org/grpc.(*Server).wrapUnaryHandler.func1({0xbb3ff28, 0x3da81d0be540}, {0xcde1918, 0x3da82540be00})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1274 +0xc3
google.golang.org/grpc.(*Server).processRPC(0x3da81daa1688, {0xcdcfad0, 0x3da8256d23c0}, 0x3da8244d01a0, 0x3da81d29d230, 0x3da81e012060, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1433 +0x11b4
google.golang.org/grpc.(*Server).handleStream(0x3da81daa1688, {0xcdd2118, 0x3da81f066d00}, 0x3da8244d01a0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1577 +0xa94
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1086 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 438023
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1097 +0x127

Root cause

weed/topology/volume_layout.go:

func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
	// since delete is guarded by accessLock.Lock(),
	// ...
	// RLock() should be safe enough        <-- the bet
	vl.accessLock.RLock()
	defer vl.accessLock.RUnlock()

	vl.setVolumeCrowded(vid)               // == vl.crowded[vid] = struct{}{} :1015-1016
}

Until #10522 (commit ef6a706, 2026-07-31) nothing read the crowded map by key under the lock, so the write-under-RLock "solved" the problem for the existing readers. #10522 added a per-vid read of vl.crowded inside GetWritableVolumeCount (volume_layout.go:833-838) — precisely on the Assign hot path — turning the assumption into a live race:

for _, vid := range vl.writables {
	if _, ok := vl.crowded[vid]; ok {   // reader under RLock

RWMutex allows concurrent RLock holders, so one heartbeating cluster with any assign traffic and any volume flipping to crowded hits fatal: concurrent map read and map write at random and kills the master. Unlike a panic, a fatal error cannot be recovered.

The event path is asynchronous: volume-state scan -> chanCrowdedVolumes (node.go) -> Topology.SetVolumeCrowded (topology_event_handling.go) -> the RLock writer above.

Suggested fix

Give the event path the write lock; it is a low-frequency single consumer and every other mutation of writables/crowded is already done under Lock():

 func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) {
-	vl.accessLock.RLock()
-	defer vl.accessLock.RUnlock()
+	vl.accessLock.Lock()
+	defer vl.accessLock.Unlock()
 	vl.setVolumeCrowded(vid)
 }

setVolumeCrowded takes no nested locks, so there is no deadlock path. The hot readers (GetWritableVolumeCount, CloneWritableVolumes) keep using RLock.

Reproducer: loop concurrent SetVolumeCrowded(vid) against GetWritableVolumeCount; a -race test flags it immediately, passes (plain and -race) with the lock change.

Impact

Random master process death in steady-state write traffic once any volume becomes crowded. No data corruption (topology is rebuilt after restart), but the master is unavailable until restarted, and fatal errors bypass every recover-based mitigation.

Activity

  1. added a commit that references this issue on Sep 7, 2026
    5d5ea18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions