libn/networkdb: don't exceed broadcast size limit

corhere · corhere · commit dacf44561421 · 2025-05-07T12:32:54.000-04:00
NetworkDB uses a hierarchy of queues to prioritize messages for
broadcast. Unfortunately the logic to pull from multiple queues is
flawed. The length of the messages pulled from the first queue is not
taken into account when pulling messages from the second queue. A list
of messages up to tiwce as long as the limit could be returned! Messages
beyond the limit will be truncated unceremoniously by memberlist.

Memberlist broadcast queues assume that all messages returned from a
GetBroadcasts call will be broadcasted to other nodes in the cluster.
Messages are popped from the queue once they have hit their retransmit
limit. On a busy system messages may be broadcast fewer times than
intended, possibly even being dropped without ever being broadcast!

Subtract the length of messages pulled from the first queue from the
broadcast size limit so the limit is not exceeded when pulling from the
second queue.

Signed-off-by: Cory Snider &lt;csnider@mirantis.com&gt;
diff --git a/libnetwork/networkdb/delegate.go b/libnetwork/networkdb/delegate.go
@@ -408,7 +408,12 @@ func (d *delegate) NotifyMsg(buf []byte) {
 
 func (d *delegate) GetBroadcasts(overhead, limit int) [][]byte {
 	msgs := d.nDB.networkBroadcasts.GetBroadcasts(overhead, limit)
-	msgs = append(msgs, d.nDB.nodeBroadcasts.GetBroadcasts(overhead, limit)...)
+	for _, m := range msgs {
+		limit -= overhead + len(m)
+	}
+	if limit > 0 {
+		msgs = append(msgs, d.nDB.nodeBroadcasts.GetBroadcasts(overhead, limit)...)
+	}
 	return msgs
 }
 
