fix(relay): never close connections when cross-node fan-out falls behind - #9302

Summary

Cross-node fan-out no longer closes connections when it falls behind.

Before this change, an event closed its local recipients when:

• it was not delivered within 30 s of this node reading it from Redis,
• its worker queue was full,
• or the access gate still could not check a recipient by that deadline.

Dispatcher lag also closed every subscribed connection. A rate-limited closer spread the closes out, at 1% of a node's connections per second with a floor.

The trouble is that a database stall slows every node's deliveries at once. So the closes landed on every node together. Each reconnect then does AUTH, membership reads, NIP-11 and REST catch-up against the database that was already slow. That turned a short stall into a reconnect storm and bursts of 503s. On small nodes the closer's floor dominated, so a node could close all of its connections within tens of seconds. Review-Proven Rule 9 (design every client-driven path for the whole fleet running it at once) is the reason: a server-forced close starts a fleet-wide reconnect.

What changes

Kept from #9195 and #9235:

• One dispatcher plus 16 workers, routed by (community, topic). Each channel stays in order and channels deliver concurrently.
• The bulk membership read in the access gate, and denying an event for a channel that no longer exists.

Removed:

• The delivery deadline and ChannelEvent.received_at.
• The live fence (fence_live_delivery, live_fenced).
• LagCloser and its rate limit.
• The queue-full closes and the dispatcher-lag closes.
• The access-gate retry loop (buzz_multinode_fanout_gate_retries_total).

Behavior now:

• A slow delivery holds only its own worker and arrives late.
• A full worker queue or dispatcher lag drops the event on this node. The drop is counted in buzz_multinode_fanout_dropped_total{reason="queue_full"} or buzz_multinode_fanout_lag_total, and the affected artifact syncs get CLOSED "resync: live delivery lagged" (NIP-AR Sync rule 5). This is the same as before the closer existed.
• A recipient the access gate cannot check misses that event, and a sync among them resyncs. Same as the local fan-out path.

Trade-off: a plain subscription that misses a dropped event is not told. Its client may replay the event on its next reconnect, but not if later deliveries moved its resume cursor past the event by more than its replay slack (5 s for the ACP client). That was the behavior before #9195. Queue-full and lag drops stay visible in the counters, so we can see whether they actually happen before adding any recovery mechanism.

What got simpler

Net −700 lines.

• multinode_fanout.rs drops from ~940 lines to ~350.
• deliver_pubsub_event loses its retry loop and goes back to the shared route_fanout_matches path. The Redis path and the local path once again apply the access gate the same way.
• ConnectionManager loses the fence flag and the split send helper. SubscriptionRegistry loses subscribed_connections.

Testing

• handlers::multinode_fanout::tests:
  • in-order delivery through the workers;
  • a full queue resyncs only the matched syncs and closes nothing, and a local echo is not counted as a drop;
  • dispatcher lag resyncs every sync and closes nothing;
  • a recipient the gate cannot check misses the event with no retry, stays connected, and still gets the next event.
• handlers::event::tests is unchanged apart from removing received_at.
• Mutants killed: the queue-full resync, the lag resync and the local-echo skip.

🤖 Generated with Claude Code