Skip to content

Commit d02aa24

Browse files
bradseilernpub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2wnpub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
authored
relay: fuzz WebSocket 1012 restart-close timing on graceful drain (BUZZ_DRAIN_JITTER_MS) (#4542)
## Problem On SIGTERM the relay sends every live WebSocket a **1012 Service Restart** close frame via `ConnectionManager::drain_all()` — all in the same instant (`main.rs` shutdown task → `state.rs::drain_all`). On a pod holding thousands of sessions, that makes every client reconnect simultaneously: the thundering-herd reconnect behind the DB pool-timeout bursts observed on each rolling deploy. Client-side jitter can't fix this — the desktop client *resets* its backoff to base on a 1012 and reconnects with only ±25% jitter (`relayClientSession.ts`), so the spread has to come from the server. ## Change Add `BUZZ_DRAIN_JITTER_MS` (default `0` = unchanged behavior). The two paths are kept **deliberately separate** so the default is byte-for-byte the previously shipped shutdown: - **Jitter off (`0`/unset, the default):** the original synchronous, all-at-once `drain_all()` runs unchanged — queue the 1012 on each connection's control channel, cancel, return. No new machinery on the default path. - **Jitter on (`> 0`):** a separate async `drain_all_jittered(jitter_ms)` spreads each connection's restart close over an independent uniform delay in **`[1, jitter_ms]`**. Each delayed close travels a dedicated `RestartClose` channel; the writer flushes the 1012 frame and **acknowledges the flush over a oneshot**, so drain waits for confirmed delivery (up to `RESTART_CLOSE_ACK_TIMEOUT` = 5s) rather than assuming it, falling back to cancellation if the channel is full/closed or the ack times out. The drain future is **owned and awaited** by the shutdown task, and the 30s hard-drain backstop is aborted only after a clean drain — so a clean roll exits `0`. The two methods can be unified and the old one dropped later once the jittered path is proven for all cases. - **`config.rs`** — `drain_jitter_ms`: non-negative parse, clamped to `MAX_DRAIN_JITTER_MS` = **20s** (leaving 10s of the 30s budget for flush). Junk fails loudly at startup; **empty/whitespace-only is treated as unset (jitter off)** so a `BUZZ_DRAIN_JITTER_MS=""` kill switch does not crashloop the relay (matches the sibling env vars in this file). - **`state.rs`** — `drain_all()` (unchanged synchronous default) + `drain_all_jittered()` (jittered + flush-ack). Both set the sticky `draining` flag before the first await. A registration that lands mid-shutdown always self-signals via the **immediate** control-frame + cancel path — jitter smears already-established sockets, not late arrivals. - **`main.rs`** — shutdown task dispatches: `drain_jitter_ms == 0` → `drain_all()`, else `drain_all_jittered(...).await`. ## Safety - **Default off is the currently-committed path.** With jitter unset/0 the shutdown runs the original synchronous `drain_all()` — no restart channel, no ack wait. Safe to deploy dark and dial up. - **Shutdown-boundary race preserved.** Sticky flag set before any await; a late registration self-signals its close with no jitter. - **Owned + backstopped.** The jittered drain future is awaited; the 30s hard-drain `process::exit(1)` remains the ceiling. `MAX_DRAIN_JITTER_MS` (20s) + `RESTART_CLOSE_ACK_TIMEOUT` (5s) = 25s, inside the 30s budget; 5s pre-sleep + 25s = 30s against `terminationGracePeriodSeconds: 60`. ## Known behavior to note (not a blocker, flagged from review) On a **successful** flush the jittered path deliberately does not cancel the connection token — teardown then depends on the client echoing our Close, or on process exit. Compliant clients echo; a silent client rides to the 30s hard exit. The default (jitter-off) path cancels deterministically as before. ## Tests - `config::tests::drain_jitter_defaults_off_and_rejects_junk` — default off, `20000`, clamp `60000`→`20000`, explicit `0`, junk `"soon"` fails, **empty `""` and whitespace-only treated as off**. - `state::tests::drain_all_is_immediate` — default path queues frame + cancels synchronously. - `state::tests::drain_all_sends_restart_close_and_cancels_every_conn`, `drain_all_full_control_buffer_still_cancels`, `register_after_drain_self_signals_restart_close_and_cancel`. - `state::tests::drain_all_jittered_defers_close_until_within_jitter_window` (paused time). - `state::tests::drain_all_jittered_waits_for_writer_acknowledgement_without_cancelling`. - `state::tests::drain_all_jittered_cancels_when_restart_channel_is_full_or_closed`. - `state::tests::drain_all_jittered_cancels_when_flush_ack_times_out` (paused time — the 5s ack-timeout fallback). Validation at `46c690940`: `cargo fmt -p buzz-relay --check`, `cargo clippy -p buzz-relay --all-targets -- -D warnings`, and the drain/config unit suite all clean. Local live SIGTERM test with a real relay process + 200 NIP-42-authenticated sockets — see the PR comment for the before/after distribution and exit codes. ## Rollout Ship with default `0`, then set `BUZZ_DRAIN_JITTER_MS` (e.g. 10000–20000) on bb-block first, watch the roll-window pool-timeout metric, then bb-public. `""` is a safe kill switch. Complements the preStop `sleep` (stops routing before close). --------- Signed-off-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: Brad Seiler <[email protected]> Co-authored-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
1 parent 0661bce commit d02aa24

9 files changed

Lines changed: 631 additions & 24 deletions

File tree

‎TESTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,7 @@ out of the box with `just setup` or `just relay`. Common overrides:
278278
| `BUZZ_REQUIRE_AUTH_TOKEN` | `false` | When true, REST requires NIP-98 (no `X-Pubkey` fallback) |
279279
| `BUZZ_REQUIRE_RELAY_MEMBERSHIP` | `false` | When true, only pubkeys in `relay_members` can connect |
280280
| `BUZZ_REQUIRE_MEDIA_GET_AUTH` | `false` | When true, `GET`/`HEAD /media/*` require Blossom kind 24242 `t=get` auth plus relay membership. |
281+
| `BUZZ_DRAIN_JITTER_MS` | `0` (off) | Per-connection upper bound, in ms, for the random delay before each live WebSocket gets its `1012 Service Restart` close on graceful shutdown. `0` closes every socket at once (the previous behavior). A positive value spreads closes uniformly over `[1, value]` ms to avoid a reconnect thundering herd on rolling deploys. Values above `20000` are capped to `20000` (`MAX_DRAIN_JITTER_MS`) to leave close-frame delivery headroom under the relay's 30s hard-drain timeout. Empty or whitespace-only is treated as unset (off); a non-integer fails startup loudly. |
281282
| `BUZZ_AUDIT_ENABLED` | `true` | Tamper-evident event/media audit log. Set `false`/`0`/`off` to skip its DB pool and writes. Does not disable the separate moderation audit trail. |
282283
| `BUZZ_AUTO_MIGRATE` | `false` | Opt in with `true`/`1`/`yes`/`on` to run embedded SQLx migrations on relay startup |
283284
| `RELAY_OWNER_PUBKEY` | unset | Bootstrapped as `owner` in `relay_members` at first start |

‎crates/buzz-relay/src/config.rs‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,10 @@ pub struct JoinPolicyConfig {
4646
pub version: String,
4747
}
4848

49+
/// Maximum configured jitter, leaving ten seconds of the hard-drain budget for
50+
/// WebSocket close-frame delivery after the final delayed cancellation.
51+
pub const MAX_DRAIN_JITTER_MS: u64 = 20_000;
52+
4953
/// Relay runtime configuration, loaded from environment variables.
5054
#[derive(Debug, Clone)]
5155
pub struct Config {
@@ -60,6 +64,20 @@ pub struct Config {
6064
/// `0` (the default) disables bounded-staleness replica routing; see
6165
/// [`buzz_db::DbConfig::replica_read_max_age_ms`].
6266
pub replica_read_max_age_ms: u64,
67+
68+
/// Upper bound, in milliseconds, of the per-connection random delay applied
69+
/// when sending the `1012 Service Restart` close frame during graceful
70+
/// shutdown (`BUZZ_DRAIN_JITTER_MS`). Each live connection is closed after
71+
/// an independent delay drawn uniformly from `[1, drain_jitter_ms]` when
72+
/// jitter is enabled, which
73+
/// spreads client reconnects across the window instead of releasing the
74+
/// whole pod's sockets in one instant (the reconnect thundering herd that
75+
/// drives DB pool-timeout bursts on rolling deploys).
76+
///
77+
/// Default `0` reproduces the previous all-at-once close. Values above
78+
/// [`MAX_DRAIN_JITTER_MS`] are capped, leaving headroom under the relay's
79+
/// 30-second hard-drain timeout for close-frame delivery.
80+
pub drain_jitter_ms: u64,
6381
/// Redis connection URL used by the pub/sub manager.
6482
pub redis_url: String,
6583
/// Maximum connections in the shared Redis pool. Defaults to 16.
@@ -453,6 +471,25 @@ impl Config {
453471
Err(_) => 0,
454472
};
455473

474+
// Drain jitter: 0 = off (default). Clamp oversized values so every
475+
// delayed close is initiated with ten seconds left in the relay's
476+
// hard-drain budget. An empty/whitespace-only value is treated as unset
477+
// (jitter off), matching the sibling vars in this file — so setting the
478+
// var to "" is a valid kill switch, not a crashloop.
479+
let drain_jitter_ms = match std::env::var("BUZZ_DRAIN_JITTER_MS") {
480+
Ok(raw) if raw.trim().is_empty() => 0,
481+
Ok(raw) => raw
482+
.trim()
483+
.parse::<u64>()
484+
.map_err(|_| {
485+
ConfigError::InvalidValue(
486+
"BUZZ_DRAIN_JITTER_MS must be a non-negative integer".to_string(),
487+
)
488+
})?
489+
.min(MAX_DRAIN_JITTER_MS),
490+
Err(_) => 0,
491+
};
492+
456493
let redis_url =
457494
std::env::var("REDIS_URL").unwrap_or_else(|_| "redis://localhost:6379".to_string());
458495

@@ -934,6 +971,7 @@ impl Config {
934971
database_url,
935972
read_database_url,
936973
replica_read_max_age_ms,
974+
drain_jitter_ms,
937975
redis_url,
938976
redis_pool_size,
939977
db_pool_size,
@@ -1267,6 +1305,60 @@ mod tests {
12671305
}
12681306
}
12691307

1308+
#[test]
1309+
fn drain_jitter_defaults_off_and_rejects_junk() {
1310+
let _guard = ENV_MUTEX.lock().unwrap();
1311+
let previous = std::env::var_os("BUZZ_DRAIN_JITTER_MS");
1312+
1313+
std::env::remove_var("BUZZ_DRAIN_JITTER_MS");
1314+
let unset = Config::from_env().expect("config").drain_jitter_ms;
1315+
1316+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", "20000");
1317+
let set = Config::from_env().expect("config").drain_jitter_ms;
1318+
1319+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", "60000");
1320+
let capped = Config::from_env().expect("config").drain_jitter_ms;
1321+
1322+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", "0");
1323+
let zero = Config::from_env().expect("config").drain_jitter_ms;
1324+
1325+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", "soon");
1326+
let junk = Config::from_env();
1327+
1328+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", "");
1329+
let empty = Config::from_env()
1330+
.expect("empty is a valid kill switch")
1331+
.drain_jitter_ms;
1332+
1333+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", " ");
1334+
let blank = Config::from_env()
1335+
.expect("whitespace-only is a valid kill switch")
1336+
.drain_jitter_ms;
1337+
1338+
if let Some(value) = previous {
1339+
std::env::set_var("BUZZ_DRAIN_JITTER_MS", value);
1340+
} else {
1341+
std::env::remove_var("BUZZ_DRAIN_JITTER_MS");
1342+
}
1343+
1344+
assert_eq!(unset, 0, "drain jitter must default off");
1345+
assert_eq!(set, MAX_DRAIN_JITTER_MS);
1346+
assert_eq!(
1347+
capped, MAX_DRAIN_JITTER_MS,
1348+
"oversized jitter leaves close-frame flush headroom"
1349+
);
1350+
assert_eq!(zero, 0, "explicit 0 is off");
1351+
assert!(
1352+
junk.is_err(),
1353+
"an unparsable jitter must fail loudly, not silently disable"
1354+
);
1355+
assert_eq!(
1356+
empty, 0,
1357+
"an empty value is treated as unset — a kill switch, not a crashloop"
1358+
);
1359+
assert_eq!(blank, 0, "a whitespace-only value is treated as unset");
1360+
}
1361+
12701362
#[test]
12711363
fn audit_logging_defaults_on_and_accepts_explicit_off() {
12721364
let _guard = ENV_MUTEX.lock().unwrap();

‎crates/buzz-relay/src/connection.rs‎

Lines changed: 88 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@ const AUTH_TIMEOUT: Duration = Duration::from_secs(5);
2929
/// Shared mutable subscription map for a single WebSocket connection.
3030
pub(crate) type ConnectionSubscriptions = Arc<Mutex<HashMap<String, Vec<Filter>>>>;
3131

32+
/// Request for the writer to flush a restart close and report the result.
33+
pub(crate) struct RestartClose {
34+
pub(crate) flushed: tokio::sync::oneshot::Sender<bool>,
35+
}
36+
3237
/// Maximum outbound data frames buffered into the websocket sink before one flush.
3338
const MAX_WS_SEND_BATCH: usize = 64;
3439

@@ -161,6 +166,11 @@ async fn handle_active_connection(
161166
// even when the data buffer is full.
162167
let (ctrl_tx, ctrl_rx) = mpsc::channel::<WsMessage>(8);
163168

169+
// Dedicated restart-close channel carries a flush acknowledgement. Keeping
170+
// ordinary control frames unchanged avoids coupling heartbeat/ban traffic
171+
// to graceful-shutdown delivery tracking.
172+
let (restart_tx, restart_rx) = mpsc::channel::<RestartClose>(1);
173+
164174
let backpressure_count = Arc::new(AtomicU8::new(0));
165175
let subscriptions = Arc::new(Mutex::new(HashMap::new()));
166176

@@ -205,6 +215,7 @@ async fn handle_active_connection(
205215
conn_id,
206216
tx.clone(),
207217
ctrl_tx.clone(),
218+
Some(restart_tx),
208219
cancel.clone(),
209220
conn.tenant.community(),
210221
Arc::clone(&backpressure_count),
@@ -215,7 +226,7 @@ async fn handle_active_connection(
215226
let (ws_send, ws_recv) = socket.split();
216227

217228
let send_cancel = cancel.child_token();
218-
let send_task = tokio::spawn(send_loop(ws_send, rx, ctrl_rx, send_cancel));
229+
let send_task = tokio::spawn(send_loop(ws_send, rx, ctrl_rx, restart_rx, send_cancel));
219230

220231
let missed_pongs = Arc::new(AtomicU8::new(0));
221232
let heartbeat_cancel = cancel.clone();
@@ -297,15 +308,17 @@ async fn send_loop(
297308
ws_send: futures_util::stream::SplitSink<WebSocket, WsMessage>,
298309
data_rx: mpsc::Receiver<WsMessage>,
299310
ctrl_rx: mpsc::Receiver<WsMessage>,
311+
restart_rx: mpsc::Receiver<RestartClose>,
300312
cancel: CancellationToken,
301313
) {
302-
send_loop_inner(ws_send, data_rx, ctrl_rx, cancel).await;
314+
send_loop_inner(ws_send, data_rx, ctrl_rx, restart_rx, cancel).await;
303315
}
304316

305317
async fn send_loop_inner<S>(
306318
mut ws_send: S,
307319
mut data_rx: mpsc::Receiver<WsMessage>,
308320
mut ctrl_rx: mpsc::Receiver<WsMessage>,
321+
mut restart_rx: mpsc::Receiver<RestartClose>,
309322
cancel: CancellationToken,
310323
) where
311324
S: Sink<WsMessage> + Unpin,
@@ -319,9 +332,21 @@ async fn send_loop_inner<S>(
319332
}
320333

321334
tokio::select! {
322-
// Biased: cancel > control > data. Cancel must win immediately
323-
// so backpressure-triggered shutdown isn't starved by queued data.
335+
// Biased: restart > cancel > ordinary control > data. A restart
336+
// command owns shutdown delivery and must flush its 1012 before
337+
// cancellation can fall back to an unacknowledged close.
324338
biased;
339+
Some(restart) = restart_rx.recv() => {
340+
let sent = ws_send
341+
.send(WsMessage::Close(Some(axum::extract::ws::CloseFrame {
342+
code: axum::extract::ws::close_code::RESTART,
343+
reason: axum::extract::ws::Utf8Bytes::from_static("relay restarting"),
344+
})))
345+
.await
346+
.is_ok();
347+
let _ = restart.flushed.send(sent);
348+
break;
349+
}
325350
_ = cancel.cancelled() => {
326351
// Drain any queued control frames before closing. A ban
327352
// disconnect queues its `OK false "blocked: …"` reason frame on
@@ -797,7 +822,8 @@ mod tests {
797822
}
798823

799824
let (sink, state) = MockSink::new(Some(1));
800-
send_loop_inner(sink, data_rx, ctrl_rx, CancellationToken::new()).await;
825+
let (_restart_tx, restart_rx) = mpsc::channel(1);
826+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
801827

802828
let state = state.lock().expect("mock sink poisoned");
803829
assert_eq!(state.flush_count, 1);
@@ -817,7 +843,8 @@ mod tests {
817843
.expect("queue data frame");
818844

819845
let (sink, state) = MockSink::new(Some(1));
820-
send_loop_inner(sink, data_rx, ctrl_rx, CancellationToken::new()).await;
846+
let (_restart_tx, restart_rx) = mpsc::channel(1);
847+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
821848

822849
let state = state.lock().expect("mock sink poisoned");
823850
assert_eq!(state.flush_count, 1);
@@ -842,7 +869,8 @@ mod tests {
842869
.expect("queue control frame");
843870

844871
let (sink, state) = MockSink::new(Some(2));
845-
send_loop_inner(sink, data_rx, ctrl_rx, CancellationToken::new()).await;
872+
let (_restart_tx, restart_rx) = mpsc::channel(1);
873+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
846874

847875
let state = state.lock().expect("mock sink poisoned");
848876
assert_eq!(state.flush_count, 2);
@@ -852,6 +880,57 @@ mod tests {
852880
);
853881
}
854882

883+
#[tokio::test]
884+
async fn send_loop_acknowledges_restart_after_flushing_exactly_one_1012() {
885+
let (_data_tx, data_rx) = mpsc::channel(1);
886+
let (_ctrl_tx, ctrl_rx) = mpsc::channel(1);
887+
let (restart_tx, restart_rx) = mpsc::channel(1);
888+
let (flushed_tx, flushed_rx) = tokio::sync::oneshot::channel();
889+
restart_tx
890+
.send(RestartClose {
891+
flushed: flushed_tx,
892+
})
893+
.await
894+
.expect("queue restart close");
895+
896+
let (sink, state) = MockSink::new(None);
897+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
898+
899+
assert_eq!(flushed_rx.await, Ok(true));
900+
let state = state.lock().expect("mock sink poisoned");
901+
assert_eq!(state.flush_count, 1, "ack follows the close flush");
902+
assert_eq!(state.messages.len(), 1, "writer exits after restart close");
903+
match &state.messages[0] {
904+
WsMessage::Close(Some(close)) => {
905+
assert_eq!(close.code, axum::extract::ws::close_code::RESTART);
906+
assert_eq!(close.reason.as_str(), "relay restarting");
907+
}
908+
other => panic!("expected one 1012 restart close, got {other:?}"),
909+
}
910+
}
911+
912+
#[tokio::test]
913+
async fn send_loop_reports_restart_flush_failure() {
914+
let (_data_tx, data_rx) = mpsc::channel(1);
915+
let (_ctrl_tx, ctrl_rx) = mpsc::channel(1);
916+
let (restart_tx, restart_rx) = mpsc::channel(1);
917+
let (flushed_tx, flushed_rx) = tokio::sync::oneshot::channel();
918+
restart_tx
919+
.send(RestartClose {
920+
flushed: flushed_tx,
921+
})
922+
.await
923+
.expect("queue restart close");
924+
925+
let (sink, state) = MockSink::new(Some(1));
926+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
927+
928+
assert_eq!(flushed_rx.await, Ok(false));
929+
let state = state.lock().expect("mock sink poisoned");
930+
assert_eq!(state.flush_count, 1);
931+
assert_eq!(state.messages.len(), 1, "no fallback close is appended");
932+
}
933+
855934
#[tokio::test]
856935
async fn send_loop_flushes_queued_control_before_close_on_cancel() {
857936
// A ban disconnect queues its `OK false "blocked: …"` reason frame on
@@ -871,7 +950,8 @@ mod tests {
871950
cancel.cancel();
872951

873952
let (sink, state) = MockSink::new(None);
874-
send_loop_inner(sink, data_rx, ctrl_rx, cancel).await;
953+
let (_restart_tx, restart_rx) = mpsc::channel(1);
954+
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, cancel).await;
875955

876956
let state = state.lock().expect("mock sink poisoned");
877957
assert_eq!(

‎crates/buzz-relay/src/handlers/event.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1459,6 +1459,7 @@ mod tests {
14591459
conn_id,
14601460
tx,
14611461
ctrl_tx,
1462+
None,
14621463
CancellationToken::new(),
14631464
buzz_core::tenant::CommunityId::from_uuid(Uuid::nil()),
14641465
Arc::new(AtomicU8::new(0)),
@@ -2098,6 +2099,7 @@ mod tests {
20982099
conn_id,
20992100
tx,
21002101
ctrl_tx,
2102+
None,
21012103
CancellationToken::new(),
21022104
buzz_core::tenant::CommunityId::from_uuid(Uuid::nil()),
21032105
Arc::new(AtomicU8::new(0)),
@@ -2423,6 +2425,7 @@ mod tests {
24232425
conn_id,
24242426
tx,
24252427
ctrl_tx,
2428+
None,
24262429
CancellationToken::new(),
24272430
community_id,
24282431
Arc::new(AtomicU8::new(0)),

0 commit comments

Comments
 (0)