Skip to content

Commit 4c2448c

Browse files
loganjLarryclaude
committed
Bound roster republishes and tighten the 0074 rollout
Review fixes for the computed member/bot label: - Background roster republishes share a relay-wide permit (two at once). Each one holds a writer connection while it waits for a channel's membership lock, so a burst of first agent sign-ins sharing channels takes at most two pool connections. - BUZZ_RECONCILE_BOT_ROSTERS is a true/false flag, parsed like BUZZ_AUTO_MIGRATE: `0` and `false` leave it off. - 0074 uses the 0049 bounds (lock_timeout 1s, statement_timeout 5s). Its comment now says that DROP TRIGGER blocks event reads as well as writes. - The role label doc names the one way an owner can be cleared: the users foreign key is ON DELETE SET NULL. - The Helm README lists the accepted flag values and the log line that confirms the one-shot repair ran. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Logan Johnson <[email protected]>
1 parent 143e4c7 commit 4c2448c

8 files changed

Lines changed: 89 additions & 38 deletions

File tree

‎crates/buzz-db/src/runtime/migration.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -782,7 +782,8 @@ mod postgres_tests {
782782
}
783783
let retire_roster_fence = strip_sql_comments(migrations[75].sql.as_str());
784784
for statement in [
785-
"SET LOCAL lock_timeout = '5s';",
785+
"SET LOCAL lock_timeout = '1s';",
786+
"SET LOCAL statement_timeout = '5s';",
786787
"DROP TRIGGER trg_events_guard_channel_roster_snapshot ON events;",
787788
"DROP FUNCTION guard_channel_roster_snapshot();",
788789
] {

‎crates/buzz-db/src/store/channel_members.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,12 @@ pub struct MemberRecord {
4242
/// `owner`, `admin` and `guest` show as stored. An agent owner is recorded
4343
/// once and never changes, so the label of a key changes at most once.
4444
///
45+
/// One exception: the `users` foreign key to the owner is `ON DELETE SET
46+
/// NULL`, so deleting the owner's `users` row clears the agent's owner. Today
47+
/// only community deletion deletes `users` rows, and it removes every row of
48+
/// the community. A user deletion that purges `users` rows must republish the
49+
/// rosters of the deleted owner's agents, or they keep showing `bot`.
50+
///
4551
/// Every read of a member's role uses this, so rosters, member lists and role
4652
/// checks agree.
4753
macro_rules! member_role_label {

‎crates/buzz-relay/src/handlers/auth.rs‎

Lines changed: 37 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2399,10 +2399,14 @@ mod tests {
23992399
/// First AUTH with a NIP-OA credential records the owner, which makes
24002400
/// the agent's ordinary rows show `bot`. The rosters must follow, but
24012401
/// a contended roster lock must not hold AUTH: the pending AUTH
2402-
/// expires after 5 s, the same as the default lock timeout.
2402+
/// expires after 5 s, the same as the default lock timeout. The
2403+
/// republish also waits for a relay-wide permit, so a burst of first
2404+
/// sign-ins holds only a few writer connections.
24032405
///
2404-
/// Mutation: await the roster republish inside
2405-
/// `materialize_nip_oa_owner` → AUTH waits on the held lock → RED.
2406+
/// Mutations: await the roster republish inside
2407+
/// `materialize_nip_oa_owner` → AUTH waits on the held lock → RED;
2408+
/// skip the republish permit → the roster changes while every permit
2409+
/// is held → RED.
24062410
#[tokio::test]
24072411
#[ignore = "requires Postgres — runs in postgres-ci nextest lane"]
24082412
async fn first_auth_is_not_held_by_a_contended_roster_lock() {
@@ -2444,7 +2448,12 @@ mod tests {
24442448
.await
24452449
.expect("add unowned agent");
24462450

2447-
// Hold the roster lock the republish needs.
2451+
// Hold every republish permit and the roster lock the republish
2452+
// needs.
2453+
let permits = Arc::clone(&state.roster_republish_semaphore)
2454+
.acquire_many_owned(crate::state::ROSTER_REPUBLISH_CONCURRENCY as u32)
2455+
.await
2456+
.expect("hold every republish permit");
24482457
let relay_pubkey = state.relay_keypair.public_key().to_bytes();
24492458
let held = state
24502459
.db
@@ -2480,12 +2489,8 @@ mod tests {
24802489
.await
24812490
.expect("read owner"));
24822491

2483-
// Once the lock is free, the background republish names the agent
2484-
// `bot` in the live roster.
2485-
held.release().await.expect("release roster lock");
24862492
let agent_hex = agent.public_key().to_hex();
2487-
let deadline = Instant::now() + std::time::Duration::from_secs(10);
2488-
loop {
2493+
let live_roster = || async {
24892494
let live: Option<serde_json::Value> = sqlx::query_scalar(
24902495
"SELECT tags FROM events WHERE community_id = $1 AND channel_id = $2 \
24912496
AND kind = 39002 AND deleted_at IS NULL",
@@ -2495,15 +2500,34 @@ mod tests {
24952500
.fetch_optional(state.db.pool())
24962501
.await
24972502
.expect("read live roster");
2498-
let is_bot = live
2499-
.as_ref()
2503+
live
2504+
};
2505+
let shows_bot = |live: &Option<serde_json::Value>| {
2506+
live.as_ref()
25002507
.and_then(|tags| tags.as_array())
25012508
.is_some_and(|tags| {
25022509
tags.iter().any(|tag| {
25032510
tag[0] == "p" && tag[1] == agent_hex.as_str() && tag[3] == "bot"
25042511
})
2505-
});
2506-
if is_bot {
2512+
})
2513+
};
2514+
2515+
// With the roster lock free but every permit held, the republish
2516+
// still waits.
2517+
held.release().await.expect("release roster lock");
2518+
let before = live_roster().await;
2519+
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
2520+
let waiting = live_roster().await;
2521+
assert_eq!(waiting, before, "no republish runs without a permit");
2522+
assert!(!shows_bot(&waiting), "{waiting:?}");
2523+
2524+
// Once a permit is free, the background republish names the agent
2525+
// `bot` in the live roster.
2526+
drop(permits);
2527+
let deadline = Instant::now() + std::time::Duration::from_secs(10);
2528+
loop {
2529+
let live = live_roster().await;
2530+
if shows_bot(&live) {
25072531
break;
25082532
}
25092533
assert!(

‎crates/buzz-relay/src/handlers/side_effects.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3824,7 +3824,9 @@ pub async fn reconcile_ordinary_role_rosters(
38243824
}
38253825

38263826
/// Republish each `(community, host, channel)` roster under its membership
3827-
/// lock, counting failures instead of stopping.
3827+
/// lock, counting failures instead of stopping. Each roster first takes a
3828+
/// relay-wide republish permit, so concurrent callers share a small bound on
3829+
/// the writer connections they hold.
38283830
async fn republish_roster_candidates(
38293831
state: &Arc<AppState>,
38303832
candidates: Vec<(buzz_core::CommunityId, String, uuid::Uuid)>,
@@ -3837,6 +3839,9 @@ async fn republish_roster_candidates(
38373839

38383840
for (community_id, host, channel_id) in candidates {
38393841
let result = async {
3842+
let _permit = Arc::clone(&state.roster_republish_semaphore)
3843+
.acquire_owned()
3844+
.await?;
38403845
// Hold the membership-writer lock from roster capture through
38413846
// replacement. Otherwise a rolling deployment can publish stale
38423847
// roster A after another relay commits and publishes roster B.

‎crates/buzz-relay/src/main.rs‎

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@ use buzz_relay::telemetry;
3030
use buzz_workflow::WorkflowEngine;
3131
use tokio_util::sync::CancellationToken;
3232

33-
fn buzz_auto_migrate_enabled(value: Option<&str>) -> bool {
33+
/// Whether an opt-in boolean environment flag is on: `true`, `1`, `yes` or
34+
/// `on`, in any case. Unset, empty and every other value mean off.
35+
fn env_flag_enabled(value: Option<&str>) -> bool {
3436
value.map(str::trim).is_some_and(|value| {
3537
matches!(
3638
value.to_ascii_lowercase().as_str(),
@@ -545,8 +547,7 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
545547
info!("Postgres connected");
546548
}
547549

548-
let auto_migrate =
549-
buzz_auto_migrate_enabled(std::env::var("BUZZ_AUTO_MIGRATE").ok().as_deref());
550+
let auto_migrate = env_flag_enabled(std::env::var("BUZZ_AUTO_MIGRATE").ok().as_deref());
550551
if auto_migrate {
551552
let step = StepTimer::start(StartupStep::DbMigrate);
552553
db.migrate().await.map_err(|e| {
@@ -1141,7 +1142,7 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
11411142
// labels: republish rosters that older relays published with the stored
11421143
// label. Off by default so no replica scans at every startup; both labels
11431144
// have the same permissions, so a stale one is only cosmetic until then.
1144-
if std::env::var("BUZZ_RECONCILE_BOT_ROSTERS").is_ok() {
1145+
if env_flag_enabled(std::env::var("BUZZ_RECONCILE_BOT_ROSTERS").ok().as_deref()) {
11451146
let role_roster_state = Arc::clone(&state);
11461147
tokio::spawn(async move {
11471148
match buzz_relay::handlers::side_effects::reconcile_ordinary_role_rosters(
@@ -3125,8 +3126,8 @@ mod tests {
31253126
use uuid::Uuid;
31263127

31273128
use super::{
3128-
apply_activity_refresh, apply_stock_refresh, buzz_auto_migrate_enabled, connect_audit_pool,
3129-
counted_partition_audit, dropped_in_memory_keys, emit_cached_fleet_usage_metrics,
3129+
apply_activity_refresh, apply_stock_refresh, connect_audit_pool, counted_partition_audit,
3130+
dropped_in_memory_keys, emit_cached_fleet_usage_metrics, env_flag_enabled,
31303131
idle_timeout_secs, jwks_next_retry_after_failed_refresh, nip_fi_jwks_refresh_loop,
31313132
record_legacy_usage_collection, refresh_due_fleet_usage,
31323133
refresh_legacy_active_gauge_recency, relay_keypair_from_config,
@@ -3495,18 +3496,18 @@ mod tests {
34953496
}
34963497

34973498
#[test]
3498-
fn buzz_auto_migrate_is_opt_in() {
3499-
assert!(!buzz_auto_migrate_enabled(None));
3500-
assert!(!buzz_auto_migrate_enabled(Some("")));
3501-
assert!(!buzz_auto_migrate_enabled(Some("false")));
3502-
assert!(!buzz_auto_migrate_enabled(Some("0")));
3503-
assert!(!buzz_auto_migrate_enabled(Some("no")));
3504-
3505-
assert!(buzz_auto_migrate_enabled(Some("true")));
3506-
assert!(buzz_auto_migrate_enabled(Some("TRUE")));
3507-
assert!(buzz_auto_migrate_enabled(Some(" 1 ")));
3508-
assert!(buzz_auto_migrate_enabled(Some("yes")));
3509-
assert!(buzz_auto_migrate_enabled(Some("on")));
3499+
fn env_flags_are_opt_in() {
3500+
assert!(!env_flag_enabled(None));
3501+
assert!(!env_flag_enabled(Some("")));
3502+
assert!(!env_flag_enabled(Some("false")));
3503+
assert!(!env_flag_enabled(Some("0")));
3504+
assert!(!env_flag_enabled(Some("no")));
3505+
3506+
assert!(env_flag_enabled(Some("true")));
3507+
assert!(env_flag_enabled(Some("TRUE")));
3508+
assert!(env_flag_enabled(Some(" 1 ")));
3509+
assert!(env_flag_enabled(Some("yes")));
3510+
assert!(env_flag_enabled(Some("on")));
35103511
}
35113512

35123513
#[test]

‎crates/buzz-relay/src/state.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1207,6 +1207,10 @@ pub const THREAD_SUMMARY_REFRESH_INTERVAL: std::time::Duration = std::time::Dura
12071207
/// local fan-out). A hung refresh is abandoned so its root keeps emitting.
12081208
pub const THREAD_SUMMARY_REFRESH_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
12091209

1210+
/// Background roster republishes that may run at once. See
1211+
/// [`AppState::roster_republish_semaphore`].
1212+
pub const ROSTER_REPUBLISH_CONCURRENCY: usize = 2;
1213+
12101214
/// Shared application state, cloned cheaply via inner `Arc` fields.
12111215
#[derive(Clone)]
12121216
pub struct AppState {
@@ -1247,6 +1251,11 @@ pub struct AppState {
12471251
pub git_semaphore: Arc<Semaphore>,
12481252
/// Semaphore limiting concurrent media upload parsing/transcoding work.
12491253
pub media_upload_semaphore: Arc<Semaphore>,
1254+
/// Semaphore limiting background roster republishes across the relay.
1255+
/// Each one holds a writer connection while it waits for a channel's
1256+
/// membership lock, so a burst of first agent sign-ins sharing channels
1257+
/// must not take more than a few pool connections.
1258+
pub roster_republish_semaphore: Arc<Semaphore>,
12501259

12511260
/// Workflow engine for background processing.
12521261
pub workflow_engine: Arc<WorkflowEngine>,
@@ -1565,6 +1574,7 @@ impl AppState {
15651574
handler_semaphore: Arc::new(Semaphore::new(max_concurrent_handlers)),
15661575
git_semaphore: Arc::new(Semaphore::new(git_max_concurrent_ops)),
15671576
media_upload_semaphore: Arc::new(Semaphore::new(media_max_concurrent_uploads)),
1577+
roster_republish_semaphore: Arc::new(Semaphore::new(ROSTER_REPUBLISH_CONCURRENCY)),
15681578
workflow_engine,
15691579
relay_keypair,
15701580
huddle_liveness_generation: Uuid::new_v4(),

‎deploy/charts/buzz/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -616,7 +616,7 @@ default so long-lived WebSocket connections have time to drain.
616616

617617
Schema migrations are embedded in the relay binary via `sqlx::migrate!` and run at startup, gated by `BUZZ_AUTO_MIGRATE` (default `true`). Multiple replicas race-safely behind a Postgres advisory lock. `helm upgrade` is the entire upgrade procedure.
618618

619-
Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. Older relays verify its roster-fence trigger before opening listeners and refuse to start if the trigger is missing or inert. Newer relays run the same check in every roster writer and do not require the trigger. Migration 0074 drops the trigger, so every relay must run a version without the startup check (#8611 or later) before 0074 is applied; an older relay then refuses to start. Relays from the 0074 release on show a computed `member`/`bot` label in rosters that the trigger would refuse, so apply 0074 before rolling them out. After that rollout, start one relay once with `BUZZ_RECONCILE_BOT_ROSTERS=1` to republish rosters that older relays published with the stored label. Apply migrations before rolling the relay; for large installations, prefer a controlled `buzz-admin migrate` job with PostgreSQL lock monitoring before the code rollout.
619+
Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. Older relays verify its roster-fence trigger before opening listeners and refuse to start if the trigger is missing or inert. Newer relays run the same check in every roster writer and do not require the trigger. Migration 0074 drops the trigger, so every relay must run a version without the startup check (#8611 or later) before 0074 is applied; an older relay then refuses to start. Relays from the 0074 release on show a computed `member`/`bot` label in rosters that the trigger would refuse, so apply 0074 before rolling them out. After that rollout, start one relay once with `BUZZ_RECONCILE_BOT_ROSTERS=1` (`true`, `yes` and `on` also work; any other value leaves it off) to republish rosters that older relays published with the stored label. Confirm that it logs `member/bot channel rosters reconciled`; a warning instead means the repair did not run, so start it again. Apply migrations before rolling the relay; for large installations, prefer a controlled `buzz-admin migrate` job with PostgreSQL lock monitoring before the code rollout.
620620

621621
If you prefer decoupling migrations from serving, set `migrate.autoMigrate=false`. **In that mode the chart does not run migrations for you** — you own running `buzz-admin migrate` (separate Pod / one-shot Job) against the database before every `helm install` / `helm upgrade`. Readiness probes only verify DB connectivity, not schema freshness, so a pod will appear healthy against an unmigrated schema and fail under load. A pre-upgrade Helm Job for this is on the chart roadmap; the values knob `migrate.preUpgradeJob.enabled` is reserved.
622622

‎migrations/0076_retire_channel_roster_trigger.sql‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,14 @@
77
-- later before this migration runs; older relays refuse to start without the
88
-- trigger. Published migrations remain checksum-frozen.
99
--
10-
-- DROP TRIGGER locks the partitioned parent and its partitions. Bound the
11-
-- lock acquisition wait. This does not bound statement duration or the time
12-
-- ingestion can be blocked while these locks are held.
13-
SET LOCAL lock_timeout = '5s';
10+
-- DROP TRIGGER takes ACCESS EXCLUSIVE on the partitioned parent and on each
11+
-- partition in turn, and keeps each lock while it waits for the next. While
12+
-- the locks are queued or held, event reads and writes both block.
13+
-- lock_timeout bounds each lock wait and statement_timeout bounds each whole
14+
-- statement, as in 0049. Either timeout rolls the migration back with no
15+
-- change, and it can be retried.
16+
SET LOCAL lock_timeout = '1s';
17+
SET LOCAL statement_timeout = '5s';
1418

1519
DROP TRIGGER trg_events_guard_channel_roster_snapshot ON events;
1620
DROP FUNCTION guard_channel_roster_snapshot();

0 commit comments

Comments
 (0)