Skip to content

Commit a4311bb

Browse files
loganjLarryclaude
committed
Refuse to start while the 0032 roster trigger is installed
This relay shows computed member/bot roster labels, which the 0032 trigger refuses. If 0074 has not run (autoMigrate off and the migration skipped, or 0074 timed out), membership changes would still commit but rosters with an owned ordinary member would stop updating, with only warnings in the log. Check the catalog at startup and refuse to serve until 0074 has dropped the trigger, the inverse of the check #8611 removed. It is one catalog query. Also, from review: the 0074 comment says the ACCESS EXCLUSIVE locks are on the partitions, and the first-auth test doc says the held permits stop the republish before it reaches the roster lock. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Logan Johnson <[email protected]>
1 parent 4c2448c commit a4311bb

6 files changed

Lines changed: 85 additions & 16 deletions

File tree

‎crates/buzz-db/src/store/channel_members.rs‎

Lines changed: 60 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -526,6 +526,31 @@ impl LockedMemberSnapshot {
526526
}
527527
}
528528

529+
/// Refuse to serve until migration 0074 has retired the 0032 roster trigger.
530+
///
531+
/// This relay shows the computed `member`/`bot` label in kind:39002 rosters.
532+
/// The 0032 trigger compares roster labels with the stored role exactly, so
533+
/// while it is installed it refuses every roster that shows `bot` for a stored
534+
/// `member`. Membership changes would still commit, but those channels'
535+
/// rosters would stop updating.
536+
pub async fn verify_channel_roster_trigger_retired<'e>(
537+
executor: impl sqlx::PgExecutor<'e>,
538+
) -> Result<()> {
539+
let installed: bool = sqlx::query_scalar(
540+
"SELECT to_regprocedure('public.guard_channel_roster_snapshot()') IS NOT NULL \
541+
OR EXISTS (SELECT 1 FROM pg_trigger \
542+
WHERE tgname = 'trg_events_guard_channel_roster_snapshot')",
543+
)
544+
.fetch_one(executor)
545+
.await?;
546+
if installed {
547+
return Err(DbError::InvalidData(
548+
"the 0032 channel roster trigger is still installed; apply migration 0074".into(),
549+
));
550+
}
551+
Ok(())
552+
}
553+
529554
/// Capture all active members while holding the same per-channel lock used by
530555
/// membership writers.
531556
///
@@ -2200,6 +2225,18 @@ pub async fn get_member_role(
22002225
}
22012226

22022227
impl Db {
2228+
/// Fail unless migration 0074 has retired the 0032 roster trigger. See
2229+
/// [`verify_channel_roster_trigger_retired`].
2230+
#[datastore_span(name = "verify_channel_roster_trigger_retired", system = "postgresql")]
2231+
pub async fn verify_channel_roster_trigger_retired(&self) -> Result<()> {
2232+
let mut connection = crate::observability::acquire_writer(
2233+
&self.pool,
2234+
crate::observability::WriterOperation::Bootstrap,
2235+
)
2236+
.await?;
2237+
verify_channel_roster_trigger_retired(&mut *connection).await
2238+
}
2239+
22032240
/// Capture the active roster while holding the membership-writer lock.
22042241
#[datastore_span(name = "lock_member_snapshot", system = "postgresql")]
22052242
pub async fn lock_member_snapshot(
@@ -5258,12 +5295,6 @@ mod postgres_tests {
52585295
(pool, name)
52595296
}
52605297

5261-
/// Create a fresh scratch database on the same server and run all migrations.
5262-
/// Returns (pool, db_name); callers should `drop_scratch_db` when done.
5263-
async fn create_scratch_db(admin: &PgPool, prefix: &str) -> (PgPool, String) {
5264-
create_scratch_db_through(admin, prefix, None).await
5265-
}
5266-
52675298
async fn drop_scratch_db(admin: &PgPool, pool: PgPool, name: &str) {
52685299
pool.close().await;
52695300
let _ = sqlx::query(sqlx::AssertSqlSafe(format!(
@@ -5305,10 +5336,13 @@ mod postgres_tests {
53055336

53065337
/// Migration 0074 retires the 0032 roster trigger: after migrating, the
53075338
/// database accepts a raw kind:39002 insert that disagrees with
5308-
/// membership, because only the app-side check guards rosters now.
5339+
/// membership, because only the app-side check guards rosters now. The
5340+
/// startup check refuses a database where the trigger is still
5341+
/// installed and accepts it after 0074.
53095342
///
5310-
/// Mutation: remove the DROP statements from 0074 → the trigger refuses
5311-
/// the insert → RED.
5343+
/// Mutations: remove the DROP statements from 0074 → the trigger refuses
5344+
/// the insert → RED; make the startup check always pass → the pre-0074
5345+
/// database is accepted → RED.
53125346
#[tokio::test]
53135347
#[ignore = "requires Postgres"]
53145348
async fn migrations_retire_the_roster_trigger() {
@@ -5317,7 +5351,23 @@ mod postgres_tests {
53175351
let admin = PgPool::connect(&admin_url().await)
53185352
.await
53195353
.expect("connect admin");
5320-
let (pool, scratch_name) = create_scratch_db(&admin, "roster_fence_retired").await;
5354+
let (pool, scratch_name) =
5355+
create_scratch_db_through(&admin, "roster_fence_retired", Some(73)).await;
5356+
let db = Db::from_pool(pool.clone());
5357+
let error = db
5358+
.verify_channel_roster_trigger_retired()
5359+
.await
5360+
.expect_err("before 0074 the trigger is installed");
5361+
assert!(
5362+
error.to_string().contains("apply migration 0074"),
5363+
"the startup check must name the missing migration: {error}"
5364+
);
5365+
migration::run_migrations(&pool)
5366+
.await
5367+
.expect("apply migration 0074");
5368+
db.verify_channel_roster_trigger_retired()
5369+
.await
5370+
.expect("0074 retires the trigger");
53215371
let function: Option<String> = sqlx::query_scalar(
53225372
"SELECT to_regprocedure('public.guard_channel_roster_snapshot()')::text",
53235373
)

‎crates/buzz-relay/src/handlers/auth.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2401,10 +2401,13 @@ mod tests {
24012401
/// a contended roster lock must not hold AUTH: the pending AUTH
24022402
/// expires after 5 s, the same as the default lock timeout. The
24032403
/// republish also waits for a relay-wide permit, so a burst of first
2404-
/// sign-ins holds only a few writer connections.
2404+
/// sign-ins holds only a few writer connections. While every permit
2405+
/// is held, the republish waits on the permit before it reaches the
2406+
/// roster lock, so AUTH is checked against the whole republish rather
2407+
/// than the lock alone.
24052408
///
24062409
/// Mutations: await the roster republish inside
2407-
/// `materialize_nip_oa_owner` → AUTH waits on the held lock → RED;
2410+
/// `materialize_nip_oa_owner` → AUTH waits on the held permits → RED;
24082411
/// skip the republish permit → the roster changes while every permit
24092412
/// is held → RED.
24102413
#[tokio::test]

‎crates/buzz-relay/src/main.rs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1114,6 +1114,18 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
11141114
);
11151115
}
11161116

1117+
// This relay shows computed `member`/`bot` roster labels, which the 0032
1118+
// trigger refuses. Fail closed until migration 0074 has dropped it, so a
1119+
// skipped migration stops the rollout instead of freezing rosters.
1120+
let step = StepTimer::start(StartupStep::ChannelRosterTriggerRetired);
1121+
if let Err(error) = state.db.verify_channel_roster_trigger_retired().await {
1122+
error!(%error, "Channel roster trigger check failed");
1123+
return Err(anyhow::anyhow!(
1124+
"Channel roster trigger check failed; apply migration 0074 before starting this relay: {error}"
1125+
));
1126+
}
1127+
step.finish();
1128+
11171129
// Repair legacy NIP-29 channel rosters that were persisted while the
11181130
// canonical member query still truncated at 1,000 rows.
11191131
let mut step = StepTimer::start(StartupStep::LargeRosterReconcile);

‎crates/buzz-relay/src/startup_steps.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ pub enum StartupStep {
7575
MeshInitialReconcile,
7676
/// Run the git object-store conformance probe (when enabled).
7777
GitConformanceProbe,
78+
/// Verify that migration 0074 retired the 0032 roster trigger.
79+
ChannelRosterTriggerRetired,
7880
/// Repair large NIP-29 channel roster snapshots.
7981
LargeRosterReconcile,
8082
/// Bind the private health listener.
@@ -102,6 +104,7 @@ impl StartupStep {
102104
Self::MeshPublish,
103105
Self::MeshInitialReconcile,
104106
Self::GitConformanceProbe,
107+
Self::ChannelRosterTriggerRetired,
105108
Self::LargeRosterReconcile,
106109
Self::HealthBind,
107110
Self::ListenerBind,
@@ -126,6 +129,7 @@ impl StartupStep {
126129
Self::MeshPublish => "mesh_publish",
127130
Self::MeshInitialReconcile => "mesh_initial_reconcile",
128131
Self::GitConformanceProbe => "git_conformance_probe",
132+
Self::ChannelRosterTriggerRetired => "channel_roster_trigger_retired",
129133
Self::LargeRosterReconcile => "large_roster_reconcile",
130134
Self::HealthBind => "health_bind",
131135
Self::ListenerBind => "listener_bind",

‎deploy/charts/buzz/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -616,7 +616,7 @@ default so long-lived WebSocket connections have time to drain.
616616

617617
Schema migrations are embedded in the relay binary via `sqlx::migrate!` and run at startup, gated by `BUZZ_AUTO_MIGRATE` (default `true`). Multiple replicas race-safely behind a Postgres advisory lock. `helm upgrade` is the entire upgrade procedure.
618618

619-
Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. Older relays verify its roster-fence trigger before opening listeners and refuse to start if the trigger is missing or inert. Newer relays run the same check in every roster writer and do not require the trigger. Migration 0074 drops the trigger, so every relay must run a version without the startup check (#8611 or later) before 0074 is applied; an older relay then refuses to start. Relays from the 0074 release on show a computed `member`/`bot` label in rosters that the trigger would refuse, so apply 0074 before rolling them out. After that rollout, start one relay once with `BUZZ_RECONCILE_BOT_ROSTERS=1` (`true`, `yes` and `on` also work; any other value leaves it off) to republish rosters that older relays published with the stored label. Confirm that it logs `member/bot channel rosters reconciled`; a warning instead means the repair did not run, so start it again. Apply migrations before rolling the relay; for large installations, prefer a controlled `buzz-admin migrate` job with PostgreSQL lock monitoring before the code rollout.
619+
Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. Older relays verify its roster-fence trigger before opening listeners and refuse to start if the trigger is missing or inert. Newer relays run the same check in every roster writer and do not require the trigger. Migration 0074 drops the trigger, so every relay must run a version without the startup check (#8611 or later) before 0074 is applied; an older relay then refuses to start. Relays from the 0074 release on show a computed `member`/`bot` label in rosters that the trigger would refuse, so they refuse to start while the trigger is installed. Apply 0074 before rolling them out. After that rollout, start one relay once with `BUZZ_RECONCILE_BOT_ROSTERS=1` (`true`, `yes` and `on` also work; any other value leaves it off) to republish rosters that older relays published with the stored label. Confirm that it logs `member/bot channel rosters reconciled`; a warning instead means the repair did not run, so start it again. Apply migrations before rolling the relay; for large installations, prefer a controlled `buzz-admin migrate` job with PostgreSQL lock monitoring before the code rollout.
620620

621621
If you prefer decoupling migrations from serving, set `migrate.autoMigrate=false`. **In that mode the chart does not run migrations for you** — you own running `buzz-admin migrate` (separate Pod / one-shot Job) against the database before every `helm install` / `helm upgrade`. Readiness probes only verify DB connectivity, not schema freshness, so a pod will appear healthy against an unmigrated schema and fail under load. A pre-upgrade Helm Job for this is on the chart roadmap; the values knob `migrate.preUpgradeJob.enabled` is reserved.
622622

‎migrations/0076_retire_channel_roster_trigger.sql‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,9 @@
77
-- later before this migration runs; older relays refuse to start without the
88
-- trigger. Published migrations remain checksum-frozen.
99
--
10-
-- DROP TRIGGER takes ACCESS EXCLUSIVE on the partitioned parent and on each
11-
-- partition in turn, and keeps each lock while it waits for the next. While
12-
-- the locks are queued or held, event reads and writes both block.
10+
-- DROP TRIGGER takes ACCESS EXCLUSIVE on each partition of events in turn,
11+
-- and keeps each lock while it waits for the next. While the locks are queued
12+
-- or held, event reads and writes both block.
1313
-- lock_timeout bounds each lock wait and statement_timeout bounds each whole
1414
-- statement, as in 0049. Either timeout rolls the migration back with no
1515
-- change, and it can be retried.

0 commit comments

Comments
 (0)