Skip to content

Commit ce41be9

Browse files
wpfleger96Hayt
andauthored
fix(relay): fire kick live side effects at convergence; persist target; fence re-add race with held lock (#7298)
## What Ensures admin kick live side effects (subscription eviction + workflow disable) fire correctly on every path — fresh HTTP, crash recovery, and upgrade recovery from pre-migration rows — without a race window where a concurrent legitimate re-add can be revoked. ### Finding 1 — Lock held through effects Replace `verify_member_still_removed` (which acquired + released the advisory lock before returning) with `membership_removal_fence`, a RAII guard that keeps the `pg_advisory_xact_lock` alive until the caller drops it. Update `apply_kick_live_side_effects` to use the fence on every path (fresh and recovery), holding it through eviction and workflow-disable. Remove the `is_recovery` parameter entirely. **The bug:** the old code released the lock before returning a bool; a concurrent `add_member` could commit between the check and the effects, silently evicting a re-added member. **New guarantee:** any `add_member` either serializes before the fence (fence observes `removed_at IS NULL`, skips effects) or waits until after effects complete. ### Finding 2 — Persisted-context path scoped to kick only Gate the persisted-context branch in `admin_action_worker.rs` on `action == "kick"`. Non-kick actions (ban, timeout, delete) always re-derive from the report. **The bug:** any action with `enforcement_target_pubkey` set took the persisted-context branch, which hard-coded `target_event_id = None`. A stranded delete either failed pre-marker recovery ("delete requires target_event_id") or finalized post-marker without the tombstone outbox row. ### Finding 3 — Legacy kick row upgrade recovery (Carl's CHANGES_REQUESTED) Migration 0047 (renumbered from 0045 to avoid collision with two main merges: 0045_retain_push_revocation_tombstones and 0046_storage_accounting_snapshots) adds `enforcement_target_pubkey`/`enforcement_channel_id` without backfilling. **The bug:** the convergence gate read only the persisted columns and rejected NULL/NULL rows as corrupt, causing every recovery retry to fail forever. A cancellation cannot release a post-mutation action, so these rows remain permanently stuck. **Fix:** when persisted columns are absent, fall back to the function parameters (which the recovery worker re-derives from the report row). Error only when both sources are absent (genuinely unresolvable). This preserves new-writer safety while allowing pre-migration rows to converge. An old writer in a rolling deployment can also create NULL rows after the DDL applies; the same fallback handles these. ### Finding 4 — Test moved to PG lane `kick_live_side_effects_clears_membership_cache_and_evicts_subscription` was in the infra-free unit lane but called `membership_removal_fence`, which requires a real Postgres connection. Moved to the PG fixture lane (`#[ignore = "requires Postgres"]`) with a real member row so the fence query sees the expected `removed_at` state. ## Tests - **`crash_recovery_redrive_fires_kick_live_side_effects`** (existing): re-drive via real recovery worker path, asserts all three side effects. - **`crash_recovery_after_readd_preserves_membership_subscriptions_and_workflows`** (updated): uses real `add_member` (not direct SQL) — re-add completes before fence is acquired. - **`membership_removal_fence_blocks_concurrent_add_member`** (existing): holds the fence explicitly, spawns `add_member`, asserts it blocks until the fence is dropped. - **`stranded_delete_pre_marker_recovers_via_worker`** (existing): stranded delete (crash before mutation) converges via `recover_one` to succeeded. - **`legacy_kick_row_pre_marker_recovers_via_worker`** (new): old-writer shaped row (NULL columns, no step_marker) converges through the full state machine via recovery worker. Falsifiable: reverting the convergence-gate fallback leaves the action stuck in enforcing. - **`legacy_kick_row_post_marker_recovers_via_worker`** (new): same but with step_marker set (post-marker crash). Falsifiable: same revert. - **`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`** (moved to PG lane): verifies cache eviction and subscription removal against a real member row. ## CI `Rust / Rust Lint` and `Windows Rust` failures in the prior run were main-inherited (same lanes fail on main at the prior base). The branch is now at main's current head. --------- Signed-off-by: Will Pfleger <[email protected]> Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
1 parent 6a86531 commit ce41be9

10 files changed

Lines changed: 2613 additions & 45 deletions

File tree

‎crates/buzz-db/src/runtime/migration.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -703,7 +703,7 @@ mod postgres_tests {
703703
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
704704
migrations.sort_by_key(|migration| migration.version);
705705

706-
assert_eq!(migrations.len(), 46);
706+
assert_eq!(migrations.len(), 47);
707707
assert_eq!(migrations[0].version, 1);
708708
assert_eq!(&*migrations[0].description, "initial schema");
709709
assert!(migrations[0]
@@ -2259,9 +2259,9 @@ mod postgres_tests {
22592259
.await
22602260
.expect("connect migrated probe database");
22612261
MIGRATOR
2262-
.run_to(39, &migrated)
2262+
.run_to(47, &migrated)
22632263
.await
2264-
.expect("apply migrations 1-39");
2264+
.expect("apply migrations 1-47");
22652265

22662266
for table in [
22672267
"relay_admin_actions",

‎crates/buzz-db/src/store/channel_members.rs‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -687,6 +687,173 @@ pub async fn is_member(
687687
Ok(cnt > 0)
688688
}
689689

690+
/// Verify that a member is still removed under the channel-membership lock,
691+
/// serializing with any concurrent [`add_member`] call on the same channel.
692+
///
693+
/// Returns `true` when `removed_at IS NOT NULL` for `(community_id, channel_id,
694+
/// pubkey)` — i.e., the removal that crash-recovery is re-applying is still the
695+
/// current state and no re-add has reversed it.
696+
///
697+
/// Uses the same `pg_advisory_xact_lock` key as [`add_member`] so that a
698+
/// concurrent re-add either sees this check complete (and then sets `removed_at
699+
/// = NULL` after recovery finishes) or serializes before it (in which case this
700+
/// check observes the re-add and returns `false`, preventing stale eviction).
701+
///
702+
/// Designed for use before firing subscription eviction and workflow
703+
/// disablement: cache invalidation is always safe (it only drops a stale
704+
/// positive), but eviction and workflow-disable must not target a member who
705+
/// was legitimately re-added after the kick committed.
706+
///
707+
/// **Use [`membership_removal_fence`] instead of this function** when the caller
708+
/// needs to hold the advisory lock through the destructive effects themselves.
709+
/// This function releases the lock immediately after the read; concurrent
710+
/// `add_member` calls can commit between the return and the caller's effects.
711+
pub async fn verify_member_still_removed(
712+
pool: &PgPool,
713+
community_id: CommunityId,
714+
channel_id: Uuid,
715+
pubkey: &[u8],
716+
) -> Result<bool> {
717+
let connection = crate::observability::acquire_writer(
718+
pool,
719+
crate::observability::WriterOperation::Authorization,
720+
)
721+
.await?;
722+
let mut tx = sqlx::Transaction::begin(connection, None).await?;
723+
acquire_channel_membership_lock(&mut tx, community_id, channel_id).await?;
724+
725+
let row = sqlx::query(
726+
"SELECT removed_at FROM channel_members \
727+
WHERE community_id = $1 AND channel_id = $2 AND pubkey = $3",
728+
)
729+
.bind(community_id.as_uuid())
730+
.bind(channel_id)
731+
.bind(pubkey)
732+
.fetch_optional(&mut *tx)
733+
.await?;
734+
735+
// Row absent → never joined or hard-deleted; treat as removed (safe to skip
736+
// eviction for someone who isn't and wasn't a member).
737+
let still_removed = match row {
738+
None => true,
739+
Some(r) => {
740+
let removed_at: Option<chrono::DateTime<Utc>> = r.try_get("removed_at")?;
741+
removed_at.is_some()
742+
}
743+
};
744+
tx.rollback().await?;
745+
Ok(still_removed)
746+
}
747+
748+
/// A guard that holds the per-channel membership advisory lock for the
749+
/// duration of kick live side effects.
750+
///
751+
/// Acquired via [`membership_removal_fence`]. The lock is released when the
752+
/// guard is dropped (the inner transaction rolls back). Callers must NOT drop
753+
/// the guard until after subscription eviction and workflow-disable complete,
754+
/// so that no concurrent [`add_member`] can commit between the `still_removed`
755+
/// observation and the destructive effects.
756+
///
757+
/// After firing all live side effects, call
758+
/// [`commit_disabling_workflows`][Self::commit_disabling_workflows] to run
759+
/// the durable workflow-disable UPDATE on this guard's own connection and
760+
/// commit the transaction (releasing the lock). If the guard is dropped
761+
/// without committing, the inner transaction rolls back — the advisory lock
762+
/// is released, but no domain writes persist.
763+
pub struct MembershipRemovalFence {
764+
/// `true` when the member row still has `removed_at IS NOT NULL` — i.e., no
765+
/// re-add has reversed the kick since it committed. When `false`, the caller
766+
/// must NOT fire eviction or workflow-disable.
767+
pub still_removed: bool,
768+
// Holds the advisory transaction lock. The caller commits via
769+
// `commit_disabling_workflows`; on drop without commit the tx rolls back.
770+
tx: Transaction<'static, Postgres>,
771+
}
772+
773+
impl MembershipRemovalFence {
774+
/// Run the workflow-disable UPDATE on this guard's own connection, then
775+
/// commit the transaction (releasing the advisory lock).
776+
///
777+
/// By running the UPDATE inside the same connection that holds the lock,
778+
/// no additional pool connection is needed — preventing the self-deadlock
779+
/// that would arise from a pool-size-N scenario where every kick holds one
780+
/// connection while trying to acquire a second for the disable write.
781+
///
782+
/// The commit makes the disable durable before the lock is released, so
783+
/// there is no window between "workflows disabled" and "lock released."
784+
///
785+
/// On failure the transaction is rolled back (the advisory lock is still
786+
/// released), and the error is returned to the caller to handle (log/skip).
787+
pub async fn commit_disabling_workflows(
788+
mut self,
789+
community_id: CommunityId,
790+
channel_id: Uuid,
791+
owner_pubkey: &[u8],
792+
) -> crate::Result<u64> {
793+
let affected = crate::workflow::disable_workflows_for_owner_in_channel_on_conn(
794+
&mut self.tx,
795+
community_id,
796+
channel_id,
797+
owner_pubkey,
798+
)
799+
.await?;
800+
self.tx.commit().await?;
801+
Ok(affected)
802+
}
803+
}
804+
805+
/// Acquire the per-channel membership advisory lock and check whether the
806+
/// kicked member is still removed, returning a [`MembershipRemovalFence`] that
807+
/// keeps the lock alive until the guard is dropped.
808+
///
809+
/// By holding the lock from the moment of the `removed_at` check until after
810+
/// the caller's subscription eviction and workflow-disable complete, this
811+
/// prevents the window where a concurrent [`add_member`] could commit between
812+
/// the check and the effects (the race that [`verify_member_still_removed`]
813+
/// leaves open, since it releases the lock before returning).
814+
///
815+
/// The guard is read-only before [`Self::commit_disabling_workflows`] is called.
816+
/// Once the caller has finished its effects, dropping the guard releases the
817+
/// lock (the transaction is implicitly rolled back if not committed). To
818+
/// durably write the workflow-disable and release the lock in one step, call
819+
/// [`Self::commit_disabling_workflows`].
820+
pub async fn membership_removal_fence(
821+
pool: &PgPool,
822+
community_id: CommunityId,
823+
channel_id: Uuid,
824+
pubkey: &[u8],
825+
) -> Result<MembershipRemovalFence> {
826+
let connection = crate::observability::acquire_writer(
827+
pool,
828+
crate::observability::WriterOperation::Authorization,
829+
)
830+
.await?;
831+
let mut tx = sqlx::Transaction::begin(connection, None).await?;
832+
acquire_channel_membership_lock(&mut tx, community_id, channel_id).await?;
833+
834+
let row = sqlx::query(
835+
"SELECT removed_at FROM channel_members \
836+
WHERE community_id = $1 AND channel_id = $2 AND pubkey = $3",
837+
)
838+
.bind(community_id.as_uuid())
839+
.bind(channel_id)
840+
.bind(pubkey)
841+
.fetch_optional(&mut *tx)
842+
.await?;
843+
844+
// Row absent → never joined or hard-deleted; treat as removed (safe to
845+
// skip eviction for someone who isn't and wasn't a member).
846+
let still_removed = match row {
847+
None => true,
848+
Some(r) => {
849+
let removed_at: Option<chrono::DateTime<Utc>> = r.try_get("removed_at")?;
850+
removed_at.is_some()
851+
}
852+
};
853+
854+
Ok(MembershipRemovalFence { still_removed, tx })
855+
}
856+
690857
/// Return which of the given (channel, pubkey) combinations are active
691858
/// memberships, restricted to non-deleted channels — one statement for any
692859
/// batch size (T2b). Semantics per pair match [`is_member`].
@@ -1364,6 +1531,45 @@ impl Db {
13641531
is_member(&self.pool, community_id, channel_id, pubkey).await
13651532
}
13661533

1534+
/// Returns `true` when the member row still has `removed_at IS NOT NULL`,
1535+
/// holding the channel-membership advisory lock so the check serializes
1536+
/// with concurrent [`add_member`] calls.
1537+
///
1538+
/// **Note:** this function releases the lock before returning. Use
1539+
/// [`Db::membership_removal_fence`] when the advisory lock must remain
1540+
/// held through subscription eviction and workflow-disable.
1541+
#[datastore_span(name = "verify_member_still_removed", system = "postgresql")]
1542+
pub async fn verify_member_still_removed(
1543+
&self,
1544+
community_id: CommunityId,
1545+
channel_id: Uuid,
1546+
pubkey: &[u8],
1547+
) -> Result<bool> {
1548+
verify_member_still_removed(&self.pool, community_id, channel_id, pubkey).await
1549+
}
1550+
1551+
/// Acquire the per-channel membership advisory lock and return a
1552+
/// [`MembershipRemovalFence`] that keeps the lock alive until it is
1553+
/// dropped or committed via [`MembershipRemovalFence::commit_disabling_workflows`].
1554+
///
1555+
/// The guard's `still_removed` field indicates whether the kick is still
1556+
/// the current state (i.e., no re-add has reversed it). When `true`, the
1557+
/// caller fires eviction and then calls
1558+
/// [`MembershipRemovalFence::commit_disabling_workflows`] to durably
1559+
/// write the workflow-disable and commit the transaction (releasing the
1560+
/// lock). When `false`, the member was re-added and effects must be skipped.
1561+
/// Dropping the guard without committing releases the advisory lock (the
1562+
/// transaction is implicitly rolled back).
1563+
#[datastore_span(name = "membership_removal_fence", system = "postgresql")]
1564+
pub async fn membership_removal_fence(
1565+
&self,
1566+
community_id: CommunityId,
1567+
channel_id: Uuid,
1568+
pubkey: &[u8],
1569+
) -> Result<MembershipRemovalFence> {
1570+
membership_removal_fence(&self.pool, community_id, channel_id, pubkey).await
1571+
}
1572+
13671573
/// Return the active (channel, pubkey) membership pairs among the given
13681574
/// sets, in one statement.
13691575
#[datastore_span(name = "membership_pairs", system = "postgresql")]

‎crates/buzz-db/src/store/relay_admin_actions.rs‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ pub struct AdminActionRecord {
4747
pub cancelled_by: Option<Vec<u8>>,
4848
/// Error from the last failure, if any.
4949
pub error_message: Option<String>,
50+
/// Authoritative target pubkey persisted at claim time (kick/ban/timeout pubkey targets).
51+
/// `None` for event/blob targets or actions with no pubkey target.
52+
pub enforcement_target_pubkey: Option<Vec<u8>>,
53+
/// Authoritative channel persisted at claim time (kick actions).
54+
/// `None` for community-wide actions.
55+
pub enforcement_channel_id: Option<Uuid>,
5056
/// Row creation time.
5157
pub created_at: DateTime<Utc>,
5258
/// Row last-updated time.
@@ -241,6 +247,7 @@ pub async fn claim_report(
241247
r#"
242248
SELECT id, report_id, report_community_id, request_id, actor_pubkey, actor_role,
243249
action, reason, timeout_until, state, step_marker, cancelled_by, error_message,
250+
enforcement_target_pubkey, enforcement_channel_id,
244251
created_at, updated_at
245252
FROM relay_admin_actions
246253
WHERE report_community_id = $1 AND report_id = $2 AND request_id = $3
@@ -269,10 +276,11 @@ pub async fn claim_report(
269276
r#"
270277
INSERT INTO relay_admin_actions (
271278
report_id, report_community_id, request_id, actor_pubkey, actor_role,
272-
action, reason, timeout_until, state
273-
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'pending')
279+
action, reason, timeout_until, state, enforcement_target_pubkey, enforcement_channel_id
280+
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'pending', $9, $10)
274281
RETURNING id, report_id, report_community_id, request_id, actor_pubkey, actor_role,
275282
action, reason, timeout_until, state, step_marker, cancelled_by, error_message,
283+
enforcement_target_pubkey, enforcement_channel_id,
276284
created_at, updated_at
277285
"#,
278286
)
@@ -284,6 +292,8 @@ pub async fn claim_report(
284292
.bind(action)
285293
.bind(reason)
286294
.bind(timeout_until)
295+
.bind(target_pubkey)
296+
.bind(channel_id)
287297
.fetch_one(&mut *tx)
288298
.await?;
289299

@@ -1242,6 +1252,7 @@ pub async fn get_action(pool: &PgPool, action_id: Uuid) -> Result<Option<AdminAc
12421252
r#"
12431253
SELECT id, report_id, report_community_id, request_id, actor_pubkey, actor_role,
12441254
action, reason, timeout_until, state, step_marker, cancelled_by, error_message,
1255+
enforcement_target_pubkey, enforcement_channel_id,
12451256
created_at, updated_at
12461257
FROM relay_admin_actions WHERE id = $1
12471258
"#,
@@ -1263,6 +1274,7 @@ pub async fn get_action_by_request(
12631274
r#"
12641275
SELECT id, report_id, report_community_id, request_id, actor_pubkey, actor_role,
12651276
action, reason, timeout_until, state, step_marker, cancelled_by, error_message,
1277+
enforcement_target_pubkey, enforcement_channel_id,
12661278
created_at, updated_at
12671279
FROM relay_admin_actions
12681280
WHERE report_community_id = $1 AND report_id = $2 AND request_id = $3
@@ -1520,7 +1532,8 @@ pub async fn claim_stranded_action_batch(
15201532
AND (action_lease_expires_at IS NULL OR action_lease_expires_at < now())
15211533
RETURNING id, report_id, report_community_id, request_id, actor_pubkey,
15221534
actor_role, action, reason, timeout_until, state, step_marker,
1523-
cancelled_by, error_message, created_at, updated_at
1535+
cancelled_by, error_message, enforcement_target_pubkey,
1536+
enforcement_channel_id, created_at, updated_at
15241537
"#,
15251538
)
15261539
.bind(id)
@@ -1618,6 +1631,8 @@ fn row_to_action(row: sqlx::postgres::PgRow) -> Result<AdminActionRecord> {
16181631
step_marker: row.try_get("step_marker")?,
16191632
cancelled_by: row.try_get("cancelled_by")?,
16201633
error_message: row.try_get("error_message")?,
1634+
enforcement_target_pubkey: row.try_get("enforcement_target_pubkey")?,
1635+
enforcement_channel_id: row.try_get("enforcement_channel_id")?,
16211636
created_at: row.try_get("created_at")?,
16221637
updated_at: row.try_get("updated_at")?,
16231638
})

‎crates/buzz-db/src/store/workflow.rs‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ use std::str::FromStr;
1515

1616
use chrono::{DateTime, Utc};
1717
use sha2::{Digest, Sha256};
18-
use sqlx::{PgPool, Row};
18+
use sqlx::{PgConnection, PgPool, Row};
1919
use uuid::Uuid;
2020

2121
use buzz_core::CommunityId;
@@ -746,6 +746,35 @@ pub async fn disable_workflows_for_owner_in_channel(
746746
Ok(affected)
747747
}
748748

749+
/// Connection-local variant: run the same workflow-disable UPDATE on the
750+
/// provided connection rather than acquiring a new one from the pool.
751+
///
752+
/// Used by [`buzz_db::channel_members::MembershipRemovalFence::commit_disabling_workflows`]
753+
/// so the disable executes on the fence's own connection, avoiding a second
754+
/// pool acquisition while the fence holds one connection.
755+
pub async fn disable_workflows_for_owner_in_channel_on_conn(
756+
conn: &mut PgConnection,
757+
community_id: CommunityId,
758+
channel_id: Uuid,
759+
owner_pubkey: &[u8],
760+
) -> Result<u64> {
761+
let affected = sqlx::query(
762+
r#"
763+
UPDATE workflows
764+
SET enabled = FALSE
765+
WHERE community_id = $1 AND channel_id = $2 AND owner_pubkey = $3 AND enabled = TRUE
766+
"#,
767+
)
768+
.bind(community_id.as_uuid())
769+
.bind(channel_id)
770+
.bind(owner_pubkey)
771+
.execute(&mut *conn)
772+
.await?
773+
.rows_affected();
774+
775+
Ok(affected)
776+
}
777+
749778
/// Delete a workflow and all its runs/approvals (CASCADE).
750779
///
751780
/// NOTE: see the cache-invalidation note on [`update_workflow`]. The relay's

0 commit comments

Comments
 (0)