Skip to content

Commit cf018e1

Browse files
npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67dTyler Longwell
andcommitted
relay: gate push enqueue on live leases; batch matcher pipeline
Two write-amplification fixes for the push pipeline (T1a repair, T1b, T2b of the write-amp plan): Migration 0023 (T1b push gate): skip the push_match_queue enqueue for communities with no active, endpoint-enabled, unexpired push lease. In lease-less communities (most of them) every durable message paid the full matcher cost — enqueue + claim + lease scan + delete — to conclude "notify no one". The gate lives in the events trigger so every durable producer is covered. The lost-wake race (event reads "no lease" while a lease activation commits concurrently) is closed with a per-community advisory lock: event inserts take it SHARED, lease transitions that can make eligibility true (accept_lease_event, replace_lease) take it EXCLUSIVE, plus a product-recovery backfill on activation. Migration 0024 (T1a repair): the 0022 TTL-refresh trigger took FOR UPDATE on the channel row before testing ttl_seconds, so every durable message in a permanent channel serialized at commit time on that tuple, each holding it across its WAL flush — one hot channel meant fully serialized commits (observed live: 0.07ms -> ~15ms commit latency at 200 QPS). The trigger now synchronizes on a shared per-channel advisory lock; TTL transitions in update_channel take the same key EXCLUSIVE, so the stale-prefetch proof from 0022 is preserved without serializing the permanent-channel hot path. T2b (matcher batching): the relay push matcher now claims per-community batches (single SKIP LOCKED CTE, cap 64) and loads events, leases, and channel membership pairs in one statement each; completes and retries are set-wise under the shared claim fence. One batch costs a constant number of statements regardless of size. The poison-job reap moves off the claim path to a 30s sweep (its scan is not served by the due partial index), and idle matcher/delivery workers back off 250ms -> 2s. The per-job claim API is deleted, not deprecated — no second code path. Tests: batch fence contract, single-community batch claim, T1a barrier (8 concurrent permanent-channel commits, zero channel tuple locks), T1b forced lost-wake ordering + activation backfill, exhausted-job sweep. Full buzz-db + buzz-relay suites green on a fresh 0001-0024 database (lone red: api::mesh_demo echo test, also red on main). Co-authored-by: Tyler Longwell <[email protected]> Signed-off-by: Tyler Longwell <[email protected]>
1 parent c5db094 commit cf018e1

10 files changed

Lines changed: 1092 additions & 133 deletions

File tree

‎crates/buzz-db/src/channel.rs‎

Lines changed: 53 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,33 @@ pub async fn is_member(
538538
Ok(cnt > 0)
539539
}
540540

541+
/// Return which of the given (channel, pubkey) combinations are active
542+
/// memberships, restricted to non-deleted channels — one statement for any
543+
/// batch size (T2b). Semantics per pair match [`is_member`].
544+
pub async fn membership_pairs(
545+
pool: &PgPool,
546+
community_id: CommunityId,
547+
channel_ids: &[Uuid],
548+
pubkeys: &[Vec<u8>],
549+
) -> Result<Vec<(Uuid, Vec<u8>)>> {
550+
if channel_ids.is_empty() || pubkeys.is_empty() {
551+
return Ok(Vec::new());
552+
}
553+
let rows = sqlx::query(
554+
"SELECT cm.channel_id, cm.pubkey FROM channel_members cm \
555+
JOIN channels c ON cm.community_id = c.community_id AND cm.channel_id = c.id AND c.deleted_at IS NULL \
556+
WHERE cm.community_id = $1 AND cm.channel_id = ANY($2) AND cm.pubkey = ANY($3) AND cm.removed_at IS NULL",
557+
)
558+
.bind(community_id.as_uuid())
559+
.bind(channel_ids)
560+
.bind(pubkeys)
561+
.fetch_all(pool)
562+
.await?;
563+
rows.into_iter()
564+
.map(|row| Ok((row.try_get("channel_id")?, row.try_get("pubkey")?)))
565+
.collect()
566+
}
567+
541568
/// Returns all active members of the given channel.
542569
///
543570
/// Returns an empty list if the channel has been soft-deleted.
@@ -1076,9 +1103,32 @@ pub async fn update_channel(
10761103
q = q.bind(community_id.as_uuid());
10771104
q = q.bind(channel_id);
10781105

1079-
let result = q.execute(pool).await?;
1080-
if result.rows_affected() == 0 {
1081-
return Err(DbError::ChannelNotFound(channel_id));
1106+
// T1a repair: a TTL change can flip this channel's event-trigger fast
1107+
// path (migration 0024 reads ttl_seconds under a SHARED per-channel
1108+
// advisory lock). Take the same key EXCLUSIVE before the UPDATE so a
1109+
// concurrent event either sees the committed TTL or strictly precedes
1110+
// this transition — whose own deadline reset is then the latest word.
1111+
// Non-TTL updates don't touch the fast path and skip the lock.
1112+
if updates.ttl_seconds.is_some() {
1113+
let mut tx = pool.begin().await?;
1114+
sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))")
1115+
.bind(format!(
1116+
"buzz_channel_ttl:{}:{}",
1117+
community_id.as_uuid(),
1118+
channel_id
1119+
))
1120+
.execute(&mut *tx)
1121+
.await?;
1122+
let result = q.execute(&mut *tx).await?;
1123+
if result.rows_affected() == 0 {
1124+
return Err(DbError::ChannelNotFound(channel_id));
1125+
}
1126+
tx.commit().await?;
1127+
} else {
1128+
let result = q.execute(pool).await?;
1129+
if result.rows_affected() == 0 {
1130+
return Err(DbError::ChannelNotFound(channel_id));
1131+
}
10821132
}
10831133

10841134
get_channel(pool, community_id, channel_id).await

‎crates/buzz-db/src/event.rs‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1561,6 +1561,14 @@ mod tests {
15611561
assert_eq!(stale_ttl, None);
15621562

15631563
let mut activation = pool.begin().await.expect("begin TTL activation");
1564+
// Model the repaired update_channel protocol (migration 0024): the
1565+
// TTL transition holds the per-channel advisory key EXCLUSIVE, which
1566+
// is what the event trigger's shared acquisition now waits on.
1567+
sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))")
1568+
.bind(format!("buzz_channel_ttl:{community_uuid}:{racing}"))
1569+
.execute(&mut *activation)
1570+
.await
1571+
.expect("acquire exclusive channel TTL key");
15641572
let activation_deadline: DateTime<Utc> = sqlx::query_scalar(
15651573
"UPDATE channels \
15661574
SET ttl_seconds = 60, ttl_deadline = clock_timestamp() + interval '60 seconds' \
@@ -1605,6 +1613,99 @@ mod tests {
16051613
);
16061614
}
16071615

1616+
/// T1a repair regression test (migration 0024): permanent-channel event
1617+
/// commits must not serialize on the channel row. The 0022 trigger took
1618+
/// `FOR UPDATE` on the channel tuple before testing `ttl_seconds`, so
1619+
/// concurrent commits into one hot permanent channel queued at commit
1620+
/// time (deferred trigger) — invisible to any single-connection test.
1621+
/// This holds N insert transactions at a barrier past their INSERTs,
1622+
/// then proves (a) while all N sit pre-commit, no transaction holds a
1623+
/// row-level lock on the channel tuple, and (b) all N commits succeed
1624+
/// with the channel row untouched (permanent ⇒ no deadline write).
1625+
#[tokio::test]
1626+
#[ignore = "requires Postgres"]
1627+
async fn permanent_channel_event_commits_do_not_lock_the_channel_row() {
1628+
const N: usize = 8;
1629+
// setup_pool's default cap (10) covers N held transactions plus the
1630+
// pg_locks inspector connection.
1631+
let pool = setup_pool().await;
1632+
let community_uuid = make_test_community(&pool).await;
1633+
let channel = make_test_channel(&pool, community_uuid, None).await;
1634+
1635+
// Open N transactions, run the full event INSERT in each (the deferred
1636+
// trigger fires at COMMIT), and park them at a barrier.
1637+
let mut txs = Vec::new();
1638+
for i in 0..N {
1639+
let mut tx = pool.begin().await.expect("begin insert txn");
1640+
let event = make_text_event(&format!("hot channel event {i}"));
1641+
sqlx::query(
1642+
"INSERT INTO events (community_id,id,pubkey,created_at,kind,tags,content,sig,received_at,channel_id) \
1643+
VALUES ($1,$2,$3,$4,9,$5,$6,$7,now(),$8)",
1644+
)
1645+
.bind(community_uuid)
1646+
.bind(event.id.as_bytes().as_slice())
1647+
.bind(event.pubkey.as_bytes().as_slice())
1648+
.bind(DateTime::from_timestamp(event.created_at.as_secs() as i64, 0).unwrap())
1649+
.bind(serde_json::to_value(&event.tags).unwrap())
1650+
.bind(&event.content)
1651+
.bind(event.sig.serialize().as_slice())
1652+
.bind(channel)
1653+
.execute(&mut *tx)
1654+
.await
1655+
.expect("insert event inside held txn");
1656+
txs.push(tx);
1657+
}
1658+
1659+
// With all N transactions holding completed INSERTs, none may hold a
1660+
// row-level lock on the channels tuple. (The 0022 trigger would not
1661+
// have taken it yet either — it locks at COMMIT — so also verify the
1662+
// commit phase below completes without mutual blocking.)
1663+
let tuple_locks: i64 = sqlx::query_scalar(
1664+
"SELECT count(*) FROM pg_locks l \
1665+
JOIN pg_class c ON c.oid = l.relation \
1666+
WHERE c.relname = 'channels' AND l.locktype = 'tuple'",
1667+
)
1668+
.fetch_one(&pool)
1669+
.await
1670+
.expect("inspect pg_locks");
1671+
assert_eq!(tuple_locks, 0, "no channel tuple locks while txns are held");
1672+
1673+
// Release all commits concurrently. Under 0022 these serialized on the
1674+
// channel row (each holding it across its WAL flush); under 0024 the
1675+
// shared advisory key admits them all. Join with a timeout so a
1676+
// regression fails fast instead of hanging the suite.
1677+
let commits = txs
1678+
.into_iter()
1679+
.map(|tx| tokio::spawn(async move { tx.commit().await }))
1680+
.collect::<Vec<_>>();
1681+
for c in commits {
1682+
tokio::time::timeout(std::time::Duration::from_secs(10), c)
1683+
.await
1684+
.expect("concurrent permanent-channel commits must not block")
1685+
.expect("join commit task")
1686+
.expect("commit succeeds");
1687+
}
1688+
1689+
let deadline: Option<DateTime<Utc>> = sqlx::query_scalar(
1690+
"SELECT ttl_deadline FROM channels WHERE community_id = $1 AND id = $2",
1691+
)
1692+
.bind(community_uuid)
1693+
.bind(channel)
1694+
.fetch_one(&pool)
1695+
.await
1696+
.expect("read deadline after commits");
1697+
assert_eq!(deadline, None, "permanent channel must remain untouched");
1698+
let stored: i64 = sqlx::query_scalar(
1699+
"SELECT count(*) FROM events WHERE community_id = $1 AND channel_id = $2",
1700+
)
1701+
.bind(community_uuid)
1702+
.bind(channel)
1703+
.fetch_one(&pool)
1704+
.await
1705+
.expect("count stored events");
1706+
assert_eq!(stored as usize, N);
1707+
}
1708+
16081709
#[tokio::test]
16091710
#[ignore = "requires Postgres"]
16101711
async fn get_event_by_id_is_scoped_when_event_id_collides_across_communities() {

‎crates/buzz-db/src/lib.rs‎

Lines changed: 36 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1220,12 +1220,13 @@ impl Db {
12201220
event::get_events_by_ids(&self.pool, community_id, ids).await
12211221
}
12221222

1223-
/// Exclusively claim the next due event-to-push matcher job.
1224-
pub async fn claim_due_push_match(
1223+
/// Exclusively claim a batch of due matcher jobs from one community.
1224+
pub async fn claim_due_push_match_batch(
12251225
&self,
1226+
limit: i64,
12261227
lease_until: DateTime<Utc>,
1227-
) -> Result<Option<push::ClaimedMatch>> {
1228-
push::claim_due_match(&self.pool, lease_until).await
1228+
) -> Result<Option<push::ClaimedMatchBatch>> {
1229+
push::claim_due_match_batch(&self.pool, limit, lease_until).await
12291230
}
12301231

12311232
/// Load active endpoint-enabled leases eligible for push matching.
@@ -1236,18 +1237,30 @@ impl Db {
12361237
push::active_match_leases(&self.pool, community).await
12371238
}
12381239

1239-
/// Complete a matcher job if its claim fence is still held.
1240-
pub async fn complete_push_match(&self, job: &push::ClaimedMatch) -> Result<bool> {
1241-
push::complete_match(&self.pool, job).await
1240+
/// Complete matcher jobs from one claimed batch while the fence holds.
1241+
pub async fn complete_push_match_batch(
1242+
&self,
1243+
community: CommunityId,
1244+
claim_id: uuid::Uuid,
1245+
event_ids: &[Vec<u8>],
1246+
) -> Result<u64> {
1247+
push::complete_match_batch(&self.pool, community, claim_id, event_ids).await
12421248
}
12431249

1244-
/// Release a matcher claim for retry at the supplied time.
1245-
pub async fn retry_push_match(
1250+
/// Release fenced matcher claims from one batch for retry.
1251+
pub async fn retry_push_match_batch(
12461252
&self,
1247-
job: &push::ClaimedMatch,
1253+
community: CommunityId,
1254+
claim_id: uuid::Uuid,
1255+
event_ids: &[Vec<u8>],
12481256
next: DateTime<Utc>,
1249-
) -> Result<bool> {
1250-
push::retry_match(&self.pool, job, next).await
1257+
) -> Result<u64> {
1258+
push::retry_match_batch(&self.pool, community, claim_id, event_ids, next).await
1259+
}
1260+
1261+
/// Delete exhausted matcher jobs (periodic sweep, off the claim path).
1262+
pub async fn reap_exhausted_push_matches(&self) -> Result<u64> {
1263+
push::reap_exhausted_matches(&self.pool).await
12511264
}
12521265

12531266
/// Idempotently enqueue a wake for a matched lease and event.
@@ -1534,6 +1547,17 @@ impl Db {
15341547
channel::is_member(&self.pool, community_id, channel_id, pubkey).await
15351548
}
15361549

1550+
/// Return the active (channel, pubkey) membership pairs among the given
1551+
/// sets, in one statement.
1552+
pub async fn membership_pairs(
1553+
&self,
1554+
community_id: CommunityId,
1555+
channel_ids: &[Uuid],
1556+
pubkeys: &[Vec<u8>],
1557+
) -> Result<Vec<(Uuid, Vec<u8>)>> {
1558+
channel::membership_pairs(&self.pool, community_id, channel_ids, pubkeys).await
1559+
}
1560+
15371561
/// Returns all active members of a channel.
15381562
pub async fn get_members(
15391563
&self,

‎crates/buzz-db/src/migration.rs‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -560,7 +560,7 @@ mod tests {
560560
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
561561
migrations.sort_by_key(|migration| migration.version);
562562

563-
assert_eq!(migrations.len(), 22);
563+
assert_eq!(migrations.len(), 24);
564564
assert_eq!(migrations[0].version, 1);
565565
assert_eq!(&*migrations[0].description, "initial schema");
566566
assert!(migrations[0]
@@ -852,6 +852,33 @@ mod tests {
852852
assert!(ttl_refresh.contains("DEFERRABLE INITIALLY DEFERRED"));
853853
assert!(ttl_refresh.contains("clock_timestamp()"));
854854
assert!(ttl_refresh.contains("NEW.kind <> 9007"));
855+
856+
// T1b push gate: the match-queue trigger only enqueues when the
857+
// community has an eligible lease, ordered against lease activations
858+
// through the shared/exclusive per-community advisory lock.
859+
assert_eq!(migrations[22].version, 23);
860+
let push_gate = migrations[22].sql.as_str();
861+
assert!(push_gate.contains("CREATE OR REPLACE FUNCTION enqueue_push_match_job"));
862+
assert!(push_gate.contains("pg_advisory_xact_lock_shared"));
863+
assert!(push_gate.contains("'buzz_push_gate:' || NEW.community_id::text"));
864+
assert!(push_gate.contains("endpoint_enabled"));
865+
866+
// T1a repair: the TTL refresh trigger synchronizes on a shared
867+
// per-channel advisory lock instead of FOR UPDATE on the channel row,
868+
// so permanent-channel commits no longer serialize.
869+
assert_eq!(migrations[23].version, 24);
870+
let ttl_shared = migrations[23].sql.as_str();
871+
assert!(ttl_shared
872+
.contains("CREATE OR REPLACE FUNCTION refresh_channel_ttl_after_event_insert"));
873+
assert!(ttl_shared.contains("pg_advisory_xact_lock_shared"));
874+
assert!(ttl_shared.contains("'buzz_channel_ttl:' || NEW.community_id::text"));
875+
// The row read must be a bare SELECT (comments describe the removed
876+
// FOR UPDATE; the executable body must not reintroduce it).
877+
assert!(ttl_shared.contains("SELECT ttl_seconds INTO channel_ttl"));
878+
assert!(!strip_sql_comments(ttl_shared)
879+
.to_lowercase()
880+
.contains("for update"));
881+
assert!(ttl_shared.contains("NEW.kind <> 9007"));
855882
}
856883

857884
#[test]
@@ -1094,7 +1121,7 @@ mod tests {
10941121
run_migrations(&pool)
10951122
.await
10961123
.expect("retry succeeds after operator repair");
1097-
assert_eq!(applied_versions(&pool).await.last().copied(), Some(22));
1124+
assert_eq!(applied_versions(&pool).await.last().copied(), Some(24));
10981125
}
10991126

11001127
#[tokio::test]

0 commit comments

Comments
 (0)