Skip to content

Commit b8b1ab5

Browse files
authored
perf(push): avoid empty push delivery claims (#8524)
Each enabled push delivery worker currently opens a claim transaction for every active community, even when its queue is empty. Discover eligible communities in bounded pages first, then use the existing claim and delivery path. No schema or index change is needed. Per-delivery optimizations (sharing serving-write leases across a batch and combining revalidation or membership checks), enqueue batching, and outbox pruning are out of scope and are being investigated separately, including their deletion, revocation, and stale-worker safety guarantees. ### Related issue No matching existing issue or PR found. ### Validation - 25 focused PostgreSQL tests passed, covering real-worker delivery, eligibility filtering, pagination fairness, concurrent claims, expired-claim recovery, and stale-worker fencing. The [full PostgreSQL CI suite](https://github.com/block/buzz/actions/runs/39675698081/job/119490943190) passed all 1,090 tests, including all seven new regressions. - Discovery was benchmarked with **43,000 communities**, existing indexes, and 30 warm executions per scenario. Empty queues required no community lookups; eight ready communities spread across the ID range required eight community index lookups. Results and the alternative-query comparison are below. - In the separate real-worker test with 1,000 empty communities, the original worker made 1,401 claim admissions in 1.8 seconds; the fixed worker made zero. One and four workers made 3 and 12 discovery reads respectively. - In the delivery test with 65 ready communities plus 1,000 empty communities, all 81 notifications reached an HTTP test gateway in 0.83 seconds versus 1.55 seconds with the original loop. - Independent standards and specification reviews, plus [Codex review](#8524 (comment)), found no remaining blockers. Final Rust linting, the full local unit/integration run, desktop JavaScript tests/build, web build, and mobile analysis/tests passed. The full local desktop test run had one unrelated timing failure in `reaps_descendant_on_timeout`; its isolated recheck passed. [CI](https://github.com/block/buzz/actions/runs/39675698081) passed, including the Windows retry after an unrelated subprocess timeout. These are isolated local measurements, not production latency guarantees or a simulation of full production concurrency. Production delivery validation remains part of the release process; this PR does not re-enable push or deploy the change. ### Alternative considered An outbox-first query selects due pending jobs and expired sending jobs with `UNION ALL`, applies the same eligibility checks, then returns distinct community IDs. This could avoid scanning communities with no ready work. However, the current query's `EXISTS` clause does not force PostgreSQL to probe the outbox once per community: the measured plans already avoid that work when queues are empty. Both query shapes were tested against the same **43,000-community** dataset using existing indexes. Median database execution times over 30 warm runs: | Scenario | Current query | Outbox-first query | | --- | ---: | ---: | | Empty outbox | 0.038 ms | 0.041 ms | | No ready work, 27,336 retained jobs | 0.060 ms | 0.066 ms | | Eight ready communities spread across the ID range, including expired sending jobs, plus retained history | 0.109 ms | 0.126 ms | | Eight ready communities, 128 invalid jobs, and 27,200 terminal jobs | 0.225 ms | 0.335 ms | Both queries returned identical community IDs in every scenario. Forced generic prepared plans produced similar results. The current query was retained because the alternative did not reduce measured execution time or eliminate a community scan present in the empty-queue plan. No query rewrite or additional index was justified by these measurements. --------- Signed-off-by: Tom Brow <[email protected]>
1 parent d727033 commit b8b1ab5

5 files changed

Lines changed: 584 additions & 24 deletions

File tree

‎crates/buzz-db/src/store/push.rs‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ use crate::error::Result;
1414
use crate::Db;
1515
use buzz_datastore_tracing::datastore_span;
1616

17+
mod delivery;
18+
pub use delivery::due_wake_communities;
19+
1720
async fn acquire_operation_connection(
1821
pool: &PgPool,
1922
operation: crate::observability::WriterOperation,
@@ -1482,6 +1485,18 @@ impl Db {
14821485
crate::push::enqueue_wakes(&self.pool, community, requests).await
14831486
}
14841487

1488+
/// Discover the next bounded page of communities with eligible delivery work.
1489+
///
1490+
/// Continue after the last returned ID until empty, then start a new sweep.
1491+
/// Discovery is advisory: claims and send-time validation remain authoritative.
1492+
#[datastore_span(name = "due_push_wake_communities", system = "postgresql")]
1493+
pub async fn due_push_wake_communities(
1494+
&self,
1495+
after: Option<CommunityId>,
1496+
) -> Result<Vec<CommunityId>> {
1497+
due_wake_communities(&self.pool, after).await
1498+
}
1499+
14851500
/// Exclusively claim due wake jobs for one community.
14861501
#[datastore_span(name = "claim_due_push_wakes", system = "postgresql")]
14871502
pub async fn claim_due_push_wakes(
@@ -1582,6 +1597,10 @@ impl Db {
15821597
}
15831598
}
15841599

1600+
#[cfg(test)]
1601+
#[path = "push/delivery_postgres_tests.rs"]
1602+
mod delivery_postgres_tests;
1603+
15851604
#[cfg(test)]
15861605
mod postgres_tests {
15871606
use super::*;
@@ -1638,7 +1657,7 @@ mod postgres_tests {
16381657
.expect("sign lease event")
16391658
}
16401659

1641-
async fn make_community(pool: &PgPool) -> CommunityId {
1660+
pub(super) async fn make_community(pool: &PgPool) -> CommunityId {
16421661
let id = Uuid::new_v4();
16431662
sqlx::query("INSERT INTO communities (id, host) VALUES ($1, $2)")
16441663
.bind(id)
@@ -1658,7 +1677,7 @@ mod postgres_tests {
16581677
}
16591678
}
16601679

1661-
async fn activate(
1680+
pub(super) async fn activate(
16621681
pool: &PgPool,
16631682
community: CommunityId,
16641683
author: &[u8],
@@ -2087,7 +2106,7 @@ mod postgres_tests {
20872106
assert_eq!(total, 3, "one row per distinct (endpoint, event) key");
20882107
}
20892108

2090-
async fn enqueue_one(
2109+
pub(super) async fn enqueue_one(
20912110
pool: &PgPool,
20922111
community: CommunityId,
20932112
author: &[u8],
@@ -2627,7 +2646,7 @@ mod postgres_tests {
26272646
.expect("seed matcher row");
26282647
}
26292648

2630-
async fn quiesce_test_community(pool: &PgPool, community: CommunityId) {
2649+
pub(super) async fn quiesce_test_community(pool: &PgPool, community: CommunityId) {
26312650
let mut lifecycle = pool.begin().await.expect("begin lifecycle fixture");
26322651
sqlx::query(
26332652
"SELECT set_config('buzz.deletion_executor_community', $1, true), \
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
use super::*;
2+
3+
// Use the existing due/recovery indexes. Filtering before LIMIT prevents
4+
// invalid jobs from starving live work.
5+
const DUE_COMMUNITIES_SQL: &str = r#"
6+
SELECT c.id
7+
FROM communities c
8+
WHERE ($1::uuid IS NULL OR c.id > $1)
9+
AND c.archived_at IS NULL
10+
AND c.deleted_at IS NULL
11+
AND c.deletion_state = 'active'
12+
AND EXISTS (
13+
SELECT 1
14+
FROM push_wake_outbox o
15+
JOIN push_leases l
16+
ON l.community_id = o.community_id
17+
AND l.author = o.author
18+
AND l.installation_id = o.installation_id
19+
AND l.generation = o.lease_generation
20+
AND l.endpoint_hash = o.endpoint_hash
21+
JOIN events e
22+
ON e.community_id = o.community_id
23+
AND e.id = o.event_id
24+
AND e.deleted_at IS NULL
25+
WHERE o.community_id = c.id
26+
AND o.expires_at > EXTRACT(EPOCH FROM now())::bigint
27+
AND o.next_attempt_at <= now()
28+
AND (o.state = 'pending' OR (o.state = 'sending' AND o.lease_until < now()))
29+
AND l.active
30+
AND l.endpoint_enabled
31+
AND l.expires_at > EXTRACT(EPOCH FROM now())::bigint
32+
)
33+
ORDER BY c.id
34+
LIMIT 64
35+
"#;
36+
37+
/// Read up to 64 communities with eligible due wakes, ordered after `after`.
38+
///
39+
/// This read takes no community admission locks and releases its connection
40+
/// before the caller claims work. A full sweep advances after each page's last
41+
/// ID, so a busy community cannot hide quieter communities beyond the first page.
42+
/// An empty page ends the sweep; newly due work behind the cursor is found on
43+
/// the next sweep. Callers must still claim and revalidate before sending.
44+
pub async fn due_wake_communities(
45+
pool: &PgPool,
46+
after: Option<CommunityId>,
47+
) -> Result<Vec<CommunityId>> {
48+
let mut connection =
49+
acquire_operation_connection(pool, crate::observability::WriterOperation::Maintenance)
50+
.await?;
51+
let ids: Vec<Uuid> = sqlx::query_scalar(DUE_COMMUNITIES_SQL)
52+
.bind(after.map(|community| *community.as_uuid()))
53+
.fetch_all(&mut *connection)
54+
.await?;
55+
Ok(ids.into_iter().map(CommunityId::from_uuid).collect())
56+
}

0 commit comments

Comments
 (0)