Skip to content

Commit 6e6173a

Browse files
wesbillmanCarlclaude
authored
test(buzz-db): fix two PostgreSQL-lane timing flakes (#8472)
## Summary Two timing flakes in the PostgreSQL lane, one fix each. ### Deletion lock order `abort_serializes_before_fence_without_deadlock` failed on main ([run 39330176879](https://github.com/block/buzz/actions/runs/39330176879)): `post-lock lease verification rejects aborted request: 1`. The test slept 50ms and assumed `abort` had queued on the community advisory lock by then. On a slow runner, `fence` queued first, won the lock, and succeeded. The test now waits until `pg_locks` shows each contender waiting on that community's lock, `abort` first and then `fence`. Postgres grants queued advisory locks in arrival order, so `abort` always wins. The sleeps and their timing guess are gone. ### Saturated-reader fixture `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold` failed this PR's first run ([run 39335038111](https://github.com/block/buzz/actions/runs/39335038111)): `establish the reader's sole connection: PoolTimedOut`. Setup opens the reader's only connection through the production lazy pool, which has a 150ms acquire timeout. On a loaded runner that first connect took longer, so the test failed before reaching the budget it measures. Setup now retries on `PoolTimedOut` within 5s. The pool and its 150ms budget are unchanged for the routed read under test. Test-only. Production code is unchanged. ## Validation - Deletion test against an isolated local Postgres 17 database: 40/40 passes. Saturated-reader test: 15/15. - `cargo fmt --check`, `cargo clippy -p buzz-db --tests -D warnings`, pre-push Rust unit tests: pass. - Three other `store::deletion::postgres_tests` fail locally on unmodified `main` as well (local environment). CI's PostgreSQL lane is the authoritative check for them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Wes <[email protected]> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent edbfbf0 commit 6e6173a

2 files changed

Lines changed: 41 additions & 7 deletions

File tree

‎crates/buzz-db/src/runtime/tests.rs‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1982,11 +1982,20 @@ async fn routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold() {
19821982
db.set_replica_read_max_age_for_tests(Some(Duration::from_secs(5)));
19831983

19841984
let read_pool = db.read_pool.clone().expect("reader pool configured");
1985-
// Establish and hold the reader's only connection: saturated.
1986-
let held = read_pool
1987-
.acquire()
1988-
.await
1989-
.expect("establish the reader's sole connection");
1985+
// Establish and hold the reader's only connection: saturated. A fresh
1986+
// connection can outlast the 150ms acquire budget on a loaded runner, and
1987+
// the budget is under test only once the connection is held.
1988+
let held = tokio::time::timeout(Duration::from_secs(5), async {
1989+
loop {
1990+
match read_pool.acquire().await {
1991+
Err(sqlx::Error::PoolTimedOut) => continue,
1992+
connection => break connection,
1993+
}
1994+
}
1995+
})
1996+
.await
1997+
.expect("reader connects within 5s")
1998+
.expect("establish the reader's sole connection");
19901999
assert_eq!(
19912000
db.read_max_connections, 1,
19922001
"reader max must report 1 for this fixture to test saturation"

‎crates/buzz-db/src/store/deletion.rs‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4833,6 +4833,29 @@ DROP FUNCTION community_write_fence_excluded_table(NAME);
48334833
.expect("contender reached community row lock");
48344834
}
48354835

4836+
async fn wait_for_community_lock_waiters(db: &Db, community: CommunityId, waiters: i64) {
4837+
tokio::time::timeout(std::time::Duration::from_secs(5), async {
4838+
loop {
4839+
let queued: i64 = sqlx::query_scalar(
4840+
"SELECT count(*) FROM pg_locks \
4841+
WHERE locktype = 'advisory' AND objsubid = 1 AND NOT granted \
4842+
AND ((classid::bigint << 32) | objid::bigint) \
4843+
= community_deletion_lock_key($1)",
4844+
)
4845+
.bind(community.as_uuid())
4846+
.fetch_one(&db.pool)
4847+
.await
4848+
.expect("inspect community lock queue");
4849+
if queued >= waiters {
4850+
break;
4851+
}
4852+
tokio::task::yield_now().await;
4853+
}
4854+
})
4855+
.await
4856+
.expect("contenders queued on community lock");
4857+
}
4858+
48364859
async fn membership_roles(db: &Db, community: CommunityId) -> Vec<(String, String)> {
48374860
sqlx::query_as(
48384861
"SELECT pubkey, role FROM relay_members WHERE community_id = $1 ORDER BY pubkey",
@@ -6778,15 +6801,17 @@ DROP FUNCTION community_write_fence_excluded_table(NAME);
67786801
.abort(request.id, "operator", "race recovery")
67796802
.await
67806803
});
6781-
tokio::time::sleep(Duration::from_millis(50)).await;
6804+
// Postgres grants queued advisory locks in arrival order, so abort must
6805+
// be queued before fence is spawned for the outcome to be deterministic.
6806+
wait_for_community_lock_waiters(&db, request.community_id, 1).await;
67826807
assert!(
67836808
!aborting.is_finished(),
67846809
"abort must wait for the community lock"
67856810
);
67866811
let forward_store = store.clone();
67876812
let lease = claim.lease.clone();
67886813
let forwarding = tokio::spawn(async move { forward_store.fence(&lease).await });
6789-
tokio::time::sleep(Duration::from_millis(50)).await;
6814+
wait_for_community_lock_waiters(&db, request.community_id, 2).await;
67906815
assert!(
67916816
!forwarding.is_finished(),
67926817
"fence must queue on the same lock"

0 commit comments

Comments
 (0)