Skip to content

Commit 51c2721

Browse files
matt2eclaude
andcommitted
fix(db): unlock the storage-accounting lease before its session closes
## Problem The PostgreSQL lane's `storage_sweep::postgres_tests::worker_publication_activates_reader_without_restart_or_mode_change` failed with "called `Option::unwrap()` on a `None` value" at crates/buzz-relay/src/storage_sweep/postgres_tests.rs:66:74 after 5.53 s, on the unrelated branch elrond/cli-credential-chain: https://github.com/block/buzz/actions/runs/41730818977/job/126253029211 ## Cause Line 66 is `db.try_lock_storage_accounting().await.unwrap().unwrap()` inside the loop over three metrics modes. `Ok(None)` means `pg_try_advisory_lock` returned false. A passing run takes about 15 s, roughly 5 s per iteration, so the failure is the start of iteration 2. Each test process gets its own database and advisory locks are per-database, so the holder is iteration 1's own worker. `StorageAccountingLeader` holds a session-level advisory lock on a detached connection. The test ended each iteration with `drop(worker)`, which only closes the socket. PostgreSQL frees session locks during backend exit, after it notices EOF, so the lock can outlive the drop on a loaded server. Iteration 2's lock attempt raced that exit. The db-crate test `lost_lock_session_cannot_overwrite_successor_snapshot` already polls for this release. The relay test didn't. ## Fix Add `StorageAccountingLeader::release(self)`. It runs `pg_advisory_unlock` on the leader's own session and then closes it, so the lease is free when it returns. This follows the unlock-then-close pattern in `migration.rs`. The relay test now releases the worker at the end of each iteration instead of dropping it. No assertions, timeouts or retries change. buzz-admin's `storage-snapshot` is left as is. It runs once per pod, and its CronJob uses `concurrencyPolicy: Forbid`, `backoffLimit: 0` and `restartPolicy: Never`, so the next lock attempt comes from a later scheduled run, not from the same process. `released_lease_admits_successor_before_old_session_exits` covers the new method. A closing backend drops its temp tables before it frees session locks. So the test locks a temp table owned by the leader's session from another transaction, which keeps the old backend alive after release. It then asserts that a fresh `try_lock` succeeds at once and that the old backend has not exited. Open PRs #9040 (edits this relay test and keeps `drop(worker)`) and #9043 (changes the `sqlx` import in storage_accounting.rs) will see small textual conflicts. ## Validation Local Linux workstation, throwaway `postgres:16-alpine` (CI uses `postgres:16`), desired-state schema applied as in the PostgreSQL lane: - Adversarial ordering, using temporary, uncommitted changes: the leader's connection was made public and the relay test held the worker backend's exit for 1 s with the same temp-table lock. Without the fix: 15/15 failed at postgres_tests.rs:66:74 after 5.1-5.3 s, the CI signature. With the fix: 10/10 passed. - Natural contention: Postgres, the test processes and busy loops were pinned to 4 CPUs. With 8 parallel runs and 4 busy loops, the unmodified test passed 24/24 without the fix. With 16 parallel runs and 16 busy loops, it passed 48/48 without the fix and 48/48 with it. Load alone did not open the window locally. - New db test: 50/50 passed. With the unlock removed from `release`, leaving only `close`, it fails with "a released lease must admit a successor at once". All four storage_accounting Postgres tests pass. - `cargo fmt --all -- --check`, `cargo clippy -p buzz-db -p buzz-relay -p buzz-admin --all-targets -- -D warnings`, `just file-size-check` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mtoohey <[email protected]> (cherry picked from commit 43fc3ddc6d785f61836dfac2964a9c8eea7e17cf)
1 parent 64d12e2 commit 51c2721

2 files changed

Lines changed: 76 additions & 2 deletions

File tree

‎crates/buzz-db/src/store/storage_accounting.rs‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
33
use buzz_datastore_tracing::datastore_span;
44
use chrono::{DateTime, Utc};
5-
use sqlx::postgres::PgConnection;
5+
use sqlx::{postgres::PgConnection, Connection};
66
use uuid::Uuid;
77

88
use crate::{observability, Db, Result};
@@ -70,6 +70,22 @@ impl StorageAccountingLeader {
7070
.await?;
7171
Ok(())
7272
}
73+
74+
/// Unlock the lease on its own session, then close that session.
75+
///
76+
/// Dropping the leader frees the lease only once PostgreSQL notices the
77+
/// closed socket and ends the old backend, which can lag on a loaded
78+
/// server. Once this returns, the lease is already free for a
79+
/// successor's [`Db::try_lock_storage_accounting`].
80+
#[datastore_span(name = "release_storage_accounting", system = "postgresql")]
81+
pub async fn release(mut self) -> Result<()> {
82+
sqlx::query("SELECT pg_advisory_unlock($1)")
83+
.bind(STORAGE_ACCOUNTING_LOCK_KEY)
84+
.execute(&mut self.connection)
85+
.await?;
86+
self.connection.close().await?;
87+
Ok(())
88+
}
7389
}
7490

7591
impl Db {
@@ -258,6 +274,62 @@ mod postgres_tests {
258274
assert_eq!(stored.code_sha, "b");
259275
}
260276

277+
#[tokio::test]
278+
#[ignore = "requires Postgres"]
279+
async fn released_lease_admits_successor_before_old_session_exits() {
280+
let (pool, _scratch) =
281+
ScratchDb::migrated(&crate::test_support::database_url(), "storage_accounting").await;
282+
let db = Db::from_pool(pool.clone());
283+
284+
let mut leader = db
285+
.try_lock_storage_accounting()
286+
.await
287+
.expect("first lock")
288+
.expect("first worker owns lock");
289+
// A closing backend drops its temp tables before it frees session
290+
// locks. Locking the leader's temp table keeps that backend alive
291+
// after release, so only an explicit unlock can admit a successor.
292+
sqlx::query("CREATE TEMP TABLE exit_fence ()")
293+
.execute(&mut leader.connection)
294+
.await
295+
.expect("create leader temp table");
296+
let (temp_schema, leader_pid) = sqlx::query_as::<_, (String, i32)>(
297+
"SELECT pg_my_temp_schema()::regnamespace::text, pg_backend_pid()",
298+
)
299+
.fetch_one(&mut leader.connection)
300+
.await
301+
.expect("load leader session");
302+
let mut exit_fence = pool.begin().await.expect("begin exit fence");
303+
sqlx::query(sqlx::AssertSqlSafe(format!(
304+
"LOCK TABLE {temp_schema}.exit_fence IN ACCESS SHARE MODE"
305+
)))
306+
.execute(&mut *exit_fence)
307+
.await
308+
.expect("hold the leader's temp-table cleanup");
309+
310+
leader.release().await.expect("release lease");
311+
let successor = db
312+
.try_lock_storage_accounting()
313+
.await
314+
.expect("successor lock attempt");
315+
assert!(
316+
successor.is_some(),
317+
"a released lease must admit a successor at once"
318+
);
319+
let leader_alive = sqlx::query_scalar::<_, bool>(
320+
"SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = $1)",
321+
)
322+
.bind(leader_pid)
323+
.fetch_one(&pool)
324+
.await
325+
.expect("load leader backend");
326+
assert!(leader_alive, "the old backend must not have exited yet");
327+
exit_fence
328+
.rollback()
329+
.await
330+
.expect("let the old backend exit");
331+
}
332+
261333
#[tokio::test]
262334
#[ignore = "requires Postgres"]
263335
async fn usage_delivery_reads_snapshot_time_and_community_states() {

‎crates/buzz-relay/src/storage_sweep/postgres_tests.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,9 @@ async fn worker_publication_activates_reader_without_restart_or_mode_change() {
145145
values["buzz_total_storage_bytes"], 0.0,
146146
"a real empty snapshot may report zero"
147147
);
148-
drop(worker);
148+
// A dropped worker keeps the lease until its backend exits, which can
149+
// outlast the next iteration's lock attempt.
150+
worker.release().await.expect("release worker lease");
149151
}
150152
pool.close().await;
151153
}

0 commit comments

Comments
 (0)