Skip to content

Commit c6340a4

Browse files
TheSentinel454Elrond
andauthored
fix(deletion): give the scheduled drain the cold-start DB connector (#8146)
## Summary The scheduled `buzz-admin deletions drain` Job never gets past its first database connection in a hosted staging deployment. Every run exits 5 after exactly 3 seconds with `pool timed out while waiting for an open connection`. The drain starts in a fresh pod with no mesh sidecar, and that pod's first connection can take longer than the relay's 3-second acquisition budget. `connect_store()` in `buzz-deletion` inherits that budget from `DbConfig::default()`, with no retry. In the same pod, the same command run 15 seconds later succeeds. TCP to Postgres, Redis, and S3 from a sidecar-less pod is fine. #7770 fixed the same failure for `buzz-admin storage-snapshot`, but only inside `buzz-admin`. This PR moves that connector into `buzz-db` as `Db::connect_cold_start` and uses it from both run-once workers: - **Pool settings:** a 30s acquisition budget, and no idle spare connections (`min_connections: 0`). - **Retries:** startup timeouts and transient transport errors retry at most twice, after 2s and 5s, so startup waits at most 97s. - **No retry:** configuration, authentication, TLS, and protocol errors fail immediately, and the error message stays free of credentials. - **Per-worker settings:** each worker keeps its own pool size. Storage-snapshot stays at 1 connection, and deletion stays at `BUZZ_DB_POOL_SIZE` (default 20). Each worker also keeps its session timeouts. - **Startup events:** the stderr events carry a worker prefix. Storage-snapshot still emits `storage_snapshot_db_connect_*`, and the deletion worker now emits `deletion_db_connect_*`. - **Error text:** the exhaustion message is now `{worker} database startup failed after …`. For storage-snapshot that reads `storage_snapshot database startup failed …` instead of `storage snapshot database startup failed …`. Nothing in this repo matches on that string. - **Session-setup failures:** SQLx retries a failed `after_connect` hook (writer floor guard, session timeouts, isolation assertion) until the acquire deadline. Those failures therefore surface as `timeout`, are retried, and fail closed after at most 97s instead of immediately. Relay pool defaults are unchanged. The deletion CLI's operator commands (`list`, `approve`, …) use the same connector. They normally run inside the warm relay container, where the first attempt succeeds. **What got simpler:** there is now one cold-start connector instead of a `buzz-admin`-private copy that the deletion worker couldn't reach. `storage_snapshot_startup.rs` is gone, and its unit tests moved with the code. The two Postgres startup tests now share one harness with the drain's tests (`postgres_storage_snapshot.rs` → `postgres_cold_start.rs`). ### Related issue Follow-up to #7770 (same failure, other worker) and #7827 (the drain CronJob). No duplicate found. ### Testing All at head `1545de7b5`: - **GitHub CI:** green on attempt 1, including `PostgreSQL Domain / PostgreSQL Tests`. - **Blox:** - fmt, `clippy --workspace --all-targets -D warnings`, and the `buzz-db` / `buzz-admin` / `buzz-deletion` tests pass. - All four `postgres_cold_start` tests pass, each against a freshly migrated database. - Falsification: with main's `connect_store` restored, the drain retry test fails. - **Agent review:** two rounds. All findings are fixed or disclosed; the error-text change is listed above. - **Agent exercise:** real `buzz-admin deletions list` / `deletions drain` against local Postgres 17 behind a fault proxy. - **5s delay:** succeeds on attempt 1 (3/3). Main fails at about 3.5s. - **Dropped handshake, then 5s delay:** attempt 1 fails with `UnexpectedEof`, then a 2s backoff, then attempt 2 succeeds (3/3). - **Wrong password, missing database or malformed URL:** exit 5 after one attempt. - **Persistent rejection:** exit 5 after three attempts, with 2s and 5s backoffs. - The deletion queue stayed empty throughout. - **Not covered:** an approved deletion, real S3 / Pod Identity, TLS and session-setup failures, and a live 97s stall. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent 7455019 commit c6340a4

8 files changed

Lines changed: 297 additions & 51 deletions

File tree

‎crates/buzz-admin/src/main.rs‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
2323
mod communities;
2424
mod deletions;
25-
mod storage_snapshot_startup;
2625

2726
use std::future::Future;
2827
use std::sync::atomic::{AtomicU64, Ordering};
@@ -196,14 +195,27 @@ async fn run(cli: Cli) -> Result<i32> {
196195
}
197196
}
198197

198+
/// One session: the worker detaches its lock-owning connection, and the
199+
/// cold-start pool opens no idle replacements while it scans S3.
200+
fn storage_snapshot_db_config(base: DbConfig) -> DbConfig {
201+
DbConfig {
202+
max_connections: 1,
203+
..base
204+
}
205+
}
206+
199207
async fn cmd_storage_snapshot(max_objects: u64) -> Result<i32> {
200208
let max_objects_db = i64::try_from(max_objects)
201209
.map_err(|_| anyhow::anyhow!("--max-objects must be at most {}", i64::MAX))?;
202210
if max_objects == 0 {
203211
return Err(anyhow::anyhow!("--max-objects must be greater than zero"));
204212
}
205213

206-
let db = storage_snapshot_startup::connect_db().await?;
214+
let db = Db::connect_cold_start(
215+
storage_snapshot_db_config(db_config_from_env()),
216+
"storage_snapshot",
217+
)
218+
.await?;
207219
let mut leader = db.try_lock_storage_accounting().await?.ok_or_else(|| {
208220
anyhow::anyhow!("another storage-snapshot worker already holds the lease")
209221
})?;
@@ -887,6 +899,17 @@ mod storage_snapshot_tests {
887899

888900
use super::*;
889901

902+
#[test]
903+
fn storage_snapshot_holds_a_single_database_session() {
904+
let config = storage_snapshot_db_config(DbConfig {
905+
max_connections: 20,
906+
lock_timeout_ms: 123,
907+
..DbConfig::default()
908+
});
909+
assert_eq!(config.max_connections, 1);
910+
assert_eq!(config.lock_timeout_ms, 123);
911+
}
912+
890913
#[tokio::test]
891914
async fn failed_fold_never_invokes_snapshot_persistence() {
892915
let persist_calls = Arc::new(AtomicUsize::new(0));

crates/buzz-admin/tests/postgres_storage_snapshot.rs renamed to crates/buzz-admin/tests/postgres_cold_start.rs

Lines changed: 85 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//! Run the real command through delayed/broken PostgreSQL connections.
1+
//! Run the real run-once workers through delayed/broken PostgreSQL connections.
22
//! The PostgreSQL nextest wrapper supplies a separate database per test.
33
44
use std::process::{ExitStatus, Stdio};
@@ -164,6 +164,39 @@ struct WorkerOutput {
164164
}
165165

166166
async fn run_worker(database_url: &str, proxy: &Server, s3: &Server) -> WorkerOutput {
167+
run_admin(
168+
database_url,
169+
proxy,
170+
&["storage-snapshot", "--max-objects", "100"],
171+
&[
172+
("BUZZ_S3_ENDPOINT", format!("http://{}", s3.address)),
173+
("BUZZ_STORAGE_SNAPSHOT_CODE_SHA", "startup-test".to_owned()),
174+
],
175+
)
176+
.await
177+
}
178+
179+
/// Run the scheduled deletion drain against an idle queue. Its Redis pool is
180+
/// lazy and an idle drain never reaches S3, so neither endpoint needs to exist.
181+
async fn run_drain(database_url: &str, proxy: &Server) -> WorkerOutput {
182+
run_admin(
183+
database_url,
184+
proxy,
185+
&["deletions", "drain", "--executor-id", "cold-start-test"],
186+
&[
187+
("BUZZ_S3_ENDPOINT", "http://127.0.0.1:9".to_owned()),
188+
("REDIS_URL", "redis://127.0.0.1:9".to_owned()),
189+
],
190+
)
191+
.await
192+
}
193+
194+
async fn run_admin(
195+
database_url: &str,
196+
proxy: &Server,
197+
args: &[&str],
198+
env: &[(&str, String)],
199+
) -> WorkerOutput {
167200
let mut url = Url::parse(database_url).expect("database URL");
168201
url.set_host(Some("127.0.0.1")).expect("proxy host");
169202
url.set_port(Some(proxy.address.port()))
@@ -176,15 +209,14 @@ async fn run_worker(database_url: &str, proxy: &Server, s3: &Server) -> WorkerOu
176209
let worker_binary = std::env::var_os("NEXTEST_BIN_EXE_buzz_admin")
177210
.unwrap_or_else(|| env!("CARGO_BIN_EXE_buzz-admin").into());
178211
let mut child = Command::new(worker_binary)
179-
.args(["storage-snapshot", "--max-objects", "100"])
212+
.args(args)
180213
.env_clear()
181214
.env("DATABASE_URL", url.as_str())
182-
.env("BUZZ_S3_ENDPOINT", format!("http://{}", s3.address))
183215
.env("BUZZ_S3_BUCKET", "test-bucket")
184216
.env("BUZZ_S3_REGION", "us-east-1")
185217
.env("BUZZ_S3_ACCESS_KEY", "test-access-key")
186218
.env("BUZZ_S3_SECRET_KEY", "test-secret-key")
187-
.env("BUZZ_STORAGE_SNAPSHOT_CODE_SHA", "startup-test")
219+
.envs(env.iter().map(|(key, value)| (key, value)))
188220
.stdout(Stdio::piped())
189221
.stderr(Stdio::piped())
190222
.kill_on_drop(true)
@@ -290,3 +322,52 @@ async fn storage_snapshot_exhaustion_never_lists_s3_or_replaces_the_last_good_sn
290322
assert_eq!(snapshot, json!({"previous": "complete"}));
291323
assert_eq!(revision, "before");
292324
}
325+
326+
#[tokio::test]
327+
#[ignore = "requires Postgres"]
328+
async fn deletion_drain_retries_then_connects_after_the_old_three_second_budget() {
329+
let (url, _pool) = database().await;
330+
let proxy = postgres_proxy(&url, 1, Duration::from_secs(5)).await;
331+
let output = run_drain(&url, &proxy).await;
332+
assert!(
333+
output.status.success(),
334+
"{}\n{}",
335+
output.stdout,
336+
output.stderr
337+
);
338+
// The rejected dial plus the successful retry. The drain's first claim
339+
// can race SQLx's asynchronous release of the startup connection and dial
340+
// one more session, so the count is a lower bound; the startup events
341+
// below are what pin the retry.
342+
assert!(proxy.requests.load(Ordering::SeqCst) >= 2);
343+
let startup = events(&output.stderr);
344+
assert!(startup
345+
.iter()
346+
.any(|event| event["event"] == "deletion_db_connect_failed"
347+
&& event["attempt"] == 1
348+
&& event["retry_in_ms"] == 2000));
349+
assert!(startup
350+
.iter()
351+
.any(|event| event["event"] == "deletion_db_connect_completed"
352+
&& event["attempt"] == 2
353+
&& event["attempt_elapsed_ms"].as_u64().unwrap() >= 5000));
354+
}
355+
356+
#[tokio::test]
357+
#[ignore = "requires Postgres"]
358+
async fn deletion_drain_exhaustion_fails_after_three_bounded_attempts() {
359+
let (url, _pool) = database().await;
360+
let proxy = postgres_proxy(&url, usize::MAX, Duration::ZERO).await;
361+
let output = run_drain(&url, &proxy).await;
362+
assert_eq!(output.status.code(), Some(5), "{}", output.stderr);
363+
assert_eq!(proxy.requests.load(Ordering::SeqCst), 3);
364+
let failures: Vec<_> = events(&output.stderr)
365+
.into_iter()
366+
.filter(|event| event["event"] == "deletion_db_connect_failed")
367+
.collect();
368+
assert_eq!(failures.len(), 3);
369+
assert!(failures.last().unwrap()["retry_in_ms"].is_null());
370+
assert!(output
371+
.stderr
372+
.contains("deletion database startup failed after 3 attempt(s)"));
373+
}

‎crates/buzz-db/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,5 +26,5 @@ rand = { workspace = true }
2626
metrics = { workspace = true }
2727

2828
[dev-dependencies]
29-
tokio = { workspace = true }
29+
tokio = { workspace = true, features = ["test-util"] }
3030
metrics-util = { workspace = true }

‎crates/buzz-db/src/lib.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,9 @@ pub mod error;
3131
mod test_support;
3232

3333
pub use runtime::{
34-
insert_mentions, migration, replica_fence, AdmittedTx, Db, DbConfig, DbConnectionOutcome,
35-
DbConnectionStep, DbPoolRole, DbPoolStats, DbReadinessOutcome, ReadSession,
34+
insert_mentions, migration, replica_fence, AdmittedTx, ColdStartError, Db, DbConfig,
35+
DbConnectionOutcome, DbConnectionStep, DbPoolRole, DbPoolStats, DbReadinessOutcome,
36+
ReadSession,
3637
};
3738

3839
/// Valid low-cardinality `(pool_role, operation)` pairs for pool-acquisition telemetry.

crates/buzz-admin/src/storage_snapshot_startup.rs renamed to crates/buzz-db/src/runtime/cold_start.rs

Lines changed: 74 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,86 @@
1-
//! Bounded cold database startup for the run-once storage accounting worker.
1+
//! Bounded cold database startup for run-once operator workers.
2+
//!
3+
//! A freshly scheduled worker pod (no mesh sidecar, cold DNS) can need more
4+
//! than the relay's three-second acquisition budget to open its first
5+
//! connection. Workers that start, do bounded work, and exit share this
6+
//! connector so a slow first connection is waited out and transient transport
7+
//! failures are retried, while configuration, authentication, TLS, and
8+
//! protocol errors during the dial still fail immediately. SQLx treats a
9+
//! failed session-setup hook (`after_connect`) as retryable until the
10+
//! acquire deadline, so such failures surface here as timeouts and are
11+
//! retried within the same bounded budget.
12+
//!
13+
//! Startup attempts are reported as JSON lines on stderr because these
14+
//! command-line workers do not install a tracing subscriber.
215
316
use std::future::Future;
417
use std::io::ErrorKind;
518
use std::time::Duration;
619

7-
use anyhow::{Context, Result};
8-
use buzz_db::{Db, DbConfig, DbError};
920
use tokio::time::{sleep, timeout, Instant};
1021

22+
use super::{Db, DbConfig};
23+
use crate::DbError;
24+
1125
const ACQUIRE_TIMEOUT: Duration = Duration::from_secs(30);
1226
const RETRY_DELAYS: [Duration; 2] = [Duration::from_secs(2), Duration::from_secs(5)];
1327

14-
/// Establish the worker's database pool before it acquires its accounting lease.
15-
pub(super) async fn connect_db() -> Result<Db> {
16-
let config = worker_config(crate::db_config_from_env());
17-
connect_with_retry(|| Db::new(&config)).await
28+
/// A run-once worker could not establish its database pool.
29+
///
30+
/// The message carries only a bounded classification: driver messages and
31+
/// URLs can contain credentials. The driver error stays available as the
32+
/// [`std::error::Error::source`].
33+
#[derive(Debug, thiserror::Error)]
34+
#[error(
35+
"{worker} database startup failed after {attempts} attempt(s) and {elapsed_ms}ms ({class})"
36+
)]
37+
pub struct ColdStartError {
38+
worker: &'static str,
39+
attempts: usize,
40+
elapsed_ms: u128,
41+
class: &'static str,
42+
#[source]
43+
source: DbError,
44+
}
45+
46+
impl Db {
47+
/// Connect a run-once worker's pool with a bounded cold-start budget.
48+
///
49+
/// The pool keeps the caller's size and session policy, opens no idle
50+
/// spare connections, and waits up to 30 seconds per acquisition. Startup
51+
/// timeouts and transient transport failures are retried at most twice,
52+
/// after two and five seconds, so startup waits at most 97 seconds.
53+
/// `worker` prefixes the stderr startup events, for example
54+
/// `deletion_db_connect_failed`.
55+
///
56+
/// Only initial connection establishment is retried. Once a caller holds
57+
/// a session-bound lock or lease, reconnecting would lose that fence.
58+
pub async fn connect_cold_start(
59+
config: DbConfig,
60+
worker: &'static str,
61+
) -> std::result::Result<Self, ColdStartError> {
62+
let config = cold_start_config(config);
63+
connect_with_retry(worker, || Db::new(&config)).await
64+
}
1865
}
1966

20-
fn worker_config(config: DbConfig) -> DbConfig {
67+
fn cold_start_config(config: DbConfig) -> DbConfig {
2168
DbConfig {
22-
max_connections: 1,
23-
// The worker detaches its lock-owning session. Do not open idle
24-
// replacements while that session scans S3 and publishes the result.
69+
// A run-once worker only needs connections it actually uses; idle
70+
// replacements would also contend with a slow cold start.
2571
min_connections: 0,
2672
acquire_timeout_secs: ACQUIRE_TIMEOUT.as_secs(),
2773
..config
2874
}
2975
}
3076

31-
// Only initial connection establishment is retried. Once the command owns
32-
// the advisory lock, reconnecting would lose its publication fence.
33-
async fn connect_with_retry<T, Connect, Attempt>(mut connect: Connect) -> Result<T>
77+
async fn connect_with_retry<T, Connect, Attempt>(
78+
worker: &'static str,
79+
mut connect: Connect,
80+
) -> std::result::Result<T, ColdStartError>
3481
where
3582
Connect: FnMut() -> Attempt,
36-
Attempt: Future<Output = buzz_db::Result<T>>,
83+
Attempt: Future<Output = crate::Result<T>>,
3784
{
3885
let started = Instant::now();
3986
// Three attempts of at most 30s, plus 2s and 5s backoffs: at most 97s.
@@ -42,7 +89,7 @@ where
4289
eprintln!(
4390
"{}",
4491
serde_json::json!({
45-
"event": "storage_snapshot_db_connect_started",
92+
"event": format!("{worker}_db_connect_started"),
4693
"stage": "db_connect",
4794
"attempt": attempt + 1,
4895
"timeout_ms": ACQUIRE_TIMEOUT.as_millis(),
@@ -57,7 +104,7 @@ where
57104
eprintln!(
58105
"{}",
59106
serde_json::json!({
60-
"event": "storage_snapshot_db_connect_completed",
107+
"event": format!("{worker}_db_connect_completed"),
61108
"stage": "db_connect",
62109
"attempt": attempt + 1,
63110
"attempt_elapsed_ms": attempt_started.elapsed().as_millis(),
@@ -78,7 +125,7 @@ where
78125
eprintln!(
79126
"{}",
80127
serde_json::json!({
81-
"event": "storage_snapshot_db_connect_failed",
128+
"event": format!("{worker}_db_connect_failed"),
82129
"stage": "db_connect",
83130
"attempt": attempt + 1,
84131
"attempt_elapsed_ms": attempt_started.elapsed().as_millis(),
@@ -98,12 +145,12 @@ where
98145
match delay {
99146
Some(delay) => sleep(delay).await,
100147
None => {
101-
return Err(error).with_context(|| {
102-
format!(
103-
"storage snapshot database startup failed after {} attempt(s) and {}ms ({class})",
104-
attempt + 1,
105-
started.elapsed().as_millis(),
106-
)
148+
return Err(ColdStartError {
149+
worker,
150+
attempts: attempt + 1,
151+
elapsed_ms: started.elapsed().as_millis(),
152+
class,
153+
source: error,
107154
});
108155
}
109156
}
@@ -128,7 +175,9 @@ fn retryable(error: &DbError) -> bool {
128175
| ErrorKind::NotFound
129176
| ErrorKind::Unsupported
130177
),
131-
// Includes authentication, TLS, URL configuration and protocol errors.
178+
// Includes authentication, TLS, URL configuration and protocol errors
179+
// raised while dialing. Session-setup failures arrive as
180+
// `PoolTimedOut` (see the module docs).
132181
_ => false,
133182
}
134183
}

0 commit comments

Comments
 (0)