Skip to content

Commit 4c2086c

Browse files
authored
feat(db): expose connection setup metrics (block#7286)
## Why Current pool metrics show checkout outcomes and pool state after the fact. They do not show when a checkout began or which writer-connection setup step failed. During startup and pool growth, operators need to distinguish pool saturation from a slow or unsafe connection setup. ## What - Count every instrumented database checkout when it starts, with a fixed operation label. - Measure writer-pool creation and every physical writer connection across `physical_connect`, `created_at_floor`, `session_timeouts`, `isolation`, and `ready`. - Publish fixed-cardinality start, terminal-outcome, duration, and waiter metrics without database URLs, SQL, raw errors, connection ordinals, or per-connection lifecycle logs. - Document how to identify the active setup bottleneck during a rollout or incident. ## How The existing typed checkout wrapper records starts and current waiters. The huddle-history path uses that wrapper instead of a raw pool checkout. The production SQLx `after_connect` hook records start, terminal outcome, and duration for each sequential safety step. A drop guard records cancellation once if setup exits before a terminal result. Fixed enums keep every label bounded. For a phase, `started_total - sum(attempts_total)` is the number of in-progress attempts on that pod. Because setup is sequential, a later phase starting also proves the earlier phases succeeded. The database path emits metrics only. This PR intentionally does not add per-connection lifecycle receipts or direct `stderr` writes. ## Risk Medium. This changes the production writer-pool `after_connect` hook and adds one counter update to instrumented checkout paths. It does not change the database safety statements or their failure behavior. The telemetry has fixed labels and no per-connection identifiers. ## Testing At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release relay was started against isolated local PostgreSQL, Redis, and MinIO services. The main health endpoint returned `ok`, readiness returned `{"status":"ready"}`, the connection-step metrics were exported, and no database lifecycle receipts were emitted. A release CLI then created a channel, sent a message, and read the same event back successfully. The earlier staging deployment used pre-rebase head `f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed multi-architecture image came from [GitHub Actions run 33780888255](https://github.com/block/buzz/actions/runs/33780888255), manifest `sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`. The [staging deployment](squareup/builderbot-platform-core-infrastructure#317) brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero restarts, and the connection metrics produced data in the [rollout dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety). That staging image predates the cleanup that removed per-connection lifecycle logs; the metric schema and database safety statements are unchanged. ## Verification - `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across the package and integration target; 255 opt-in tests remained ignored. - Four opt-in production-path PostgreSQL regressions passed: initial minimum connections, post-startup pool growth, isolation failure, and session-timeout setup failure. - All 35 relay media tests passed against the isolated PostgreSQL database. - Rust formatting, workspace clippy, desktop/Tauri clippy, web checks, mobile analysis, security checks, and file-size policy checks passed. - The repository-wide unit stage also exposed three unrelated existing `buzz-acp` failures: one timing-sensitive keepalive test and two environment-default tests. ## Bigger picture This is the database-metrics part of the startup and rollout observability work. The early-startup lifecycle foundation merged in block#7258, so the rebase removed that duplicate commit from this PR. Originating discussion: buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71 Generated with Codex --------- Signed-off-by: Ravneet Arora <[email protected]>
1 parent 6b1e45d commit 4c2086c

8 files changed

Lines changed: 935 additions & 25 deletions

File tree

‎crates/buzz-db/src/lib.rs‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ pub mod error;
3131
mod test_support;
3232

3333
pub use runtime::{
34-
insert_mentions, migration, replica_fence, Db, DbConfig, DbPoolRole, DbPoolStats,
35-
DbReadinessOutcome, ReadSession,
34+
insert_mentions, migration, replica_fence, Db, DbConfig, DbConnectionOutcome, DbConnectionStep,
35+
DbPoolRole, DbPoolStats, DbReadinessOutcome, ReadSession,
3636
};
3737

3838
/// Valid low-cardinality `(pool_role, operation)` pairs for pool-acquisition telemetry.
@@ -42,6 +42,21 @@ pub const DB_POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 11] =
4242
/// Raw Prometheus series ceiling per relay pod for the operation-aware contract.
4343
pub const DB_POOL_ACQUIRE_RAW_SERIES_PER_POD: usize =
4444
runtime::observability::POOL_ACQUIRE_RAW_SERIES_PER_POD;
45+
46+
/// Valid database connection role/step pairs with a start counter.
47+
pub const DB_CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 4] =
48+
runtime::CONNECTION_STARTED_STEPS;
49+
50+
/// Valid database connection role/step pairs with a duration histogram.
51+
pub const DB_CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 4] =
52+
runtime::CONNECTION_DURATION_STEPS;
53+
54+
/// Valid database connection role/step/outcome terminal combinations.
55+
pub const DB_CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 15] =
56+
runtime::CONNECTION_TERMINALS;
57+
58+
/// Raw Prometheus series ceiling per pod for connection-step telemetry.
59+
pub const DB_CONNECTION_RAW_SERIES_PER_POD: usize = runtime::CONNECTION_RAW_SERIES_PER_POD;
4560
pub(crate) use runtime::{
4661
insert_mentions_in_transaction, observability, route_proof, ReadSessionInner, RouteDecision,
4762
RoutePredicate,
Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
//! Fixed-vocabulary metrics for writer-pool connection setup.
2+
//!
3+
//! SQLx exposes the point immediately after a physical connection succeeds,
4+
//! but it does not expose a callback immediately before each physical dial.
5+
//! Consequently, `physical_connect` is a success milestone rather than a
6+
//! duration phase. The aggregate `writer_pool` phase owns failures that occur
7+
//! before `after_connect`, while the session phases own their exact failures.
8+
9+
use std::time::{Duration, Instant};
10+
11+
use super::DbPoolRole;
12+
13+
/// Fixed writer connection-setup steps.
14+
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
15+
pub enum DbConnectionStep {
16+
/// Construct the writer pool and satisfy its initial minimum size.
17+
WriterPool,
18+
/// A physical connection has completed DNS/network/TLS/authentication.
19+
PhysicalConnect,
20+
/// Install the created-at replica-fence floor.
21+
CreatedAtFloor,
22+
/// Install lock, idle-transaction, and statement timeouts.
23+
SessionTimeouts,
24+
/// Verify READ COMMITTED transaction isolation.
25+
Isolation,
26+
/// The physical connection passed every required session premise.
27+
Ready,
28+
}
29+
30+
impl DbConnectionStep {
31+
/// Complete metric-label vocabulary.
32+
pub const ALL: [Self; 6] = [
33+
Self::WriterPool,
34+
Self::PhysicalConnect,
35+
Self::CreatedAtFloor,
36+
Self::SessionTimeouts,
37+
Self::Isolation,
38+
Self::Ready,
39+
];
40+
41+
/// Stable metric label.
42+
pub const fn as_str(self) -> &'static str {
43+
match self {
44+
Self::WriterPool => "writer_pool",
45+
Self::PhysicalConnect => "physical_connect",
46+
Self::CreatedAtFloor => "created_at_floor",
47+
Self::SessionTimeouts => "session_timeouts",
48+
Self::Isolation => "isolation",
49+
Self::Ready => "ready",
50+
}
51+
}
52+
}
53+
54+
/// Bounded terminal outcome for a connection-setup step.
55+
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
56+
pub enum DbConnectionOutcome {
57+
/// The step completed successfully.
58+
Succeeded,
59+
/// The step failed.
60+
Failed,
61+
/// The aggregate pool deadline expired.
62+
TimedOut,
63+
/// The owning future was dropped before a terminal.
64+
Cancelled,
65+
}
66+
67+
impl DbConnectionOutcome {
68+
/// Stable metric label.
69+
pub const fn as_str(self) -> &'static str {
70+
match self {
71+
Self::Succeeded => "succeeded",
72+
Self::Failed => "failed",
73+
Self::TimedOut => "timed_out",
74+
Self::Cancelled => "cancelled",
75+
}
76+
}
77+
}
78+
79+
/// Valid role/step pairs with an explicit start counter.
80+
pub const CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 4] = [
81+
(DbPoolRole::Writer, DbConnectionStep::WriterPool),
82+
(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor),
83+
(DbPoolRole::Writer, DbConnectionStep::SessionTimeouts),
84+
(DbPoolRole::Writer, DbConnectionStep::Isolation),
85+
];
86+
87+
/// Valid role/step pairs with a duration histogram.
88+
pub const CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 4] = CONNECTION_STARTED_STEPS;
89+
90+
/// Valid role/step/outcome terminal combinations.
91+
pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 15] = [
92+
(
93+
DbPoolRole::Writer,
94+
DbConnectionStep::WriterPool,
95+
DbConnectionOutcome::Succeeded,
96+
),
97+
(
98+
DbPoolRole::Writer,
99+
DbConnectionStep::WriterPool,
100+
DbConnectionOutcome::Failed,
101+
),
102+
(
103+
DbPoolRole::Writer,
104+
DbConnectionStep::WriterPool,
105+
DbConnectionOutcome::TimedOut,
106+
),
107+
(
108+
DbPoolRole::Writer,
109+
DbConnectionStep::WriterPool,
110+
DbConnectionOutcome::Cancelled,
111+
),
112+
(
113+
DbPoolRole::Writer,
114+
DbConnectionStep::PhysicalConnect,
115+
DbConnectionOutcome::Succeeded,
116+
),
117+
(
118+
DbPoolRole::Writer,
119+
DbConnectionStep::CreatedAtFloor,
120+
DbConnectionOutcome::Succeeded,
121+
),
122+
(
123+
DbPoolRole::Writer,
124+
DbConnectionStep::CreatedAtFloor,
125+
DbConnectionOutcome::Failed,
126+
),
127+
(
128+
DbPoolRole::Writer,
129+
DbConnectionStep::CreatedAtFloor,
130+
DbConnectionOutcome::Cancelled,
131+
),
132+
(
133+
DbPoolRole::Writer,
134+
DbConnectionStep::SessionTimeouts,
135+
DbConnectionOutcome::Succeeded,
136+
),
137+
(
138+
DbPoolRole::Writer,
139+
DbConnectionStep::SessionTimeouts,
140+
DbConnectionOutcome::Failed,
141+
),
142+
(
143+
DbPoolRole::Writer,
144+
DbConnectionStep::SessionTimeouts,
145+
DbConnectionOutcome::Cancelled,
146+
),
147+
(
148+
DbPoolRole::Writer,
149+
DbConnectionStep::Isolation,
150+
DbConnectionOutcome::Succeeded,
151+
),
152+
(
153+
DbPoolRole::Writer,
154+
DbConnectionStep::Isolation,
155+
DbConnectionOutcome::Failed,
156+
),
157+
(
158+
DbPoolRole::Writer,
159+
DbConnectionStep::Isolation,
160+
DbConnectionOutcome::Cancelled,
161+
),
162+
(
163+
DbPoolRole::Writer,
164+
DbConnectionStep::Ready,
165+
DbConnectionOutcome::Succeeded,
166+
),
167+
];
168+
169+
/// Four start counters + four 13-series histograms + fifteen terminal counters.
170+
pub const CONNECTION_RAW_SERIES_PER_POD: usize = 4 + (4 * 13) + 15;
171+
172+
pub(crate) struct DbConnectionStepAttempt {
173+
pool_role: DbPoolRole,
174+
step: DbConnectionStep,
175+
started: Instant,
176+
finished: bool,
177+
}
178+
179+
impl DbConnectionStepAttempt {
180+
pub(crate) fn start(pool_role: DbPoolRole, step: DbConnectionStep) -> Self {
181+
metrics::counter!(
182+
"buzz_db_connection_step_started_total",
183+
"pool_role" => pool_role.as_str(),
184+
"step" => step.as_str(),
185+
)
186+
.increment(1);
187+
Self {
188+
pool_role,
189+
step,
190+
started: Instant::now(),
191+
finished: false,
192+
}
193+
}
194+
195+
pub(crate) fn succeed(self) {
196+
self.finish(DbConnectionOutcome::Succeeded);
197+
}
198+
199+
pub(crate) fn fail(self) {
200+
self.finish(DbConnectionOutcome::Failed);
201+
}
202+
203+
pub(crate) fn time_out(self) {
204+
self.finish(DbConnectionOutcome::TimedOut);
205+
}
206+
207+
fn finish(mut self, outcome: DbConnectionOutcome) {
208+
record_terminal(
209+
self.pool_role,
210+
self.step,
211+
outcome,
212+
Some(self.started.elapsed()),
213+
);
214+
self.finished = true;
215+
}
216+
}
217+
218+
impl Drop for DbConnectionStepAttempt {
219+
fn drop(&mut self) {
220+
if self.finished {
221+
return;
222+
}
223+
let outcome = if std::thread::panicking() {
224+
DbConnectionOutcome::Failed
225+
} else {
226+
DbConnectionOutcome::Cancelled
227+
};
228+
record_terminal(
229+
self.pool_role,
230+
self.step,
231+
outcome,
232+
Some(self.started.elapsed()),
233+
);
234+
self.finished = true;
235+
}
236+
}
237+
238+
pub(crate) fn record_milestone(pool_role: DbPoolRole, step: DbConnectionStep) {
239+
record_terminal(pool_role, step, DbConnectionOutcome::Succeeded, None);
240+
}
241+
242+
fn record_terminal(
243+
pool_role: DbPoolRole,
244+
step: DbConnectionStep,
245+
outcome: DbConnectionOutcome,
246+
elapsed: Option<Duration>,
247+
) {
248+
metrics::counter!(
249+
"buzz_db_connection_step_attempts_total",
250+
"pool_role" => pool_role.as_str(),
251+
"step" => step.as_str(),
252+
"outcome" => outcome.as_str(),
253+
)
254+
.increment(1);
255+
if let Some(elapsed) = elapsed {
256+
metrics::histogram!(
257+
"buzz_db_connection_step_duration_seconds",
258+
"pool_role" => pool_role.as_str(),
259+
"step" => step.as_str(),
260+
)
261+
.record(elapsed.as_secs_f64());
262+
}
263+
}
264+
265+
pub(crate) fn classify_pool_outcome(error: &sqlx::Error) -> DbConnectionOutcome {
266+
if matches!(error, sqlx::Error::PoolTimedOut) {
267+
DbConnectionOutcome::TimedOut
268+
} else {
269+
DbConnectionOutcome::Failed
270+
}
271+
}
272+
273+
#[cfg(test)]
274+
mod tests {
275+
use super::*;
276+
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
277+
278+
#[test]
279+
fn vocabulary_and_series_budget_are_frozen() {
280+
assert_eq!(
281+
DbConnectionStep::ALL.map(DbConnectionStep::as_str),
282+
[
283+
"writer_pool",
284+
"physical_connect",
285+
"created_at_floor",
286+
"session_timeouts",
287+
"isolation",
288+
"ready",
289+
]
290+
);
291+
assert_eq!(CONNECTION_RAW_SERIES_PER_POD, 71);
292+
}
293+
294+
#[test]
295+
fn dropped_step_is_cancelled_exactly_once_without_sensitive_labels() {
296+
let recorder = DebuggingRecorder::new();
297+
let snapshotter = recorder.snapshotter();
298+
let _guard = metrics::set_default_local_recorder(&recorder);
299+
let attempt =
300+
DbConnectionStepAttempt::start(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor);
301+
drop(attempt);
302+
303+
let metrics = snapshotter.snapshot().into_vec();
304+
assert_eq!(metrics.len(), 3);
305+
for (key, _, _, value) in metrics {
306+
let labels = key
307+
.key()
308+
.labels()
309+
.map(|label| (label.key(), label.value()))
310+
.collect::<std::collections::BTreeMap<_, _>>();
311+
assert_eq!(labels.get("pool_role"), Some(&"writer"));
312+
assert_eq!(labels.get("step"), Some(&"created_at_floor"));
313+
assert!(!labels.contains_key("reason"));
314+
assert!(!labels.contains_key("connection_ordinal"));
315+
match value {
316+
DebugValue::Counter(value) => {
317+
assert_eq!(value, 1);
318+
if key.key().name() == "buzz_db_connection_step_attempts_total" {
319+
assert_eq!(labels.get("outcome"), Some(&"cancelled"));
320+
}
321+
}
322+
DebugValue::Histogram(values) => assert_eq!(values.len(), 1),
323+
DebugValue::Gauge(_) => panic!("connection setup has no gauges"),
324+
}
325+
}
326+
}
327+
328+
#[test]
329+
fn pool_timeout_is_distinguished_from_other_failures() {
330+
assert_eq!(
331+
classify_pool_outcome(&sqlx::Error::PoolTimedOut),
332+
DbConnectionOutcome::TimedOut
333+
);
334+
assert_eq!(
335+
classify_pool_outcome(&sqlx::Error::PoolClosed),
336+
DbConnectionOutcome::Failed
337+
);
338+
let io = sqlx::Error::Io(std::io::Error::new(
339+
std::io::ErrorKind::ConnectionRefused,
340+
"postgres://secret-user:[email protected]/private",
341+
));
342+
assert_eq!(classify_pool_outcome(&io), DbConnectionOutcome::Failed);
343+
}
344+
}

0 commit comments

Comments
 (0)