Skip to content

Commit 113a33b

Browse files
Add database pressure observability (#6700)
## Why Database pressure currently collapses several distinct delays into one symptom. This adds the evidence layer needed to distinguish pool acquisition wait, logical database operation time, advisory-lock wait, and selected transaction duration before changing timeout or retry policy. This is the phase 2 Lane A observability bundle for [#26](TheSentinel454#26), [#28](TheSentinel454#28), and [#33](TheSentinel454#33). It is stacked on #6668. ## What - Record explicit reader/writer checkout wait and acquisition outcomes with `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total`. - Extend the compile-time `#[datastore_span(name = "...")]` seam with `buzz_db_operation_duration_seconds`, so operation labels remain static source literals instead of request data. - Route correctness-critical replacement, membership, push-gate, deletion, and migration/schema-safety advisory locks through one observer without changing their SQL, order, scope, or blocking behavior. - Measure six internally owned transaction lifetimes with `buzz_db_transaction_duration_seconds`, starting after `BEGIN` succeeds and ending after explicit commit/rollback or scope exit. - Emit root slow-operation warnings at 500 ms, logging the first slow completion and then 1/100 per call site with only `operation`, `outcome`, and `elapsed_ms`. - Document names, units, fixed label vocabularies, measurement boundaries, and blind spots in this PR description. Fixed labels are deliberately small: - `pool_role`: `writer`, `reader` - `lock_type`: `replacement`, `membership`, `push_gate`, `deletion`, `migration_schema_safety` - `outcome`: `success`, `error`, `timeout` where SQLx/PostgreSQL can distinguish it accurately - `operation`: compile-time datastore names plus the six closed transaction operation names documented in the runbook No metric or slow warning contains community IDs, event IDs, event kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors, or event content. ## Coverage boundaries - Operation duration is the complete annotated logical function body, not pure SQL execution; it may include implicit checkout, lock wait, nested operations, and application work. Cancelled futures do not reach its completion hook. - Pool timing covers explicit helper checkouts, including proved-reader routing and selected writer-owned transactions. Implicit SQLx checkout through `&PgPool` remains folded into operation duration. - Lock timing covers application-side blocking locks in the five named families. Trigger/stored-procedure locks, channel-TTL locking, the usage try-lock, and the audit service session lock remain outside this slice. - Transaction timing covers only the six wholly owned boundaries documented in the runbook. It excludes pool wait, `BEGIN`, asynchronous rollback cleanup after an early return, and caller-owned `Db::begin_transaction` lifetime. ## Relationship to #6229 #6229 is the incident-driven timeout precursor. This PR does not add or change `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout`, retries, audit durability, or client-visible conflicts. It provides the missing distributions needed to evaluate those policies later and intentionally leaves #6229's open audit retry/durability finding untouched. The branches overlap in `crates/buzz-db/src/lib.rs` and `crates/buzz-db/src/migration.rs`, so a later rebase may need textual conflict resolution, but the behavior is complementary rather than duplicated. ## Risk assessment Moderate-low. The primary risk is instrumentation overhead and added static series. Cardinality is source-bounded, slow logs are sampled/redacted root events, and the lock/transaction changes wrap existing awaits without changing policy or ordering. ## Verification Author workstation: `buzz-tornquist-db-pressure-observability` (`2010927`), exact head `d7cf833e26c528adfcde3917ded80daf6f4ddac9`, parent `6f50e6b2b2a996349149af61d35bdd6a355f77fd`. - `cargo fmt --all --check` — passed - `cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p buzz-search -p buzz-relay --all-targets -- -D warnings` — passed - `cargo test -p buzz-datastore-tracing --quiet` — 4 passed - `cargo test -p buzz-db --quiet` — 109 passed, 200 ignored - `cargo test -p buzz-audit -p buzz-search --quiet` — 16 passed, 25 ignored - `cargo test -p buzz-relay --lib --quiet -- --test-threads=1` — 906 passed, 48 ignored - Native PostgreSQL focused tests for pool success/timeout/error, lock success/contention/timeout/error, replacement, membership serialization, push ordering, deletion fencing, migration/schema exclusion, and reader fallback — 8 passed The default-parallel relay run passed once; subsequent runs exposed the existing load-sensitive `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` 504 at the end of the suite. That test passes in isolation and the full relay suite passes serially. Independent exact-head review workstation: `buzz-tornquist-db-pressure-observability-review` (`2013067`). Formatting, the same all-target clippy command, datastore instrumentation tests, DB unit tests, source privacy guards, and diff/non-goal audits passed; no review findings. Generated with Codex --------- Signed-off-by: tornquist <[email protected]>
1 parent 12f3fea commit 113a33b

17 files changed

Lines changed: 1086 additions & 123 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -692,6 +692,18 @@ jobs:
692692
env:
693693
DATABASE_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/buzz
694694
TEST_DATABASE_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/buzz
695+
- name: Database pressure observability PostgreSQL tests
696+
# Explicit pool acquisition and advisory-lock metrics require real
697+
# Postgres and are ignored by the infrastructure-free unit-test job.
698+
run: |
699+
filter='package(buzz-db) and test(/observability::tests::(pool_acquire_records_success_timeout_and_error_with_wait_time|advisory_lock_records_success_contention_timeout_and_error)/)'
700+
cargo nextest run \
701+
--archive-file target/ci/backend-integration-tests.tar.zst \
702+
-E "${filter}" \
703+
--run-ignored ignored-only
704+
env:
705+
DATABASE_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/buzz
706+
TEST_DATABASE_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/buzz
695707
- name: Start relay
696708
run: |
697709
chmod +x ./target/ci/buzz-relay

‎Cargo.lock‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/buzz-audit/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ serde_json = { workspace = true }
1717
uuid = { workspace = true }
1818
chrono = { workspace = true }
1919
tracing = { workspace = true }
20+
metrics = { workspace = true }
2021
thiserror = { workspace = true }
2122
sha2 = { workspace = true }
2223
hex = { workspace = true }

‎crates/buzz-datastore-tracing/Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ quote = "1"
1616
syn = { version = "2", features = ["full"] }
1717

1818
[dev-dependencies]
19+
metrics = { workspace = true }
20+
metrics-util = { workspace = true }
1921
opentelemetry = { workspace = true }
2022
opentelemetry_sdk = { workspace = true, features = ["testing"] }
2123
tokio = { workspace = true }

‎crates/buzz-datastore-tracing/src/lib.rs‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,9 @@ impl Parse for DatastoreArgs {
7070
/// PostgreSQL spans always omit function arguments, use the `buzz_datastore`
7171
/// target, and expose only canonical semantic fields plus explicitly supplied
7272
/// safe fields. An `Err` sets `otel.status_code` without inspecting the error.
73+
/// The literal `name` also labels a logical-operation duration histogram. Slow
74+
/// completions are sampled and logged with only that name, outcome, and elapsed
75+
/// time; arguments, error values, and return values are never formatted.
7376
#[proc_macro_attribute]
7477
pub fn datastore_span(args: TokenStream, item: TokenStream) -> TokenStream {
7578
let args = parse_macro_input!(args as DatastoreArgs);
@@ -129,9 +132,46 @@ pub fn datastore_span(args: TokenStream, item: TokenStream) -> TokenStream {
129132
}
130133
}
131134
});
135+
let outcome = if returns_result {
136+
quote! {
137+
if #result.is_err() { "error" } else { "success" }
138+
}
139+
} else {
140+
quote!("success")
141+
};
132142
function.block = Box::new(syn::parse_quote!({
143+
let __buzz_datastore_started_7f3a9c = ::std::time::Instant::now();
133144
let #result: #return_type = (async #original_body).await;
134145
#record_error
146+
let __buzz_datastore_outcome_7f3a9c = #outcome;
147+
let __buzz_datastore_elapsed_7f3a9c = __buzz_datastore_started_7f3a9c.elapsed();
148+
::metrics::histogram!(
149+
"buzz_db_operation_duration_seconds",
150+
"operation" => #name,
151+
"outcome" => __buzz_datastore_outcome_7f3a9c,
152+
)
153+
.record(__buzz_datastore_elapsed_7f3a9c.as_secs_f64());
154+
if __buzz_datastore_elapsed_7f3a9c >= ::std::time::Duration::from_millis(500) {
155+
static __BUZZ_DATASTORE_SLOW_SAMPLE_7F3A9C:
156+
::std::sync::atomic::AtomicU64 = ::std::sync::atomic::AtomicU64::new(0);
157+
if __BUZZ_DATASTORE_SLOW_SAMPLE_7F3A9C.fetch_add(
158+
1,
159+
::std::sync::atomic::Ordering::Relaxed,
160+
) % 100 == 0 {
161+
let __buzz_datastore_elapsed_ms_7f3a9c =
162+
__buzz_datastore_elapsed_7f3a9c
163+
.as_millis()
164+
.min(::std::primitive::u64::MAX as u128) as u64;
165+
::tracing::warn!(
166+
target: "buzz_datastore",
167+
parent: None,
168+
operation = #name,
169+
outcome = __buzz_datastore_outcome_7f3a9c,
170+
elapsed_ms = __buzz_datastore_elapsed_ms_7f3a9c,
171+
"slow datastore operation"
172+
);
173+
}
174+
}
135175
#result
136176
}));
137177

‎crates/buzz-datastore-tracing/tests/runtime.rs‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
11
use buzz_datastore_tracing::datastore_span;
2+
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
23
use opentelemetry::trace::{SpanKind, Status, TracerProvider as _};
34
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider};
5+
use std::collections::BTreeMap;
6+
use std::sync::{Arc, Mutex};
7+
use tracing::field::{Field, Visit};
8+
use tracing::{Event, Subscriber};
9+
use tracing_subscriber::layer::{Context, Layer};
410
use tracing_subscriber::prelude::*;
511

612
const DIRECT_ERROR: &str = "raw-secret-direct-error";
@@ -27,8 +33,48 @@ async fn operation(
2733
Ok(limit)
2834
}
2935

36+
#[datastore_span(name = "slow_test_operation", system = "postgresql")]
37+
async fn slow_operation(delay: std::time::Duration) -> Result<(), &'static str> {
38+
tokio::time::sleep(delay).await;
39+
Err(DIRECT_ERROR)
40+
}
41+
42+
#[derive(Default)]
43+
struct EventFields(BTreeMap<String, String>);
44+
45+
impl Visit for EventFields {
46+
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
47+
self.0.insert(field.name().to_owned(), format!("{value:?}"));
48+
}
49+
50+
fn record_str(&mut self, field: &Field, value: &str) {
51+
self.0.insert(field.name().to_owned(), value.to_owned());
52+
}
53+
54+
fn record_u64(&mut self, field: &Field, value: u64) {
55+
self.0.insert(field.name().to_owned(), value.to_string());
56+
}
57+
}
58+
59+
#[derive(Clone, Default)]
60+
struct EventCapture(Arc<Mutex<Vec<EventFields>>>);
61+
62+
impl<S> Layer<S> for EventCapture
63+
where
64+
S: Subscriber,
65+
{
66+
fn on_event(&self, event: &Event<'_>, _context: Context<'_, S>) {
67+
let mut fields = EventFields::default();
68+
event.record(&mut fields);
69+
self.0.lock().expect("capture lock").push(fields);
70+
}
71+
}
72+
3073
#[tokio::test(flavor = "current_thread")]
3174
async fn exports_policy_fields_without_error_or_argument_data() {
75+
let recorder = DebuggingRecorder::new();
76+
let snapshotter = recorder.snapshotter();
77+
let _metrics_guard = metrics::set_default_local_recorder(&recorder);
3278
let exporter = InMemorySpanExporter::default();
3379
let provider = SdkTracerProvider::builder()
3480
.with_simple_exporter(exporter.clone())
@@ -41,6 +87,37 @@ async fn exports_policy_fields_without_error_or_argument_data() {
4187
assert_eq!(operation(8, true, false).await, Err(DIRECT_ERROR));
4288
assert_eq!(operation(9, false, true).await, Err(QUESTION_ERROR));
4389

90+
let operation_samples = snapshotter
91+
.snapshot()
92+
.into_vec()
93+
.into_iter()
94+
.filter(|(key, ..)| key.key().name() == "buzz_db_operation_duration_seconds")
95+
.map(|(key, _, _, value)| {
96+
let DebugValue::Histogram(samples) = value else {
97+
panic!("operation duration must be a histogram");
98+
};
99+
let labels = key
100+
.key()
101+
.labels()
102+
.map(|label| (label.key().to_owned(), label.value().to_owned()))
103+
.collect::<std::collections::BTreeMap<_, _>>();
104+
(labels, samples)
105+
})
106+
.collect::<Vec<_>>();
107+
assert_eq!(operation_samples.len(), 2);
108+
for (labels, samples) in operation_samples {
109+
assert_eq!(
110+
labels.get("operation").map(String::as_str),
111+
Some("test_operation")
112+
);
113+
assert!(matches!(
114+
labels.get("outcome").map(String::as_str),
115+
Some("success" | "error")
116+
));
117+
assert!(!samples.is_empty());
118+
assert!(samples.iter().all(|sample| sample.into_inner() >= 0.0));
119+
}
120+
44121
provider.force_flush().expect("spans flush");
45122
let spans = exporter.get_finished_spans().expect("exported spans");
46123
assert_eq!(spans.len(), 3);
@@ -78,3 +155,55 @@ async fn exports_policy_fields_without_error_or_argument_data() {
78155
}
79156
}
80157
}
158+
159+
#[tokio::test(flavor = "current_thread")]
160+
async fn slow_operation_logging_is_guarded_sampled_and_redacted() {
161+
let capture = EventCapture::default();
162+
let subscriber = tracing_subscriber::registry().with(capture.clone());
163+
let _subscriber_guard = tracing::subscriber::set_default(subscriber);
164+
165+
assert_eq!(
166+
slow_operation(std::time::Duration::from_millis(1)).await,
167+
Err(DIRECT_ERROR)
168+
);
169+
assert_eq!(
170+
slow_operation(std::time::Duration::from_millis(510)).await,
171+
Err(DIRECT_ERROR)
172+
);
173+
assert_eq!(
174+
slow_operation(std::time::Duration::from_millis(510)).await,
175+
Err(DIRECT_ERROR)
176+
);
177+
178+
let events = capture.0.lock().expect("capture lock");
179+
let slow = events
180+
.iter()
181+
.filter(|event| {
182+
event
183+
.0
184+
.get("message")
185+
.is_some_and(|message| message.contains("slow datastore operation"))
186+
})
187+
.collect::<Vec<_>>();
188+
assert_eq!(
189+
slow.len(),
190+
1,
191+
"first slow call is logged, next 99 are sampled out"
192+
);
193+
let fields = &slow[0].0;
194+
assert_eq!(
195+
fields.get("operation").map(String::as_str),
196+
Some("slow_test_operation")
197+
);
198+
assert_eq!(fields.get("outcome").map(String::as_str), Some("error"));
199+
assert!(fields
200+
.get("elapsed_ms")
201+
.and_then(|value| value.parse::<u64>().ok())
202+
.is_some_and(|elapsed| elapsed >= 500));
203+
assert_eq!(
204+
fields.len(),
205+
4,
206+
"only message and fixed safe fields are logged"
207+
);
208+
assert!(!format!("{fields:?}").contains(DIRECT_ERROR));
209+
}

‎crates/buzz-db/src/channel.rs‎

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -478,14 +478,17 @@ async fn acquire_channel_membership_lock(
478478
community_id: CommunityId,
479479
channel_id: Uuid,
480480
) -> Result<()> {
481-
sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))")
482-
.bind(format!(
483-
"{CHANNEL_MEMBERSHIP_LOCK_NAMESPACE}{}:{}",
484-
community_id.as_uuid(),
485-
channel_id
486-
))
487-
.execute(&mut **tx)
488-
.await?;
481+
crate::observability::observe_advisory_lock(
482+
crate::observability::LockType::Membership,
483+
sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))")
484+
.bind(format!(
485+
"{CHANNEL_MEMBERSHIP_LOCK_NAMESPACE}{}:{}",
486+
community_id.as_uuid(),
487+
channel_id
488+
))
489+
.execute(&mut **tx),
490+
)
491+
.await?;
489492
Ok(())
490493
}
491494

@@ -631,10 +634,13 @@ pub async fn lock_member_snapshot(
631634
relay_pubkey,
632635
Some(channel_id.as_bytes()),
633636
);
634-
sqlx::query("SELECT pg_advisory_xact_lock($1)")
635-
.bind(replacement_lock)
636-
.execute(&mut *tx)
637-
.await?;
637+
crate::observability::observe_advisory_lock(
638+
crate::observability::LockType::Replacement,
639+
sqlx::query("SELECT pg_advisory_xact_lock($1)")
640+
.bind(replacement_lock)
641+
.execute(&mut *tx),
642+
)
643+
.await?;
638644
acquire_channel_membership_lock(&mut tx, community_id, channel_id).await?;
639645
let rows = sqlx::query(
640646
r#"

‎crates/buzz-db/src/community.rs‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -325,10 +325,13 @@ impl Db {
325325

326326
// Serialize on the owner pubkey so concurrent creates to the same
327327
// owner cannot both pass the ownership count check.
328-
sqlx::query("SELECT pg_advisory_xact_lock($1)")
329-
.bind(relay_members::owner_count_advisory_lock_key(&owner_pubkey))
330-
.execute(&mut *tx)
331-
.await?;
328+
crate::observability::observe_advisory_lock(
329+
crate::observability::LockType::Membership,
330+
sqlx::query("SELECT pg_advisory_xact_lock($1)")
331+
.bind(relay_members::owner_count_advisory_lock_key(&owner_pubkey))
332+
.execute(&mut *tx),
333+
)
334+
.await?;
332335

333336
let row = sqlx::query(
334337
r#"

0 commit comments

Comments
 (0)