Skip to content

Commit 359839f

Browse files
AresonIan Oberstclaude
authored
feat(db): label advisory-lock metrics with the locked event kind (#9472)
## Why On one deployment, `buzz_db_advisory_lock_wait_seconds{lock_type="replacement"}` added up to about 245,000 seconds of wait in 7 days. The metric cannot say which coordinates wait. Replacement locks serialize writers of one `(kind, pubkey[, d_tag | channel])` coordinate. Relay-owned lists (kinds 13535, 39000–39002) go through the same generic replacement paths as user replaceables (kinds 0, 3, 10002, 30078, …). The 13534 membership list takes its coordinate lock under `lock_type="membership"`. Before we change how these lists are stored, we need to know which kinds the wait comes from. ## What - `buzz_db_advisory_lock_wait_seconds` and `buzz_db_advisory_lock_acquisitions_total` get a `kind` label. - Locks keyed by an event coordinate record that coordinate's kind, through the new `observe_event_lock(lock_type, kind, …)`. This covers all 4 replacement call sites (parameterized replace, `take_replacement_lock` for addressable replace and group-state snapshots, the kind:39002 member snapshot, and workflow deletion) and the kind:13534 membership-list lock. The 13534 lock keeps `lock_type="membership"`, so totals by `lock_type` do not change. - All other locks record `kind="none"`, and their call sites do not change. `observe_advisory_lock` has a `debug_assert` that rejects `LockType::Replacement`, so a replacement lock cannot lose its kind by mistake. - `buzz_core::kind::registered_kind_label(kind)` gives the label value: the decimal kind when it is in `ALL_KINDS`, otherwise `other`. ## Change to the #6700 label contract #6700 says no database metric carries event kinds, and `observability_source.rs` enforces this with a blanket ban on `"kind" =>`. This PR makes one narrow exception: the two advisory-lock metrics. - **Bounded:** the value set is the kind registry, not request data. Of the 135 registered kinds, 39 are replaceable or addressable and can reach a replacement lock. Any other kind a client sends becomes `other`. Non-replacement locks are always `none`. New series in the worst case: 39 kinds × 4 outcomes, for the histogram and the counter. - **Not sensitive:** no community, pubkey, coordinate, or d-tag is attached. The kind of a replacement lock does not expose any of those. - **The guard is narrowed, not removed:** `"kind" =>` must appear exactly twice, inside the advisory-lock observer `observe_lock`, whose kind must come from `registered_kind_label` inside `observe_event_lock`. It is still forbidden everywhere else in `observability.rs` and in the datastore tracing macro. The #7195 guard that forbids a literal `"other"` in `observability.rs` is unchanged, because the fallback lives in `buzz-core`. ## Dashboards and alerts Both metrics gain a label, so existing selectors keep working: `lock_type="replacement"` still matches every replacement series, because Prometheus and Datadog selectors match on a subset of labels. A consumer is affected only if it depends on the exact label set: - Recording rules or panels that read a series without aggregating it now get one series per kind. Aggregate with `sum by (lock_type, outcome) (...)` to keep the old shape. - Vector matching with `on(lock_type, outcome)` or `ignoring(...)` now has more series on one side. Aggregate before the join. To read the new attribution, group by `kind`, for example `sum by (lock_type, kind) (rate(buzz_db_advisory_lock_wait_seconds_sum[5m]))`. No consumer in this repository depends on the exact label set. ## Tests - `buzz-core`: `registered_kind_label_bounds_unregistered_kinds` checks that a registered kind passes through and that an unregistered or negative kind becomes `other`. - `buzz-db` recorder test: checks the replacement series for a registered kind (`13534`) and an unregistered kind (`other`), and the `kind="none"` series for a membership lock, on both metrics. - `kindless_observer_rejects_replacement_locks`: the `debug_assert` guard panics. - `membership_list_lock_records_the_roster_kind` (Postgres): a 13534 list write records exactly one `membership` acquisition with `kind="13534"`. - `group_state_replacement_takes_its_lock_once` (Postgres): now counts acquisitions only under the written kind. It expects `kind="39000"` for the group-state snapshot path and `kind="0"` for `replace_addressable_event`, so if `take_replacement_lock` drops or swaps the kind, the count becomes 0 and the test fails. - `observability_source.rs`: the narrowed guard described above. Commands run: - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo test -p buzz-core --lib` - `cargo test -p buzz-db --lib` - `cargo test -p buzz-db --test observability_source` - `scripts/postgres-test-run.sh -p buzz-db --lib --tests` (the CI PostgreSQL lane, run locally) ## Local relay check The relay ran from this branch against an isolated database. `/metrics` showed one series per locked kind after each write: - `users set-profile`: `replacement`, `kind="0"` - `users set-status`: `replacement`, `kind="30315"` - `channels create`: `replacement`, `kind="39000"`, `"39001"`, `"39002"` - `channels add-member`: `replacement`, `kind="39002"` - `agents archive`: `replacement`, `kind="13535"` - `moderation ban`: `replacement`, `kind="30622"`, plus `membership`, `kind="none"` - Invite mint and claim (`e2e_relay::test_invite_mint_and_claim_admits_new_pubkey`): `membership`, `kind="13534"` - Background loops: `deletion` and `partition_maintenance`, `kind="none"` `buzz_db_advisory_lock_wait_seconds_count` matched `buzz_db_advisory_lock_acquisitions_total` for every series. The author also tested it against the local relay. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Ian Oberst <[email protected]> Co-authored-by: Ian Oberst <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 59ae028 commit 359839f

8 files changed

Lines changed: 237 additions & 26 deletions

File tree

‎crates/buzz-core/src/kind.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -843,6 +843,18 @@ pub const ALL_KINDS: &[u32] = &[
843843
KIND_PROJECT,
844844
];
845845

846+
/// Metric label for an event kind: the decimal kind when it is registered in
847+
/// [`ALL_KINDS`], otherwise `"other"`.
848+
///
849+
/// The registry bounds the label set, so the label is safe on metrics whose
850+
/// kind comes from client events.
851+
pub fn registered_kind_label(kind: i32) -> String {
852+
match u32::try_from(kind) {
853+
Ok(kind) if ALL_KINDS.contains(&kind) => kind.to_string(),
854+
_ => "other".to_owned(),
855+
}
856+
}
857+
846858
/// Returns `true` if `kind` is in the ephemeral range (20000–29999).
847859
pub const fn is_ephemeral(kind: u32) -> bool {
848860
kind >= EPHEMERAL_KIND_MIN && kind <= EPHEMERAL_KIND_MAX
@@ -994,6 +1006,18 @@ mod tests {
9941006
}
9951007
}
9961008

1009+
#[test]
1010+
fn registered_kind_label_bounds_unregistered_kinds() {
1011+
assert_eq!(
1012+
registered_kind_label(KIND_NIP43_MEMBERSHIP_LIST as i32),
1013+
"13534"
1014+
);
1015+
assert_eq!(registered_kind_label(KIND_PROFILE as i32), "0");
1016+
assert!(!ALL_KINDS.contains(&30999));
1017+
assert_eq!(registered_kind_label(30999), "other");
1018+
assert_eq!(registered_kind_label(-1), "other");
1019+
}
1020+
9971021
#[test]
9981022
fn nip43_membership_snapshot_is_relay_only() {
9991023
assert!(is_relay_only_kind(KIND_NIP43_MEMBERSHIP_LIST));

‎crates/buzz-db/src/runtime/observability.rs‎

Lines changed: 111 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
//!
33
//! Label values come only from the closed enums in this module. Callers must
44
//! never derive labels from tenant data, events, SQL text, or query identifiers.
5+
//! The one exception is the advisory-lock `kind` label: the event kind of the
6+
//! locked coordinate, bounded by the kind registry
7+
//! (`buzz_core::kind::registered_kind_label`).
58
69
use std::future::Future;
710
use std::sync::Mutex;
@@ -539,7 +542,47 @@ pub(crate) async fn begin_transaction(
539542
Ok((transaction, TransactionTimer::start(operation)))
540543
}
541544

545+
/// `kind` label of an advisory lock that no event kind keys.
546+
const UNKEYED_LOCK_KIND: &str = "none";
547+
548+
/// Observe an advisory-lock wait whose key has no event kind; it records
549+
/// `kind="none"`. Locks keyed by an event coordinate, including every
550+
/// replacement lock, use [`observe_event_lock`].
542551
pub(crate) async fn observe_advisory_lock<T, F>(lock_type: LockType, future: F) -> sqlx::Result<T>
552+
where
553+
F: Future<Output = sqlx::Result<T>>,
554+
{
555+
debug_assert_ne!(
556+
lock_type,
557+
LockType::Replacement,
558+
"replacement locks carry an event kind; use observe_event_lock"
559+
);
560+
observe_lock(lock_type, UNKEYED_LOCK_KIND.into(), future).await
561+
}
562+
563+
/// Observe an advisory-lock wait on a coordinate of event `kind`. The `kind`
564+
/// label is the registered kind, or `other` for an unregistered one.
565+
pub(crate) async fn observe_event_lock<T, F>(
566+
lock_type: LockType,
567+
kind: i32,
568+
future: F,
569+
) -> sqlx::Result<T>
570+
where
571+
F: Future<Output = sqlx::Result<T>>,
572+
{
573+
observe_lock(
574+
lock_type,
575+
buzz_core::kind::registered_kind_label(kind).into(),
576+
future,
577+
)
578+
.await
579+
}
580+
581+
async fn observe_lock<T, F>(
582+
lock_type: LockType,
583+
kind: metrics::SharedString,
584+
future: F,
585+
) -> sqlx::Result<T>
543586
where
544587
F: Future<Output = sqlx::Result<T>>,
545588
{
@@ -552,12 +595,14 @@ where
552595
metrics::histogram!(
553596
"buzz_db_advisory_lock_wait_seconds",
554597
"lock_type" => lock_type.as_str(),
598+
"kind" => kind.clone(),
555599
"outcome" => outcome.as_str(),
556600
)
557601
.record(started.elapsed().as_secs_f64());
558602
metrics::counter!(
559603
"buzz_db_advisory_lock_acquisitions_total",
560604
"lock_type" => lock_type.as_str(),
605+
"kind" => kind,
561606
"outcome" => outcome.as_str(),
562607
)
563608
.increment(1);
@@ -606,9 +651,9 @@ impl Drop for TransactionTimer {
606651
mod tests {
607652
use super::{
608653
acquire_reader_with_legacy_metrics, acquire_writer, acquire_writer_with_legacy_metrics,
609-
begin_transaction, observe_advisory_lock, record_pool_acquire, refresh_pool_waiters,
610-
LockType, Outcome, PoolAcquireAttempt, PoolOperation, ReaderOperation,
611-
TransactionOperation, TransactionTimer, WriterOperation,
654+
begin_transaction, observe_advisory_lock, observe_event_lock, record_pool_acquire,
655+
refresh_pool_waiters, LockType, Outcome, PoolAcquireAttempt, PoolOperation,
656+
ReaderOperation, TransactionOperation, TransactionTimer, WriterOperation,
612657
};
613658
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
614659
use std::collections::{BTreeMap, BTreeSet};
@@ -736,9 +781,18 @@ mod tests {
736781
Duration::from_millis(34),
737782
true,
738783
);
739-
let lock_ok: sqlx::Result<()> =
740-
observe_advisory_lock(LockType::Replacement, async { Ok(()) }).await;
784+
let lock_ok: sqlx::Result<()> = observe_event_lock(
785+
LockType::Replacement,
786+
buzz_core::kind::KIND_NIP43_MEMBERSHIP_LIST as i32,
787+
async { Ok(()) },
788+
)
789+
.await;
741790
assert!(lock_ok.is_ok());
791+
let unregistered_ok: sqlx::Result<()> =
792+
observe_event_lock(LockType::Replacement, 30999, async { Ok(()) }).await;
793+
assert!(unregistered_ok.is_ok());
794+
// The fallback label lives in buzz-core; this file never names it.
795+
let unregistered_kind = buzz_core::kind::registered_kind_label(30999);
742796
let lock_error: sqlx::Result<()> =
743797
observe_advisory_lock(LockType::Membership, async { Err(sqlx::Error::PoolClosed) })
744798
.await;
@@ -788,39 +842,71 @@ mod tests {
788842
for expected in [
789843
(
790844
"buzz_db_advisory_lock_wait_seconds",
791-
[("lock_type", "replacement"), ("outcome", "success")],
845+
&[
846+
("lock_type", "replacement"),
847+
("kind", "13534"),
848+
("outcome", "success"),
849+
][..],
792850
),
793851
(
794852
"buzz_db_advisory_lock_wait_seconds",
795-
[("lock_type", "membership"), ("outcome", "error")],
853+
&[
854+
("lock_type", "replacement"),
855+
("kind", unregistered_kind.as_str()),
856+
("outcome", "success"),
857+
],
858+
),
859+
(
860+
"buzz_db_advisory_lock_wait_seconds",
861+
&[
862+
("lock_type", "membership"),
863+
("kind", "none"),
864+
("outcome", "error"),
865+
],
866+
),
867+
(
868+
"buzz_db_advisory_lock_acquisitions_total",
869+
&[
870+
("lock_type", "replacement"),
871+
("kind", "13534"),
872+
("outcome", "success"),
873+
],
796874
),
797875
(
798876
"buzz_db_advisory_lock_acquisitions_total",
799-
[("lock_type", "replacement"), ("outcome", "success")],
877+
&[
878+
("lock_type", "replacement"),
879+
("kind", unregistered_kind.as_str()),
880+
("outcome", "success"),
881+
],
800882
),
801883
(
802884
"buzz_db_advisory_lock_acquisitions_total",
803-
[("lock_type", "membership"), ("outcome", "error")],
885+
&[
886+
("lock_type", "membership"),
887+
("kind", "none"),
888+
("outcome", "error"),
889+
],
804890
),
805891
(
806892
"buzz_db_transaction_duration_seconds",
807-
[
893+
&[
808894
("operation", "replace_parameterized_event"),
809895
("outcome", "success"),
810896
],
811897
),
812898
(
813899
"buzz_db_transaction_duration_seconds",
814-
[
900+
&[
815901
("operation", "accept_push_lease_event"),
816902
("outcome", "error"),
817903
],
818904
),
819905
] {
820906
let expected_labels = expected
821907
.1
822-
.into_iter()
823-
.map(|(key, value)| (key.to_owned(), value.to_owned()))
908+
.iter()
909+
.map(|&(key, value)| (key.to_owned(), value.to_owned()))
824910
.collect::<BTreeMap<_, _>>();
825911
assert!(
826912
keys.contains(&(expected.0.to_owned(), expected_labels)),
@@ -1761,8 +1847,9 @@ mod tests {
17611847
let _guard = metrics::set_default_local_recorder(&recorder);
17621848

17631849
let mut success_tx = pool.begin().await.expect("begin success transaction");
1764-
observe_advisory_lock(
1850+
observe_event_lock(
17651851
LockType::Replacement,
1852+
buzz_core::kind::KIND_NIP29_GROUP_MEMBERS as i32,
17661853
sqlx::query("SELECT pg_advisory_xact_lock($1)")
17671854
.bind(0x62757a7a6f627331_i64)
17681855
.execute(&mut *success_tx),
@@ -1890,6 +1977,16 @@ mod tests {
18901977
);
18911978
}
18921979

1980+
/// A replacement lock must carry its kind. Mutation: drop the guard in
1981+
/// `observe_advisory_lock` → RED.
1982+
#[cfg(debug_assertions)]
1983+
#[tokio::test]
1984+
#[should_panic(expected = "replacement locks carry an event kind")]
1985+
async fn kindless_observer_rejects_replacement_locks() {
1986+
let _: sqlx::Result<()> =
1987+
observe_advisory_lock(LockType::Replacement, async { Ok(()) }).await;
1988+
}
1989+
18931990
mod postgres_tests {
18941991
#[tokio::test(flavor = "current_thread")]
18951992
#[ignore = "requires Postgres"]

‎crates/buzz-db/src/store/channel.rs‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3327,8 +3327,9 @@ mod postgres_tests {
33273327
/// A group-state write takes its replacement lock once: the snapshot
33283328
/// takes it and the replacement runs under it, so the lock metrics see
33293329
/// one acquisition per write, as they do for
3330-
/// `Db::replace_addressable_event`. Mutation: take the lock again in
3331-
/// `replace_addressable_event_under_lock` → RED.
3330+
/// `Db::replace_addressable_event`, each labelled with the written kind.
3331+
/// Mutation: take the lock again in `replace_addressable_event_under_lock`,
3332+
/// or drop the kind from `take_replacement_lock` → RED.
33323333
#[tokio::test]
33333334
#[ignore = "requires Postgres"]
33343335
async fn group_state_replacement_takes_its_lock_once() {
@@ -3356,7 +3357,7 @@ mod postgres_tests {
33563357
let _guard = metrics::set_default_local_recorder(&recorder);
33573358
// A debugging snapshot drains what it reads, so each call returns the
33583359
// acquisitions since the previous one.
3359-
let replacement_acquisitions_since_last = || {
3360+
let replacement_acquisitions_since_last = |kind: &str| {
33603361
snapshotter
33613362
.snapshot()
33623363
.into_vec()
@@ -3368,6 +3369,10 @@ mod postgres_tests {
33683369
&& label.value()
33693370
== crate::observability::LockType::Replacement.as_str()
33703371
})
3372+
&& key
3373+
.key()
3374+
.labels()
3375+
.any(|label| label.key() == "kind" && label.value() == kind)
33713376
})
33723377
.map(|(_, _, _, value)| match value {
33733378
DebugValue::Counter(count) => count,
@@ -3390,7 +3395,7 @@ mod postgres_tests {
33903395
.expect("sign 39000");
33913396
assert!(snapshot.replace(&event).await.expect("store").1);
33923397
assert_eq!(
3393-
replacement_acquisitions_since_last(),
3398+
replacement_acquisitions_since_last("39000"),
33943399
1,
33953400
"one lock per group-state write"
33963401
);
@@ -3406,7 +3411,7 @@ mod postgres_tests {
34063411
.1
34073412
);
34083413
assert_eq!(
3409-
replacement_acquisitions_since_last(),
3414+
replacement_acquisitions_since_last("0"),
34103415
1,
34113416
"one lock per addressable replace"
34123417
);

‎crates/buzz-db/src/store/channel_members.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -575,14 +575,16 @@ pub async fn lock_member_snapshot(
575575
// this key before INSERT; migration 0032 then takes the membership key in
576576
// the INSERT trigger. Taking both in that order avoids mixed-version
577577
// duplicate heads without introducing a lock-order inversion.
578+
let members_kind = buzz_core::kind::KIND_NIP29_GROUP_MEMBERS as i32;
578579
let replacement_lock = crate::replaceable::event_replacement_lock_key(
579580
community_id,
580-
39002,
581+
members_kind,
581582
relay_pubkey,
582583
Some(channel_id.as_bytes()),
583584
);
584-
crate::observability::observe_advisory_lock(
585+
crate::observability::observe_event_lock(
585586
crate::observability::LockType::Replacement,
587+
members_kind,
586588
sqlx::query("SELECT pg_advisory_xact_lock($1)")
587589
.bind(replacement_lock)
588590
.execute(tx.conn()),

‎crates/buzz-db/src/store/relay_members.rs‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1082,8 +1082,9 @@ pub(crate) async fn write_nip43_membership_snapshot(
10821082
pubkey_bytes.as_slice(),
10831083
None,
10841084
);
1085-
observability::observe_advisory_lock(
1085+
observability::observe_event_lock(
10861086
observability::LockType::Membership,
1087+
kind_i32,
10871088
sqlx::query("SELECT pg_advisory_xact_lock($1)")
10881089
.bind(lock_key)
10891090
.execute(tx.conn()),
@@ -1723,6 +1724,57 @@ mod postgres_tests {
17231724
format!("{:064x}", Uuid::new_v4().as_u128())
17241725
}
17251726

1727+
/// The roster write records its list lock under `membership` with the
1728+
/// roster's kind. Mutation: observe the list lock without its kind → RED.
1729+
#[tokio::test]
1730+
#[ignore = "requires Postgres"]
1731+
async fn membership_list_lock_records_the_roster_kind() {
1732+
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
1733+
1734+
let pool = setup_pool().await;
1735+
let keys = nostr::Keys::generate();
1736+
let community = make_test_community(&pool).await;
1737+
let recorder = DebuggingRecorder::new();
1738+
let snapshotter = recorder.snapshotter();
1739+
let _guard = metrics::set_default_local_recorder(&recorder);
1740+
1741+
let mut tx = crate::begin_community_write_transaction(
1742+
&pool,
1743+
community,
1744+
observability::WriterOperation::Authorization,
1745+
)
1746+
.await
1747+
.expect("begin roster write");
1748+
write_nip43_membership_snapshot(&mut tx, &keys)
1749+
.await
1750+
.expect("write roster");
1751+
tx.commit().await.expect("commit roster");
1752+
1753+
let roster_kind = buzz_core::kind::KIND_NIP43_MEMBERSHIP_LIST.to_string();
1754+
let acquisitions = snapshotter
1755+
.snapshot()
1756+
.into_vec()
1757+
.into_iter()
1758+
.filter(|(key, _, _, _)| {
1759+
let label = |name: &str| {
1760+
key.key()
1761+
.labels()
1762+
.find(|label| label.key() == name)
1763+
.map(|label| label.value().to_owned())
1764+
};
1765+
key.key().name() == "buzz_db_advisory_lock_acquisitions_total"
1766+
&& label("lock_type").as_deref()
1767+
== Some(observability::LockType::Membership.as_str())
1768+
&& label("kind") == Some(roster_kind.clone())
1769+
})
1770+
.map(|(_, _, _, value)| match value {
1771+
DebugValue::Counter(count) => count,
1772+
other => panic!("acquisitions are a counter, got {other:?}"),
1773+
})
1774+
.sum::<u64>();
1775+
assert_eq!(acquisitions, 1, "one roster list lock per write");
1776+
}
1777+
17261778
async fn assert_role(pool: &PgPool, community: CommunityId, pubkey: &str, role: &str) {
17271779
assert_eq!(
17281780
get_relay_member(pool, community, pubkey)

0 commit comments

Comments
 (0)