feat(db): label advisory-lock metrics with the locked event kind - #9472

Why

On one deployment, buzz_db_advisory_lock_wait_seconds{lock_type="replacement"} added up to about 245,000 seconds of wait in 7 days. The metric cannot say which coordinates wait. Replacement locks serialize writers of one (kind, pubkey[, d_tag | channel]) coordinate. Relay-owned lists (kinds 13535, 39000–39002) go through the same generic replacement paths as user replaceables (kinds 0, 3, 10002, 30078, …). The 13534 membership list takes its coordinate lock under lock_type="membership". Before we change how these lists are stored, we need to know which kinds the wait comes from.

What

• buzz_db_advisory_lock_wait_seconds and buzz_db_advisory_lock_acquisitions_total get a kind label.
  • Locks keyed by an event coordinate record that coordinate's kind, through the new observe_event_lock(lock_type, kind, …). This covers all 4 replacement call sites (parameterized replace, take_replacement_lock for addressable replace and group-state snapshots, the kind:39002 member snapshot, and workflow deletion) and the kind:13534 membership-list lock. The 13534 lock keeps lock_type="membership", so totals by lock_type do not change.
  • All other locks record kind="none", and their call sites do not change. observe_advisory_lock has a debug_assert that rejects LockType::Replacement, so a replacement lock cannot lose its kind by mistake.
• buzz_core::kind::registered_kind_label(kind) gives the label value: the decimal kind when it is in ALL_KINDS, otherwise other.

Change to the #6700 label contract

#6700 says no database metric carries event kinds, and observability_source.rs enforces this with a blanket ban on "kind" =>. This PR makes one narrow exception: the two advisory-lock metrics.

• Bounded: the value set is the kind registry, not request data. Of the 135 registered kinds, 39 are replaceable or addressable and can reach a replacement lock. Any other kind a client sends becomes other. Non-replacement locks are always none. New series in the worst case: 39 kinds × 4 outcomes, for the histogram and the counter.
• Not sensitive: no community, pubkey, coordinate, or d-tag is attached. The kind of a replacement lock does not expose any of those.
• The guard is narrowed, not removed: "kind" => must appear exactly twice, inside the advisory-lock observer observe_lock, whose kind must come from registered_kind_label inside observe_event_lock. It is still forbidden everywhere else in observability.rs and in the datastore tracing macro. The Add operation-aware database pool acquisition metrics #7195 guard that forbids a literal "other" in observability.rs is unchanged, because the fallback lives in buzz-core.

Dashboards and alerts

Both metrics gain a label, so existing selectors keep working: lock_type="replacement" still matches every replacement series, because Prometheus and Datadog selectors match on a subset of labels. A consumer is affected only if it depends on the exact label set:

• Recording rules or panels that read a series without aggregating it now get one series per kind. Aggregate with sum by (lock_type, outcome) (...) to keep the old shape.
• Vector matching with on(lock_type, outcome) or ignoring(...) now has more series on one side. Aggregate before the join.

To read the new attribution, group by kind, for example sum by (lock_type, kind) (rate(buzz_db_advisory_lock_wait_seconds_sum[5m])). No consumer in this repository depends on the exact label set.

Tests

• buzz-core: registered_kind_label_bounds_unregistered_kinds checks that a registered kind passes through and that an unregistered or negative kind becomes other.
• buzz-db recorder test: checks the replacement series for a registered kind (13534) and an unregistered kind (other), and the kind="none" series for a membership lock, on both metrics.
• kindless_observer_rejects_replacement_locks: the debug_assert guard panics.
• membership_list_lock_records_the_roster_kind (Postgres): a 13534 list write records exactly one membership acquisition with kind="13534".
• group_state_replacement_takes_its_lock_once (Postgres): now counts acquisitions only under the written kind. It expects kind="39000" for the group-state snapshot path and kind="0" for replace_addressable_event, so if take_replacement_lock drops or swaps the kind, the count becomes 0 and the test fails.
• observability_source.rs: the narrowed guard described above.

Commands run:

• cargo clippy --workspace --all-targets -- -D warnings
• cargo test -p buzz-core --lib
• cargo test -p buzz-db --lib
• cargo test -p buzz-db --test observability_source
• scripts/postgres-test-run.sh -p buzz-db --lib --tests (the CI PostgreSQL lane, run locally)

Local relay check

The relay ran from this branch against an isolated database. /metrics showed one series per locked kind after each write:

• users set-profile: replacement, kind="0"
• users set-status: replacement, kind="30315"
• channels create: replacement, kind="39000", "39001", "39002"
• channels add-member: replacement, kind="39002"
• agents archive: replacement, kind="13535"
• moderation ban: replacement, kind="30622", plus membership, kind="none"
• Invite mint and claim (e2e_relay::test_invite_mint_and_claim_admits_new_pubkey): membership, kind="13534"
• Background loops: deletion and partition_maintenance, kind="none"

buzz_db_advisory_lock_wait_seconds_count matched buzz_db_advisory_lock_acquisitions_total for every series. The author also tested it against the local relay.

buzz-review-completed

🤖 Generated with Claude Code