You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 359839f
Browse filesBrowse the repository at this point in the historyBrowse files
authored
feat(db): label advisory-lock metrics with the locked event kind (#9472)
## Why
On one deployment,
`buzz_db_advisory_lock_wait_seconds{lock_type="replacement"}` added up
to about 245,000 seconds of wait in 7 days. The metric cannot say which
coordinates wait. Replacement locks serialize writers of one `(kind,
pubkey[, d_tag | channel])` coordinate. Relay-owned lists (kinds 13535,
39000–39002) go through the same generic replacement paths as user
replaceables (kinds 0, 3, 10002, 30078, …). The 13534 membership list
takes its coordinate lock under `lock_type="membership"`. Before we
change how these lists are stored, we need to know which kinds the wait
comes from.
## What
- `buzz_db_advisory_lock_wait_seconds` and
`buzz_db_advisory_lock_acquisitions_total` get a `kind` label.
- Locks keyed by an event coordinate record that coordinate's kind,
through the new `observe_event_lock(lock_type, kind, …)`. This covers
all 4 replacement call sites (parameterized replace,
`take_replacement_lock` for addressable replace and group-state
snapshots, the kind:39002 member snapshot, and workflow deletion) and
the kind:13534 membership-list lock. The 13534 lock keeps
`lock_type="membership"`, so totals by `lock_type` do not change.
- All other locks record `kind="none"`, and their call sites do not
change. `observe_advisory_lock` has a `debug_assert` that rejects
`LockType::Replacement`, so a replacement lock cannot lose its kind by
mistake.
- `buzz_core::kind::registered_kind_label(kind)` gives the label value:
the decimal kind when it is in `ALL_KINDS`, otherwise `other`.
## Change to the #6700 label contract
#6700 says no database metric carries event kinds, and
`observability_source.rs` enforces this with a blanket ban on `"kind"
=>`. This PR makes one narrow exception: the two advisory-lock metrics.
- **Bounded:** the value set is the kind registry, not request data. Of
the 135 registered kinds, 39 are replaceable or addressable and can
reach a replacement lock. Any other kind a client sends becomes `other`.
Non-replacement locks are always `none`. New series in the worst case:
39 kinds × 4 outcomes, for the histogram and the counter.
- **Not sensitive:** no community, pubkey, coordinate, or d-tag is
attached. The kind of a replacement lock does not expose any of those.
- **The guard is narrowed, not removed:** `"kind" =>` must appear
exactly twice, inside the advisory-lock observer `observe_lock`, whose
kind must come from `registered_kind_label` inside `observe_event_lock`.
It is still forbidden everywhere else in `observability.rs` and in the
datastore tracing macro. The #7195 guard that forbids a literal
`"other"` in `observability.rs` is unchanged, because the fallback lives
in `buzz-core`.
## Dashboards and alerts
Both metrics gain a label, so existing selectors keep working:
`lock_type="replacement"` still matches every replacement series,
because Prometheus and Datadog selectors match on a subset of labels. A
consumer is affected only if it depends on the exact label set:
- Recording rules or panels that read a series without aggregating it
now get one series per kind. Aggregate with `sum by (lock_type, outcome)
(...)` to keep the old shape.
- Vector matching with `on(lock_type, outcome)` or `ignoring(...)` now
has more series on one side. Aggregate before the join.
To read the new attribution, group by `kind`, for example `sum by
(lock_type, kind) (rate(buzz_db_advisory_lock_wait_seconds_sum[5m]))`.
No consumer in this repository depends on the exact label set.
## Tests
- `buzz-core`: `registered_kind_label_bounds_unregistered_kinds` checks
that a registered kind passes through and that an unregistered or
negative kind becomes `other`.
- `buzz-db` recorder test: checks the replacement series for a
registered kind (`13534`) and an unregistered kind (`other`), and the
`kind="none"` series for a membership lock, on both metrics.
- `kindless_observer_rejects_replacement_locks`: the `debug_assert`
guard panics.
- `membership_list_lock_records_the_roster_kind` (Postgres): a 13534
list write records exactly one `membership` acquisition with
`kind="13534"`.
- `group_state_replacement_takes_its_lock_once` (Postgres): now counts
acquisitions only under the written kind. It expects `kind="39000"` for
the group-state snapshot path and `kind="0"` for
`replace_addressable_event`, so if `take_replacement_lock` drops or
swaps the kind, the count becomes 0 and the test fails.
- `observability_source.rs`: the narrowed guard described above.
Commands run:
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test -p buzz-core --lib`
- `cargo test -p buzz-db --lib`
- `cargo test -p buzz-db --test observability_source`
- `scripts/postgres-test-run.sh -p buzz-db --lib --tests` (the CI
PostgreSQL lane, run locally)
## Local relay check
The relay ran from this branch against an isolated database. `/metrics`
showed one series per locked kind after each write:
- `users set-profile`: `replacement`, `kind="0"`
- `users set-status`: `replacement`, `kind="30315"`
- `channels create`: `replacement`, `kind="39000"`, `"39001"`, `"39002"`
- `channels add-member`: `replacement`, `kind="39002"`
- `agents archive`: `replacement`, `kind="13535"`
- `moderation ban`: `replacement`, `kind="30622"`, plus `membership`,
`kind="none"`
- Invite mint and claim
(`e2e_relay::test_invite_mint_and_claim_admits_new_pubkey`):
`membership`, `kind="13534"`
- Background loops: `deletion` and `partition_maintenance`,
`kind="none"`
`buzz_db_advisory_lock_wait_seconds_count` matched
`buzz_db_advisory_lock_acquisitions_total` for every series. The author
also tested it against the local relay.
buzz-review-completed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Ian Oberst <[email protected]>
Co-authored-by: Ian Oberst <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
0 commit comments