Skip to content

Commit c3a314e

Browse files
Ian Oberstclaude
andcommitted
fix(relay): export partition audit counters at zero on startup
Prometheus increase() ignores the first sample of a new series. The audit run and failure counters were created only when an outcome first occurred, so the first periodic failure read as no increase, and a table's first run with a new outcome could look like a stopped audit loop to the documented alerts. Register every buzz_partition_audit_runs_total{table, outcome} series and buzz_partition_audit_failures_total at 0 when the metrics exporter starts, before the startup audit, following the existing auth series baselines. Document the remaining gap: a failure in the startup audit itself can land before the first scrape, and the stale rule covers it. Also gate the two range-based alert rules on the counter being exported now. Range selectors keep reading a terminated pod's samples until they leave the window, so BuzzPartitionAuditNotRunning could fire for a pod that was replaced during normal turnover, and BuzzPartitionAuditFailing could keep firing for up to an hour after it. The counters now exist for the whole life of a relay, so `and <counter>` limits both rules to live pods. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Ian Oberst <[email protected]>
1 parent 1d91e38 commit c3a314e

3 files changed

Lines changed: 96 additions & 14 deletions

File tree

‎crates/buzz-db/src/store/partition.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1093,6 +1093,24 @@ fn qualified_relation_name(schema: &str, relation: &str) -> String {
10931093
)
10941094
}
10951095

1096+
/// Export every `buzz_partition_audit_runs_total` series at zero.
1097+
///
1098+
/// Prometheus `increase()` ignores the first sample of a new series, so an
1099+
/// outcome first seen mid-run would otherwise be invisible to rate-based
1100+
/// alerts. Call once at process start, before the first audit.
1101+
pub fn initialize_audit_metric_series() {
1102+
for &table in PARTITIONED_TABLES {
1103+
for outcome in ["ok", "degraded", "error"] {
1104+
metrics::counter!(
1105+
"buzz_partition_audit_runs_total",
1106+
"table" => table,
1107+
"outcome" => outcome
1108+
)
1109+
.increment(0);
1110+
}
1111+
}
1112+
}
1113+
10961114
/// Keep a table's last success timestamp exported through failed audits
10971115
/// without changing it, so the exporter's idle timeout cannot evict it. A
10981116
/// table that never succeeded exports 0, which reads as maximally stale.

‎crates/buzz-relay/src/metrics.rs‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,7 @@ pub fn try_install(port: u16, gauge_idle_timeout_secs: u64) -> Result<(), Metric
298298
describe_db_pool_metrics();
299299
describe_auth_metrics();
300300
initialize_auth_metric_series();
301+
initialize_partition_audit_metric_series();
301302
tokio::spawn(exporter);
302303
Ok(())
303304
}
@@ -521,6 +522,13 @@ pub(crate) fn initialize_auth_metric_series() {
521522
metrics::gauge!("buzz_ws_authenticated_connections_active").set(0.0);
522523
}
523524

525+
/// Export the partition audit counters at zero before the first audit, so
526+
/// `increase()`-based alerts count the first failure and every new outcome.
527+
pub(crate) fn initialize_partition_audit_metric_series() {
528+
buzz_db::partition::initialize_audit_metric_series();
529+
metrics::counter!("buzz_partition_audit_failures_total").increment(0);
530+
}
531+
524532
/// Record one issued NIP-42 challenge after it enters the connection writer.
525533
///
526534
/// Before this lifecycle contract, `buzz_auth_attempts_total` incremented only
@@ -1232,6 +1240,46 @@ mod partition_alert_tests {
12321240
assert_eq!(samples(&scrape, LAST_SUCCESS), zero, "{scrape}");
12331241
}
12341242

1243+
#[test]
1244+
fn audit_counters_export_zero_baselines_before_the_first_audit() {
1245+
let (recorder, handle) = super::readiness_test_recorder_with_idle_timeout(1);
1246+
metrics::with_local_recorder(&recorder, super::initialize_partition_audit_metric_series);
1247+
1248+
let scrape = handle.render();
1249+
let zero_runs = scrape
1250+
.lines()
1251+
.filter(|line| line.starts_with(&format!("{RUNS}{{")) && line.ends_with(" 0"))
1252+
.count();
1253+
assert_eq!(zero_runs, MANAGED_TABLES.len() * 3, "{scrape}");
1254+
assert!(
1255+
scrape
1256+
.lines()
1257+
.any(|line| line == "buzz_partition_audit_failures_total 0"),
1258+
"{scrape}"
1259+
);
1260+
1261+
// A first failure then reads as an increase from the zero baseline.
1262+
metrics::with_local_recorder(&recorder, || {
1263+
current_thread_runtime().block_on(async {
1264+
assert!(unreachable_db().audit_partitions(3).await.is_err());
1265+
});
1266+
});
1267+
let scrape = handle.render();
1268+
for table in MANAGED_TABLES {
1269+
for (outcome, value) in [("ok", 0), ("degraded", 0), ("error", 1)] {
1270+
let line = scrape
1271+
.lines()
1272+
.find(|line| {
1273+
line.starts_with(&format!("{RUNS}{{"))
1274+
&& line.contains(&format!(r#"table="{table}""#))
1275+
&& line.contains(&format!(r#"outcome="{outcome}""#))
1276+
})
1277+
.unwrap_or_else(|| panic!("{table}/{outcome} missing:\n{scrape}"));
1278+
assert!(line.ends_with(&format!(" {value}")), "{line}");
1279+
}
1280+
}
1281+
}
1282+
12351283
mod postgres_tests {
12361284
use super::*;
12371285

‎docs/partition-catalog-monitoring.md‎

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -50,12 +50,14 @@ rising. A table that has never been audited successfully exports `0`. The
5050
timestamp disappears only when audits stop running for longer than the idle
5151
timeout; the other per-table gauges can be evicted during sustained failures.
5252

53-
Counters are not idle-evicted while the process runs:
53+
Counters are not idle-evicted while the process runs. The relay exports every
54+
run outcome for both managed tables, and the failure counter, at 0 when its
55+
metrics exporter starts, so `increase()` counts the first failure and the first
56+
run with a new outcome:
5457

5558
- `buzz_partition_audit_runs_total{table, outcome}` increments once per table
5659
per audit. The outcome is `ok`, `degraded`, or `error`, including audits run
57-
inside startup partition creation. After the first audit, every live relay
58-
exports this counter for both managed tables.
60+
inside startup partition creation.
5961
- `buzz_partition_audit_failures_total` counts failed explicit audit attempts:
6062
the startup fallback audit that runs when startup partition creation fails, and
6163
each periodic audit. A startup creation failure is not counted by itself. If
@@ -66,9 +68,14 @@ Counters are not idle-evicted while the process runs:
6668
## Alerts
6769

6870
Use all three rules below. Aggregate with `without (...)`, not `by (...)`, so
69-
cluster, account, environment, namespace, and pod labels stay in the alert. When
70-
a pod terminates, its scrape target disappears and its series go stale, so
71-
normal pod turnover cannot leave an alert firing.
71+
cluster, account, environment, namespace, and pod labels stay in the alert.
72+
73+
When a pod terminates, its scrape target disappears and its series go stale.
74+
Instant selectors drop a stale series at once, but range selectors such as
75+
`[30m]` keep reading its earlier samples until they leave the window. Each
76+
range-based rule therefore also requires the counter to be exported now, which
77+
it is for the whole life of a relay, so a terminated pod cannot keep or start an
78+
alert.
7279

7380
```yaml
7481
# The last success is older than two default audit intervals. This also fires
@@ -84,17 +91,25 @@ normal pod turnover cannot leave an alert firing.
8491
# timeout (2,700 s by default), so this fires before the stopped loop's
8592
# timestamp is evicted and the stale rule resolves.
8693
- alert: BuzzPartitionAuditNotRunning
87-
expr: sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
94+
expr: |
95+
sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
96+
and sum without (outcome) (buzz_partition_audit_runs_total)
8897
for: 5m
8998

9099
# Explicit audit attempts keep failing, even if a later attempt succeeds.
91100
- alert: BuzzPartitionAuditFailing
92-
expr: increase(buzz_partition_audit_failures_total[1h]) >= 2
101+
expr: |
102+
increase(buzz_partition_audit_failures_total[1h]) >= 2
103+
and buzz_partition_audit_failures_total
93104
```
94105
95-
All three rules read series that exist after the first audit attempt, whether
96-
or not it succeeded. To also catch a live relay that never exports audit runs,
97-
compare its scrape target health with the run counter, for example
106+
The counters exist from relay start, and the success timestamp exists after the
107+
first audit attempt, whether or not it succeeded. The startup audit runs within
108+
seconds of the zero baselines, usually before the first scrape, so
109+
`BuzzPartitionAuditFailing` can miss a failure in that one attempt; if it does
110+
not recover, `BuzzPartitionAuditStale` fires. To also catch a live relay that
111+
never exports audit runs, compare its scrape target health with the run
112+
counter, for example
98113
`up == 1 unless on (namespace, pod) count by (namespace, pod) (buzz_partition_audit_runs_total)`,
99114
scoped to the relay job.
100115

@@ -105,8 +120,9 @@ problems show directly in `buzz_partition_serving_safe` and
105120
`buzz_partition_uncovered_months`.
106121

107122
The tests in `crates/buzz-relay/src/metrics.rs` (`partition_alert_tests`) check
108-
that the production exporter and audit emit the series these rules read: a `0`
109-
timestamp before any success, the last success kept through failures past the
110-
idle timeout, and run counters that outlive an evicted timestamp. They do not
123+
that the production exporter and audit emit the series these rules read: zero
124+
counter baselines before the first audit, a `0` timestamp before any success,
125+
the last success kept through failures past the idle timeout, and run counters
126+
that outlive an evicted timestamp. They do not
111127
evaluate the rules, their `for` durations, or label matching; validate those
112128
with `promtool test rules` against your alerting configuration.

0 commit comments

Comments
 (0)