Skip to content

Commit 1d91e38

Browse files
Ian Oberstclaude
andcommitted
fix(relay): keep the partition audit success timestamp through failures
Refresh `buzz_partition_audit_last_success_timestamp_seconds` without changing it on each failed table audit, so the exporter's idle timeout no longer evicts it during sustained failures and a never-successful table exports 0. Timestamp age alone now covers failures with no false resolve. Replace the absence rule with a counter-only rule for a stopped audit loop that fires before the stale timestamp is evicted. Document that runs_total has no `unsafe` outcome, and that the tests check emitted series rather than evaluating the rules. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Ian Oberst <[email protected]>
1 parent 3d57a48 commit 1d91e38

4 files changed

Lines changed: 97 additions & 88 deletions

File tree

‎crates/buzz-db/src/store/partition.rs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -453,6 +453,7 @@ async fn audit_partition_catalog_report_at_with_timeout(
453453
"outcome" => "error"
454454
)
455455
.increment(1);
456+
refresh_last_success_recency(table);
456457
metrics::histogram!(
457458
"buzz_partition_audit_duration_seconds",
458459
"table" => table
@@ -470,6 +471,7 @@ async fn audit_partition_catalog_report_at_with_timeout(
470471
"outcome" => "error"
471472
)
472473
.increment(1);
474+
refresh_last_success_recency(table);
473475
metrics::histogram!(
474476
"buzz_partition_audit_duration_seconds",
475477
"table" => table
@@ -1091,6 +1093,17 @@ fn qualified_relation_name(schema: &str, relation: &str) -> String {
10911093
)
10921094
}
10931095

1096+
/// Keep a table's last success timestamp exported through failed audits
1097+
/// without changing it, so the exporter's idle timeout cannot evict it. A
1098+
/// table that never succeeded exports 0, which reads as maximally stale.
1099+
fn refresh_last_success_recency(table: &'static str) {
1100+
metrics::gauge!(
1101+
"buzz_partition_audit_last_success_timestamp_seconds",
1102+
"table" => table
1103+
)
1104+
.increment(0.0);
1105+
}
1106+
10941107
fn emit_audit_metrics(audit: &PartitionTableAudit, duration_seconds: f64, now: DateTime<Utc>) {
10951108
let outcome = if audit.degraded() { "degraded" } else { "ok" };
10961109
let uncovered = audit

‎crates/buzz-relay/src/metrics.rs‎

Lines changed: 43 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1161,48 +1161,38 @@ mod tests {
11611161
}
11621162
}
11631163

1164-
/// Fixtures for the absence-aware partition audit alerts in
1165-
/// `docs/partition-catalog-monitoring.md`, evaluated against the production
1166-
/// exporter configuration and the production audit metric emission.
1164+
/// Fixtures for the partition audit alerts in
1165+
/// `docs/partition-catalog-monitoring.md`: the production exporter configuration
1166+
/// and audit metric emission export the series those rules read.
11671167
#[cfg(test)]
11681168
mod partition_alert_tests {
1169-
use std::collections::BTreeSet;
1169+
use std::collections::BTreeMap;
11701170
use std::time::Duration;
11711171

11721172
const RUNS: &str = "buzz_partition_audit_runs_total";
11731173
const LAST_SUCCESS: &str = "buzz_partition_audit_last_success_timestamp_seconds";
11741174
const MANAGED_TABLES: [&str; 2] = ["delivery_log", "events"];
11751175

1176-
/// `table` label values of every sample of `metric` in a scrape.
1177-
fn tables(scrape: &str, metric: &str) -> BTreeSet<String> {
1176+
/// Sample value by `table` label for every sample of `metric` in a scrape.
1177+
/// Counters with several `outcome` labels keep the last sample per table.
1178+
fn samples(scrape: &str, metric: &str) -> BTreeMap<String, f64> {
11781179
scrape
11791180
.lines()
11801181
.filter_map(|line| line.strip_prefix(metric)?.strip_prefix('{'))
11811182
.filter_map(|labels| {
11821183
let (_, rest) = labels.split_once(r#"table=""#)?;
1183-
rest.split_once('"').map(|(table, _)| table.to_string())
1184+
let (table, rest) = rest.split_once('"')?;
1185+
let (_, value) = rest.rsplit_once(' ')?;
1186+
Some((table.to_string(), value.parse().expect("numeric sample")))
11841187
})
11851188
.collect()
11861189
}
11871190

1188-
/// Evaluate the documented absence rule for one scrape target:
1189-
/// `max without (outcome) (buzz_partition_audit_runs_total)
1190-
/// unless buzz_partition_audit_last_success_timestamp_seconds`.
1191-
fn missing_success_series(scrape: &str) -> BTreeSet<String> {
1192-
&tables(scrape, RUNS) - &tables(scrape, LAST_SUCCESS)
1193-
}
1194-
1195-
fn last_success_value(scrape: &str, table: &str) -> Option<f64> {
1196-
scrape
1197-
.lines()
1198-
.find(|line| {
1199-
line.starts_with(LAST_SUCCESS) && line.contains(&format!(r#"table="{table}""#))
1200-
})
1201-
.and_then(|line| line.rsplit_once(' '))
1202-
.map(|(_, value)| value.parse().expect("numeric sample"))
1191+
fn tables(scrape: &str, metric: &str) -> Vec<String> {
1192+
samples(scrape, metric).into_keys().collect()
12031193
}
12041194

1205-
fn managed_tables() -> BTreeSet<String> {
1195+
fn managed_tables() -> Vec<String> {
12061196
MANAGED_TABLES
12071197
.iter()
12081198
.map(|table| table.to_string())
@@ -1225,7 +1215,7 @@ mod partition_alert_tests {
12251215
}
12261216

12271217
#[test]
1228-
fn never_successful_audits_leave_expected_series_without_a_success_timestamp() {
1218+
fn never_successful_audits_export_a_zero_success_timestamp() {
12291219
let (recorder, handle) = super::readiness_test_recorder_with_idle_timeout(1);
12301220
metrics::with_local_recorder(&recorder, || {
12311221
current_thread_runtime().block_on(async {
@@ -1235,58 +1225,62 @@ mod partition_alert_tests {
12351225

12361226
let scrape = handle.render();
12371227
assert_eq!(tables(&scrape, RUNS), managed_tables(), "{scrape}");
1238-
assert!(tables(&scrape, LAST_SUCCESS).is_empty(), "{scrape}");
1239-
assert_eq!(missing_success_series(&scrape), managed_tables());
1228+
let zero = MANAGED_TABLES
1229+
.iter()
1230+
.map(|table| (table.to_string(), 0.0))
1231+
.collect::<BTreeMap<_, _>>();
1232+
assert_eq!(samples(&scrape, LAST_SUCCESS), zero, "{scrape}");
12401233
}
12411234

12421235
mod postgres_tests {
12431236
use super::*;
12441237

12451238
#[test]
12461239
#[ignore = "requires Postgres"]
1247-
fn evicted_success_timestamps_leave_expected_series_behind() {
1240+
fn failed_audits_keep_the_last_success_past_the_idle_timeout() {
12481241
let idle_timeout = Duration::from_secs(1);
12491242
let (recorder, handle) =
12501243
crate::metrics::readiness_test_recorder_with_idle_timeout(idle_timeout.as_secs());
12511244
let runtime = current_thread_runtime();
1252-
let failing = metrics::with_local_recorder(&recorder, || {
1245+
let (working, failing) = metrics::with_local_recorder(&recorder, || {
12531246
runtime.block_on(async {
12541247
let pool = sqlx::PgPool::connect(&crate::test_support::database_url())
12551248
.await
12561249
.expect("connect test database");
1257-
buzz_db::Db::from_pool(pool)
1258-
.audit_partitions(3)
1259-
.await
1260-
.expect("successful audit");
1261-
unreachable_db()
1250+
let working = buzz_db::Db::from_pool(pool);
1251+
working.audit_partitions(3).await.expect("successful audit");
1252+
(working, unreachable_db())
12621253
})
12631254
});
12641255

12651256
let scrape = handle.render();
1266-
assert!(missing_success_series(&scrape).is_empty(), "{scrape}");
1267-
let first = last_success_value(&scrape, "events").expect("success timestamp");
1257+
let first = samples(&scrape, LAST_SUCCESS);
1258+
assert_eq!(first.keys().cloned().collect::<Vec<_>>(), managed_tables());
12681259

1269-
let audit_failure = || {
1260+
let audit = |db: &buzz_db::Db| {
12701261
metrics::with_local_recorder(&recorder, || {
1271-
runtime.block_on(async {
1272-
assert!(failing.audit_partitions(3).await.is_err());
1273-
});
1274-
});
1262+
runtime.block_on(async { db.audit_partitions(3).await.is_ok() })
1263+
})
12751264
};
1276-
// A failed audit never makes the last success look fresh.
1277-
audit_failure();
1278-
let scrape = handle.render();
1279-
assert_eq!(last_success_value(&scrape, "events"), Some(first));
1280-
assert!(missing_success_series(&scrape).is_empty(), "{scrape}");
1265+
// A failed audit never makes the last success look fresh, and
1266+
// sustained failures keep it exported past the idle timeout, so its
1267+
// age keeps rising instead of the series disappearing.
1268+
assert!(!audit(&failing));
1269+
assert_eq!(samples(&handle.render(), LAST_SUCCESS), first);
1270+
std::thread::sleep(idle_timeout + Duration::from_millis(250));
1271+
assert!(!audit(&failing));
1272+
assert_eq!(samples(&handle.render(), LAST_SUCCESS), first);
12811273

1282-
// Sustained failures let the exporter evict the success timestamp;
1283-
// the per-table run counters still identify what to expect.
1274+
// When audits stop running altogether the timestamp is evicted,
1275+
// but the run counters remain for the not-running rule.
12841276
std::thread::sleep(idle_timeout + Duration::from_millis(250));
1285-
audit_failure();
12861277
let scrape = handle.render();
12871278
assert!(tables(&scrape, LAST_SUCCESS).is_empty(), "{scrape}");
12881279
assert_eq!(tables(&scrape, RUNS), managed_tables(), "{scrape}");
1289-
assert_eq!(missing_success_series(&scrape), managed_tables());
1280+
1281+
// The next success exports a fresh timestamp again.
1282+
assert!(audit(&working));
1283+
assert_eq!(tables(&handle.render(), LAST_SUCCESS), managed_tables());
12901284
}
12911285
}
12921286
}

‎crates/buzz-relay/src/state.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1192,9 +1192,9 @@ impl AppState {
11921192
/// Snapshot the last completed audit without accessing the database.
11931193
///
11941194
/// Periodic refresh failures retain the last-known-good audit by design.
1195-
/// Alert on both the age and the absence of
1196-
/// `buzz_partition_audit_last_success_timestamp_seconds`; the exporter
1197-
/// evicts it during sustained failures. See
1195+
/// Alert on the age of
1196+
/// `buzz_partition_audit_last_success_timestamp_seconds`, which failed
1197+
/// audits keep exported, and on audit runs stopping. See
11981198
/// `docs/partition-catalog-monitoring.md`.
11991199
pub fn partition_audit_snapshot(&self) -> Option<buzz_db::partition::PartitionAudit> {
12001200
let cached = match self.partition_audit.read() {

‎docs/partition-catalog-monitoring.md‎

Lines changed: 38 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,7 @@ gate traffic on them.
88

99
## Outcomes
1010

11-
`buzz-admin partition-audit` prints one outcome. The relay uses the same
12-
per-table verdict for `buzz_partition_audit_runs_total{outcome}`.
11+
`buzz-admin partition-audit` prints one outcome for the whole catalog.
1312

1413
| Outcome | Meaning | `buzz-admin` exit code |
1514
|---|---|---|
@@ -18,6 +17,11 @@ per-table verdict for `buzz_partition_audit_runs_total{outcome}`.
1817
| `unsafe` | A table cannot route a row timestamped now. | 2 |
1918
| `error` | At least one table could not be audited. | 5 |
2019

20+
The relay's `buzz_partition_audit_runs_total{outcome}` is per table and is only
21+
`ok`, `degraded`, or `error`. There is no `unsafe` outcome there: a table that
22+
cannot route a row timestamped now is `degraded` and has
23+
`buzz_partition_serving_safe` 0.
24+
2125
A table is `degraded` when it has a partition-key mismatch, a `DEFAULT` or
2226
anomalous child, trigger drift, an occupied catch-all or `DEFAULT` leaf, or a
2327
current or lookahead month not covered by a bounded leaf. A bounded legacy leaf
@@ -37,12 +41,14 @@ Per-table gauges are set only by a successful audit of that table:
3741
`buzz_partition_trigger_parity_extra`. A failed audit leaves them unchanged, so a
3842
failure never makes the last success look fresh.
3943

40-
The exporter removes a gauge that has not been set within its idle timeout. The
41-
default is 2,700 seconds: three times the larger of the usage-metrics and
42-
partition-audit intervals. Sustained audit failures therefore remove the last
43-
success timestamp, and a table that has never been audited successfully has no
44-
timestamp at all. An alert based only on timestamp age has no input in either
45-
case.
44+
The exporter removes a gauge that has not been updated within its idle timeout.
45+
The default is 2,700 seconds: three times the larger of the usage-metrics and
46+
partition-audit intervals. A failed audit of a table refreshes that table's
47+
`buzz_partition_audit_last_success_timestamp_seconds` without changing its
48+
value, so sustained failures keep the last success exported and its age keeps
49+
rising. A table that has never been audited successfully exports `0`. The
50+
timestamp disappears only when audits stop running for longer than the idle
51+
timeout; the other per-table gauges can be evicted during sustained failures.
4652

4753
Counters are not idle-evicted while the process runs:
4854

@@ -59,52 +65,48 @@ Counters are not idle-evicted while the process runs:
5965

6066
## Alerts
6167

62-
Use all three rules below. The run counter defines the expected live instances
63-
and managed tables. When a pod terminates, its scrape target disappears and its
64-
series go stale. Normal pod turnover therefore cannot leave a persistent
65-
missing-series alert. Aggregate with `without (...)`, not `by (...)`, so
66-
cluster, account, environment, namespace, and pod labels stay in the alert.
68+
Use all three rules below. Aggregate with `without (...)`, not `by (...)`, so
69+
cluster, account, environment, namespace, and pod labels stay in the alert. When
70+
a pod terminates, its scrape target disappears and its series go stale, so
71+
normal pod turnover cannot leave an alert firing.
6772

6873
```yaml
69-
# A live relay audits this table but exports no success timestamp. This fires
70-
# when a table has never audited successfully, and after sustained failures
71-
# have evicted a previous success timestamp.
72-
- alert: BuzzPartitionAuditSuccessMissing
73-
expr: |
74-
max without (outcome) (buzz_partition_audit_runs_total)
75-
unless buzz_partition_audit_last_success_timestamp_seconds
76-
for: 15m
77-
78-
# The last success is older than two default audit intervals. Scale 1800 with
79-
# BUZZ_PARTITION_AUDIT_INTERVAL_SECS. Keep threshold plus `for` below the gauge
80-
# idle timeout (2,700 s by default) so this fires before the absence rule
81-
# takes over.
74+
# The last success is older than two default audit intervals. This also fires
75+
# for a table that has never audited successfully, whose timestamp is 0, and
76+
# keeps firing through sustained failures. Scale 1800 with
77+
# BUZZ_PARTITION_AUDIT_INTERVAL_SECS.
8278
- alert: BuzzPartitionAuditStale
8379
expr: time() - buzz_partition_audit_last_success_timestamp_seconds > 1800
8480
for: 10m
8581

82+
# The relay is running but its audit loop has stopped. Use a window of about
83+
# two audit intervals, and keep the window plus `for` below the gauge idle
84+
# timeout (2,700 s by default), so this fires before the stopped loop's
85+
# timestamp is evicted and the stale rule resolves.
86+
- alert: BuzzPartitionAuditNotRunning
87+
expr: sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
88+
for: 5m
89+
8690
# Explicit audit attempts keep failing, even if a later attempt succeeds.
8791
- alert: BuzzPartitionAuditFailing
8892
expr: increase(buzz_partition_audit_failures_total[1h]) >= 2
8993
```
9094
91-
To also catch a live relay that never exports audit runs, compare its scrape
92-
target health with the run counter, for example
95+
All three rules read series that exist after the first audit attempt, whether
96+
or not it succeeded. To also catch a live relay that never exports audit runs,
97+
compare its scrape target health with the run counter, for example
9398
`up == 1 unless on (namespace, pod) count by (namespace, pod) (buzz_partition_audit_runs_total)`,
9499
scoped to the relay job.
95100

96-
Backends without set subtraction can express the absence rule as "every audit
97-
of this table on this pod failed in the window." Divide the per-pod, per-table
98-
count of `outcome="error"` runs by all runs over a window of at least four audit
99-
intervals. Alert when the ratio is 1 and there were attempts. Both queries read
100-
counters, so this works after the timestamp gauge is evicted.
101-
102101
`degraded` asks for operator attention; it does not mean rows cannot be
103102
written. Because historical legacy leaves no longer count, a repaired catalog
104103
returns to `ok` once the current month passes the repaired range. Coverage
105104
problems show directly in `buzz_partition_serving_safe` and
106105
`buzz_partition_uncovered_months`.
107106

108107
The tests in `crates/buzz-relay/src/metrics.rs` (`partition_alert_tests`) check
109-
that the production exporter and audit emit the series these rules need, in
110-
both the never-successful and evicted-after-success cases.
108+
that the production exporter and audit emit the series these rules read: a `0`
109+
timestamp before any success, the last success kept through failures past the
110+
idle timeout, and run counters that outlive an evicted timestamp. They do not
111+
evaluate the rules, their `for` durations, or label matching; validate those
112+
with `promtool test rules` against your alerting configuration.

0 commit comments

Comments
 (0)