Skip to content

Commit e47f7a2

Browse files
Ian Oberstclaude
andcommitted
docs(relay): keep the audit-loop alert quiet on new pods
After a successful startup audit, the first periodic audit runs a full interval later, so a new pod's run counter is flat for that interval and BuzzPartitionAuditNotRunning fired on every rollout. Require the counter to have existed one window ago; a loop that stops in a pod's first window is then caught at about 35 minutes. Check the rules in with a promtool test covering a fresh pod, a stopped loop, early loop death, a terminated pod, a never-successful table, and repeated failures. Drop the up-based query, which the zero baselines made unable to fire, and note that catch-all-covered lookahead months keep a repaired catalog degraded until advancement. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Ian Oberst <[email protected]>
1 parent 803db3a commit e47f7a2

3 files changed

Lines changed: 167 additions & 10 deletions

File tree

‎docs/partition-catalog-monitoring.md‎

Lines changed: 29 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,11 @@ range-based rule therefore also requires the counter to be exported now, which
8080
it is for the whole life of a relay, so a terminated pod cannot keep or start an
8181
alert.
8282

83+
`BuzzPartitionAuditNotRunning` also requires the run counter to have existed one
84+
window ago. After a successful startup audit, the first periodic audit runs a
85+
full interval later, so a new pod's counter stays flat until then and would
86+
otherwise read as a stopped loop.
87+
8388
```yaml
8489
# The last success is older than two default audit intervals. This also fires
8590
# for a table that has never audited successfully, whose timestamp is 0, and
@@ -92,11 +97,14 @@ alert.
9297
# The relay is running but its audit loop has stopped. Use a window of about
9398
# two audit intervals, and keep the window plus `for` below the gauge idle
9499
# timeout (2,700 s by default), so this fires before the stopped loop's
95-
# timestamp is evicted and the stale rule resolves.
100+
# timestamp is evicted and the stale rule resolves. Use the same duration for
101+
# the window and the offset. A loop that stops in a pod's first window is
102+
# caught once the pod is a window old, at about 35 minutes.
96103
- alert: BuzzPartitionAuditNotRunning
97104
expr: |
98105
sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
99106
and sum without (outcome) (buzz_partition_audit_runs_total)
107+
and sum without (outcome) (buzz_partition_audit_runs_total offset 30m)
100108
for: 5m
101109

102110
# Explicit audit attempts keep failing, even if a later attempt succeeds.
@@ -110,15 +118,16 @@ The counters exist from relay start, and the success timestamp exists after the
110118
first audit attempt, whether or not it succeeded. The startup audit runs within
111119
seconds of the zero baselines, usually before the first scrape, so
112120
`BuzzPartitionAuditFailing` can miss a failure in that one attempt; if it does
113-
not recover, `BuzzPartitionAuditStale` fires. To also catch a live relay that
114-
never exports audit runs, compare its scrape target health with the run
115-
counter, for example
116-
`up == 1 unless on (namespace, pod) count by (namespace, pod) (buzz_partition_audit_runs_total)`,
117-
scoped to the relay job.
121+
not recover, `BuzzPartitionAuditStale` fires. The run counters are exported
122+
before the relay connects to the database, so a live relay that is stuck before
123+
its first audit shows flat zero counters, which `BuzzPartitionAuditNotRunning`
124+
catches.
118125

119126
`degraded` asks for operator attention; it does not mean rows cannot be
120-
written. Because historical legacy leaves no longer count, a repaired catalog
121-
returns to `ok` once the current month passes the repaired range. Coverage
127+
written. Historical legacy leaves no longer count, so a repaired catalog
128+
returns to `ok` once the current month passes the repaired range and every
129+
current and lookahead month is a bounded monthly partition. Months still covered
130+
by a catch-all stay `degraded` until advancement moves the catch-all. Coverage
122131
problems show directly in `buzz_partition_serving_safe` and
123132
`buzz_partition_uncovered_months`.
124133

@@ -127,5 +136,15 @@ that the production exporter and audit emit the series these rules read: zero
127136
counter baselines before the first audit, a `0` timestamp before any success,
128137
the last success kept through failures past the idle timeout, and run counters
129138
that outlive an evicted timestamp. They do not
130-
evaluate the rules, their `for` durations, or label matching; validate those
131-
with `promtool test rules` against your alerting configuration.
139+
evaluate the rules, their `for` durations, or label matching.
140+
`partition-catalog-monitoring.rules.test.yml` evaluates the rules in
141+
`partition-catalog-monitoring.rules.yml`, including their `for` durations,
142+
against a fresh pod, a stopped loop, a terminated pod, a table that never
143+
succeeded, and repeated failures. CI does not run it. Keep the rules file
144+
identical to the block above, and run it after changing either:
145+
146+
```bash
147+
promtool test rules docs/partition-catalog-monitoring.rules.test.yml
148+
```
149+
150+
Then validate label matching against your own alerting configuration.
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Run with: promtool test rules docs/partition-catalog-monitoring.rules.test.yml
2+
#
3+
# Series are sampled every minute and use the default 900 s audit interval. The
4+
# startup audit finishes before the first scrape, so run counters start at 1.
5+
rule_files:
6+
- partition-catalog-monitoring.rules.yml
7+
8+
evaluation_interval: 1m
9+
10+
tests:
11+
# A healthy new pod audits at startup and then every 15 minutes. Its counter
12+
# is flat for the first interval, which must not read as a stopped loop.
13+
- interval: 1m
14+
input_series:
15+
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="ok"}'
16+
values: '1x14 2x14 3x14 4x14 5x14'
17+
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="degraded"}'
18+
values: '0x74'
19+
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="fresh",table="events"}'
20+
values: '0x14 900x14 1800x14 2700x14 3600x14'
21+
alert_rule_test:
22+
- eval_time: 10m
23+
alertname: BuzzPartitionAuditNotRunning
24+
exp_alerts: []
25+
- eval_time: 14m
26+
alertname: BuzzPartitionAuditNotRunning
27+
exp_alerts: []
28+
- eval_time: 29m
29+
alertname: BuzzPartitionAuditNotRunning
30+
exp_alerts: []
31+
- eval_time: 70m
32+
alertname: BuzzPartitionAuditNotRunning
33+
exp_alerts: []
34+
- eval_time: 70m
35+
alertname: BuzzPartitionAuditStale
36+
exp_alerts: []
37+
38+
# The audit loop stops after the 15-minute audit while the pod keeps running.
39+
# Both rules fire before the timestamp's 2,700 s idle eviction at 60 minutes.
40+
- interval: 1m
41+
input_series:
42+
- series: 'buzz_partition_audit_runs_total{pod="stopped",table="events",outcome="ok"}'
43+
values: '1x14 2x60'
44+
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="stopped",table="events"}'
45+
values: '0x14 900x45'
46+
alert_rule_test:
47+
- eval_time: 40m
48+
alertname: BuzzPartitionAuditNotRunning
49+
exp_alerts: []
50+
- eval_time: 52m
51+
alertname: BuzzPartitionAuditNotRunning
52+
exp_alerts:
53+
- exp_labels: {pod: stopped, table: events}
54+
- eval_time: 57m
55+
alertname: BuzzPartitionAuditStale
56+
exp_alerts:
57+
- exp_labels: {pod: stopped, table: events}
58+
59+
# The audit loop stops right after the startup audit. It is caught once the
60+
# pod is one window old.
61+
- interval: 1m
62+
input_series:
63+
- series: 'buzz_partition_audit_runs_total{pod="early",table="events",outcome="ok"}'
64+
values: '1x80'
65+
alert_rule_test:
66+
- eval_time: 33m
67+
alertname: BuzzPartitionAuditNotRunning
68+
exp_alerts: []
69+
- eval_time: 36m
70+
alertname: BuzzPartitionAuditNotRunning
71+
exp_alerts:
72+
- exp_labels: {pod: early, table: events}
73+
74+
# A pod terminates at 20 minutes. Range selectors still read its earlier
75+
# samples, but it must not start an alert.
76+
- interval: 1m
77+
input_series:
78+
- series: 'buzz_partition_audit_runs_total{pod="gone",table="events",outcome="ok"}'
79+
values: '1x14 2x5 stale'
80+
- series: 'buzz_partition_audit_failures_total{pod="gone"}'
81+
values: '0x10 1 2 3x7 stale'
82+
alert_rule_test:
83+
- eval_time: 45m
84+
alertname: BuzzPartitionAuditNotRunning
85+
exp_alerts: []
86+
- eval_time: 60m
87+
alertname: BuzzPartitionAuditNotRunning
88+
exp_alerts: []
89+
- eval_time: 30m
90+
alertname: BuzzPartitionAuditFailing
91+
exp_alerts: []
92+
93+
# A table that has never audited successfully exports a 0 timestamp.
94+
- interval: 1m
95+
input_series:
96+
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="never",table="events"}'
97+
values: '0x60'
98+
alert_rule_test:
99+
- eval_time: 35m
100+
alertname: BuzzPartitionAuditStale
101+
exp_alerts: []
102+
- eval_time: 45m
103+
alertname: BuzzPartitionAuditStale
104+
exp_alerts:
105+
- exp_labels: {pod: never, table: events}
106+
107+
# One failure does not alert; a second within an hour does.
108+
- interval: 1m
109+
input_series:
110+
- series: 'buzz_partition_audit_failures_total{pod="once"}'
111+
values: '0x10 1x50'
112+
- series: 'buzz_partition_audit_failures_total{pod="twice"}'
113+
values: '0x10 1x10 2x40'
114+
alert_rule_test:
115+
- eval_time: 40m
116+
alertname: BuzzPartitionAuditFailing
117+
exp_alerts:
118+
- exp_labels: {pod: twice}
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Alert rules from docs/partition-catalog-monitoring.md. Keep them identical to
2+
# the doc; partition-catalog-monitoring.rules.test.yml tests them.
3+
groups:
4+
- name: buzz-partition-catalog
5+
rules:
6+
- alert: BuzzPartitionAuditStale
7+
expr: time() - buzz_partition_audit_last_success_timestamp_seconds > 1800
8+
for: 10m
9+
10+
- alert: BuzzPartitionAuditNotRunning
11+
expr: |
12+
sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
13+
and sum without (outcome) (buzz_partition_audit_runs_total)
14+
and sum without (outcome) (buzz_partition_audit_runs_total offset 30m)
15+
for: 5m
16+
17+
- alert: BuzzPartitionAuditFailing
18+
expr: |
19+
increase(buzz_partition_audit_failures_total[1h]) >= 2
20+
and buzz_partition_audit_failures_total

0 commit comments

Comments
 (0)