Repository navigation
Commit c3a314e
fix(relay): export partition audit counters at zero on startup
Prometheus increase() ignores the first sample of a new series. The audit
run and failure counters were created only when an outcome first occurred,
so the first periodic failure read as no increase, and a table's first run
with a new outcome could look like a stopped audit loop to the documented
alerts.
Register every buzz_partition_audit_runs_total{table, outcome} series and
buzz_partition_audit_failures_total at 0 when the metrics exporter starts,
before the startup audit, following the existing auth series baselines.
Document the remaining gap: a failure in the startup audit itself can land
before the first scrape, and the stale rule covers it.
Also gate the two range-based alert rules on the counter being exported
now. Range selectors keep reading a terminated pod's samples until they
leave the window, so BuzzPartitionAuditNotRunning could fire for a pod that
was replaced during normal turnover, and BuzzPartitionAuditFailing could
keep firing for up to an hour after it. The counters now exist for the
whole life of a relay, so `and <counter>` limits both rules to live pods.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Ian Oberst <[email protected]>1 parent 1d91e38 commit c3a314e
3 files changed
Lines changed: 96 additions & 14 deletions
File tree
- crates
- buzz-db/src/store
- buzz-relay/src
- docs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1093 | 1093 | | |
1094 | 1094 | | |
1095 | 1095 | | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
1096 | 1114 | | |
1097 | 1115 | | |
1098 | 1116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
298 | 298 | | |
299 | 299 | | |
300 | 300 | | |
| 301 | + | |
301 | 302 | | |
302 | 303 | | |
303 | 304 | | |
| |||
521 | 522 | | |
522 | 523 | | |
523 | 524 | | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
524 | 532 | | |
525 | 533 | | |
526 | 534 | | |
| |||
1232 | 1240 | | |
1233 | 1241 | | |
1234 | 1242 | | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
1235 | 1283 | | |
1236 | 1284 | | |
1237 | 1285 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
54 | 57 | | |
55 | 58 | | |
56 | 59 | | |
57 | | - | |
58 | | - | |
| 60 | + | |
59 | 61 | | |
60 | 62 | | |
61 | 63 | | |
| |||
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
69 | | - | |
70 | | - | |
71 | | - | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
72 | 79 | | |
73 | 80 | | |
74 | 81 | | |
| |||
84 | 91 | | |
85 | 92 | | |
86 | 93 | | |
87 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
88 | 97 | | |
89 | 98 | | |
90 | 99 | | |
91 | 100 | | |
92 | | - | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
93 | 104 | | |
94 | 105 | | |
95 | | - | |
96 | | - | |
97 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
98 | 113 | | |
99 | 114 | | |
100 | 115 | | |
| |||
105 | 120 | | |
106 | 121 | | |
107 | 122 | | |
108 | | - | |
109 | | - | |
110 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
111 | 127 | | |
112 | 128 | | |
0 commit comments