Repository navigation
Commit 6d07a29
perf(relay): bound fleet usage metrics collection (#7176)
## Why
The usage poller pays memory and query costs proportional to community
count even when per-community emission is disabled. Every pod loads the
full host map, and the leader materializes all per-community aggregates
before suppressing their labels. That makes telemetry collection itself
a scaling constraint.
## What changed
- Fleet-only usage metrics are now the default. `all` remains available
as an explicit rollback mode.
- In fleet-only mode, the poller skips the community host map and all
community-sized query results.
- Exact community counting is replaced with the PostgreSQL planner's
table-size estimate.
- Stock totals are collected as one fixed row, scanning each stock table
once per hour.
- 1d/7d/30d active-user totals come from one bounded 30-day scan per
day.
- Fleet database snapshots run only on a proved read replica, with a
telemetry-specific freshness budget and server-side statement timeouts.
If no such replica is available, the collection is skipped; it never
falls back to the writer.
- Successful fixed-row snapshots are cached and re-emitted on each poll
tick, so gauges stay scrape-visible without raising query frequency.
- `buzz_usage_snapshot_available{family}` reports whether the latest due
refresh of that family succeeded. After a skipped or failed refresh, the
last values stay visible (with `buzz_usage_snapshot_age_seconds`) but
the family reads unavailable until a 60-second retry succeeds.
- The `all` rollback collector publishes the same availability gauges,
so dashboards gated on availability keep working during a rollback.
- Each due refresh outcome is applied by `apply_stock_refresh` /
`apply_activity_refresh`, the single production mapping from `Ok(Some)`
/ `Ok(None)` / `Err` to cache, availability, skip counter, and retry;
the regression test drives that mapping directly.
- `buzz_usage_snapshot_age_seconds` is emitted in the `all` rollback
too, so age-based alerts keep firing through a rollback.
- Every fleet telemetry attempt that completes lands in
`buzz_db_route_decision`; a query error on a proved reader records
`skipped/replica_error`. An attempt abandoned at the relay deadline
records no route decision and appears only as
`buzz_usage_query_skipped_total{reason="timeout"}`. The shared
`proved_reader` warnings no longer claim "routing to writer", because
the telemetry caller never does (serving callers still record their
writer decision in the route counter).
- The PostgreSQL production-method matrix now exercises
`usage_fleet_stock_snapshot`, so the new `reader/maintenance` pool pair
is proven to be emitted by a real Db method.
- Each fleet collection, reader proof included, runs under a relay-side
`tokio::time::timeout` (10s stock, 20s activity, both above their
server-side `statement_timeout`). A reader that stops answering expires
into the same failure mapping with
`buzz_usage_query_skipped_total{reason="timeout"}`, so it cannot stall
the leader's poller, hold the advisory lock past a liveness check, or
leave a frozen snapshot reporting available.
- Fleet telemetry reader checkouts (`ReaderOperation::Maintenance`) opt
into SQLx `close_on_drop` before their first post-acquire await, so an
abandoned collection's connection is closed within SQLx's 5s
`CLOSE_ON_DROP_TIMEOUT` instead of going through the unbounded
return-to-pool ping while it holds a slot in the reader pool that
serving reads share. Serving reads keep pooled reuse; the cost is one
reader reconnect per fleet refresh.
- On leader demotion, cached snapshots and availability are cleared, and
a fresh collection is forced after the lock is reacquired.
- Fleet-only mode passes no host map to the worker-snapshot storage
emitter. It therefore omits community attribution instead of reporting
every byte as `buzz_storage_unmapped_community_bytes`.
- Lifetime stored-message and active-channel totals are no longer
queried in fleet-only mode. Event throughput remains available through
the existing stored-event counter.
- No test-selector changes: main's `config::tests::` selector in `just
test-unit` and the `run-tests.sh unit` fallback already runs the new
replica-budget config test, and the new fleet schedule, availability,
and storage tests run through the existing bin `tests::` and
`storage_sweep::tests::` selectors.
- Added seeded PostgreSQL contract tests and operator docs covering
rollout, rollback, availability, the no-replica behavior, and leader
election.
No migrations, tables, indexes, or new persistent storage are added.
### What got simpler
The relay-side storage half of the original branch (totals-only S3
listing and `buzz_storage_community_breakdown_available`) was dropped.
Main moved S3 listing out of the relay in #7543 and #7845, so that code
was dead on arrival. The rework adds no storage code path; it only gates
attribution on the presence of a host map.
## Risk
Medium. The default changes which telemetry is emitted:
- In fleet-only mode, `buzz_communities_estimated` replaces the exact
`buzz_communities_total`.
- `buzz_total_messages` and `buzz_total_active_channels` are not
refreshed in fleet-only mode (also called out in the rollout steps of
`docs/usage-metrics.md`).
- Deployments without a verified reader (including local compose and
single-node self-hosts) skip database-derived fleet snapshots instead of
querying the writer, and report both families unavailable.
- Multi-pod dashboards must filter database and storage series to
`buzz_usage_poller_is_leader == 1`.
Setting `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` restores the legacy
collector for temporary rollback.
## Verification
At head `1879ee3aedf4227942120a3b521c8d35fc7839fe` (tree `78477e74`;
merges main `da651ad6`; the PR's diff against main is byte-identical to
`1613823f`'s, minus the Justfile and `run-tests.sh` selector lines that
main's `config::tests::` selector now subsumes):
- On Blox: `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::`
34/34, `config::tests::` 55/55, `storage_sweep::tests::` 10/10 and
`store::usage::tests` pass, as do the four `fleet_` and lock PostgreSQL
tests that use scratch databases, including
`abandoned_fleet_collection_releases_its_reader_slot`.
- Against a freshly migrated PostgreSQL database:
`store::usage::postgres_tests::` 14/14,
`storage_sweep::postgres_tests::` and `buzz-db --test
observability_source` 14/14 pass. The new reader code passes main's
crate-wide typed-acquisition and serving-table policy checks.
- E2E at `1613823f` is reused for this head. It ran in an isolated relay
against a real PostgreSQL primary and streaming standby. In 2/2 runs a
stock read went dark mid-result; the deadline abandoned it at about 10s,
the reader pool's active and size counts dropped to zero about 1s later,
and a fresh activity read and the scheduled stock retry succeeded on new
backends while the dark session stayed open. The `all`-mode and
healthy-replica controls also passed. Main's merged changes are on the
writer path and in startup step timing, and touch none of the reader
checkout, the `close_on_drop` seam, the fleet poller or the deadlines.
At head `1613823fa8ef1a423bc1a499c57bc40d692fcd4f` (tree `12bc6d1b`;
test/docs-only on top of `57749418`), on a Blox workstation:
- `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::` 34/34
pass.
- New PostgreSQL test
`abandoned_fleet_collection_releases_its_reader_slot`: a one-slot reader
pool behind a loopback proxy that goes dark mid-session; the collection
is cut at a deadline, then a later acquire must succeed in under 8s on a
fresh connection. The dark call runs on a fresh `Db` with a cold Aurora
capability cache, so the first post-acquire await is the capability
probe. It passes in 3.4s. With `close_on_drop` removed, or moved below
the probe await, it fails with `PoolTimedOut` after the 15s acquire
timeout. The other three `fleet_` PostgreSQL tests that use scratch
databases pass.
- `each_fleet_family_is_abandoned_at_its_own_deadline` now has an outer
timeout: replacing the stock deadline with a bare `.await` fails it
(`Elapsed`) instead of hanging.
At head `fd194ec79d6c6796d6764564de225ad56485cfc7` (tree `79e94727`,
main merged at `932228936`), on a Blox workstation:
- `cargo fmt --all -- --check` and `cargo clippy -p buzz-relay -p
buzz-db --all-targets -- -D warnings` pass.
- Relay bin `tests::` nextest selection: 27/27 pass (plus
`store::usage::tests`), including
`fleet_refresh_outcomes_drive_availability_through_production_mapping`
and the new `stalled_fleet_collection_times_out_into_failed_refresh`
(paused time: two never-completing collections are abandoned at exactly
10s + 20s; both families read unavailable with cached values kept,
`reason="timeout"` skips, retry scheduled; the next tick runs and
recovers; no route decision is recorded for an abandoned attempt) and
`each_fleet_family_is_abandoned_at_its_own_deadline`. Compile-time
asserts keep each deadline above its `statement_timeout`.
- Falsifiability: replacing the stock deadline with a bare `.await`
makes the new test fail (the tick stalls); swapping the two deadlines
fails the per-family test; deleting `schedule.mark_stock_failure(now)`
from the `Ok(None)` arm fails the mapping test. Trees restored and
re-verified.
- At `970414db`: CI PostgreSQL Tests 809/809 including the
production-method matrix with `reader/maintenance`; Unit Tests ran
`finish_usage_read_records_every_attempt_in_route_decisions`.
- `just ci` / `just test` at `5b18914a` on Blox: not green for
environmental reasons (`mobile-check` could not reach pub.dev;
`_ensure-services` failed so service-backed suites did not run;
`buzz-acp git_bootstrap` fails identically on `origin/main`). PostgreSQL
and integration coverage comes from GitHub CI at this head.
E2E at the previous head `0a01e5f22` (isolated relay, real Postgres
primary plus streaming replica, worker storage snapshot, desktop
frontend in relay mode): with no reader, both families report
skipped/unavailable and route `skipped/disabled` (no writer queries);
fleet-only storage emits totals without `buzz_community_storage_*` or
unmapped bytes; `all` mode reports both availability gauges `1` with
per-community storage series; with a fresh replica, both fleet paths
route `replica/fresh` on `reader/maintenance` acquisitions; desktop
send/receive works. The cached success → due failure → recovery
transition was not reached live (hourly/daily cadence); it is covered by
the seam test above.
The original branch also recorded live verification at `aa3e7023`,
before this rework: reduced-idle cache re-emission, two-relay leader
takeover with demoted availability clearing, the no-reader skip with
60-second retry, `all` rollback restoring per-community series, and a
desktop smoke test. Those runs predate the availability and storage
changes and are not claimed for this head.
## Linear tracking
Related work: BUZZ-167.
---------
Signed-off-by: tornquist <[email protected]>
Signed-off-by: Luke Tornquist <[email protected]>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>1 parent c96d707 commit 6d07a29
11 files changed
Lines changed: 1976 additions & 89 deletions
File tree
- crates
- buzz-db/src
- runtime
- store
- buzz-relay/src
- storage_sweep
- deploy/charts/buzz
- docs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
30 | 40 | | |
31 | 41 | | |
32 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
219 | 219 | | |
220 | 220 | | |
221 | 221 | | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
222 | 226 | | |
223 | 227 | | |
224 | 228 | | |
| |||
598 | 602 | | |
599 | 603 | | |
600 | 604 | | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
601 | 609 | | |
602 | 610 | | |
603 | 611 | | |
| |||
628 | 636 | | |
629 | 637 | | |
630 | 638 | | |
| 639 | + | |
631 | 640 | | |
632 | 641 | | |
633 | 642 | | |
| |||
690 | 699 | | |
691 | 700 | | |
692 | 701 | | |
| 702 | + | |
| 703 | + | |
693 | 704 | | |
694 | 705 | | |
695 | 706 | | |
696 | 707 | | |
697 | 708 | | |
698 | 709 | | |
699 | 710 | | |
| 711 | + | |
700 | 712 | | |
701 | 713 | | |
702 | 714 | | |
| |||
931 | 943 | | |
932 | 944 | | |
933 | 945 | | |
| 946 | + | |
934 | 947 | | |
935 | 948 | | |
936 | 949 | | |
| |||
950 | 963 | | |
951 | 964 | | |
952 | 965 | | |
| 966 | + | |
953 | 967 | | |
954 | 968 | | |
955 | 969 | | |
| |||
1055 | 1069 | | |
1056 | 1070 | | |
1057 | 1071 | | |
1058 | | - | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
1062 | | - | |
1063 | | - | |
1064 | | - | |
1065 | | - | |
1066 | | - | |
1067 | | - | |
1068 | | - | |
1069 | | - | |
1070 | | - | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
1071 | 1084 | | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
1072 | 1090 | | |
1073 | 1091 | | |
1074 | 1092 | | |
| |||
1103 | 1121 | | |
1104 | 1122 | | |
1105 | 1123 | | |
1106 | | - | |
| 1124 | + | |
1107 | 1125 | | |
1108 | 1126 | | |
1109 | 1127 | | |
1110 | | - | |
| 1128 | + | |
1111 | 1129 | | |
1112 | 1130 | | |
1113 | 1131 | | |
1114 | 1132 | | |
1115 | 1133 | | |
1116 | 1134 | | |
1117 | 1135 | | |
1118 | | - | |
| 1136 | + | |
1119 | 1137 | | |
1120 | 1138 | | |
1121 | 1139 | | |
| |||
1368 | 1386 | | |
1369 | 1387 | | |
1370 | 1388 | | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
1371 | 1422 | | |
1372 | 1423 | | |
1373 | 1424 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
76 | 77 | | |
77 | 78 | | |
78 | 79 | | |
| 80 | + | |
| 81 | + | |
79 | 82 | | |
80 | 83 | | |
81 | 84 | | |
82 | 85 | | |
83 | | - | |
| 86 | + | |
84 | 87 | | |
85 | 88 | | |
86 | 89 | | |
| 90 | + | |
87 | 91 | | |
88 | 92 | | |
89 | 93 | | |
90 | 94 | | |
91 | 95 | | |
92 | 96 | | |
93 | 97 | | |
| 98 | + | |
94 | 99 | | |
95 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
96 | 114 | | |
97 | 115 | | |
98 | 116 | | |
99 | | - | |
| 117 | + | |
100 | 118 | | |
101 | 119 | | |
102 | 120 | | |
| |||
108 | 126 | | |
109 | 127 | | |
110 | 128 | | |
| 129 | + | |
111 | 130 | | |
112 | 131 | | |
113 | 132 | | |
114 | 133 | | |
115 | | - | |
116 | | - | |
117 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
118 | 138 | | |
119 | 139 | | |
120 | 140 | | |
| |||
130 | 150 | | |
131 | 151 | | |
132 | 152 | | |
133 | | - | |
| 153 | + | |
134 | 154 | | |
135 | 155 | | |
136 | 156 | | |
| |||
139 | 159 | | |
140 | 160 | | |
141 | 161 | | |
142 | | - | |
| 162 | + | |
143 | 163 | | |
144 | 164 | | |
145 | 165 | | |
| |||
151 | 171 | | |
152 | 172 | | |
153 | 173 | | |
| 174 | + | |
154 | 175 | | |
155 | 176 | | |
156 | | - | |
| 177 | + | |
157 | 178 | | |
158 | 179 | | |
159 | 180 | | |
| |||
610 | 631 | | |
611 | 632 | | |
612 | 633 | | |
| 634 | + | |
613 | 635 | | |
614 | 636 | | |
615 | | - | |
| 637 | + | |
616 | 638 | | |
617 | 639 | | |
618 | 640 | | |
| |||
1494 | 1516 | | |
1495 | 1517 | | |
1496 | 1518 | | |
| 1519 | + | |
1497 | 1520 | | |
1498 | 1521 | | |
1499 | 1522 | | |
| |||
0 commit comments