Skip to content

Commit 6d07a29

Browse files
TheSentinel454Elrond
andauthored
perf(relay): bound fleet usage metrics collection (#7176)
## Why The usage poller pays memory and query costs proportional to community count even when per-community emission is disabled. Every pod loads the full host map, and the leader materializes all per-community aggregates before suppressing their labels. That makes telemetry collection itself a scaling constraint. ## What changed - Fleet-only usage metrics are now the default. `all` remains available as an explicit rollback mode. - In fleet-only mode, the poller skips the community host map and all community-sized query results. - Exact community counting is replaced with the PostgreSQL planner's table-size estimate. - Stock totals are collected as one fixed row, scanning each stock table once per hour. - 1d/7d/30d active-user totals come from one bounded 30-day scan per day. - Fleet database snapshots run only on a proved read replica, with a telemetry-specific freshness budget and server-side statement timeouts. If no such replica is available, the collection is skipped; it never falls back to the writer. - Successful fixed-row snapshots are cached and re-emitted on each poll tick, so gauges stay scrape-visible without raising query frequency. - `buzz_usage_snapshot_available{family}` reports whether the latest due refresh of that family succeeded. After a skipped or failed refresh, the last values stay visible (with `buzz_usage_snapshot_age_seconds`) but the family reads unavailable until a 60-second retry succeeds. - The `all` rollback collector publishes the same availability gauges, so dashboards gated on availability keep working during a rollback. - Each due refresh outcome is applied by `apply_stock_refresh` / `apply_activity_refresh`, the single production mapping from `Ok(Some)` / `Ok(None)` / `Err` to cache, availability, skip counter, and retry; the regression test drives that mapping directly. - `buzz_usage_snapshot_age_seconds` is emitted in the `all` rollback too, so age-based alerts keep firing through a rollback. - Every fleet telemetry attempt that completes lands in `buzz_db_route_decision`; a query error on a proved reader records `skipped/replica_error`. An attempt abandoned at the relay deadline records no route decision and appears only as `buzz_usage_query_skipped_total{reason="timeout"}`. The shared `proved_reader` warnings no longer claim "routing to writer", because the telemetry caller never does (serving callers still record their writer decision in the route counter). - The PostgreSQL production-method matrix now exercises `usage_fleet_stock_snapshot`, so the new `reader/maintenance` pool pair is proven to be emitted by a real Db method. - Each fleet collection, reader proof included, runs under a relay-side `tokio::time::timeout` (10s stock, 20s activity, both above their server-side `statement_timeout`). A reader that stops answering expires into the same failure mapping with `buzz_usage_query_skipped_total{reason="timeout"}`, so it cannot stall the leader's poller, hold the advisory lock past a liveness check, or leave a frozen snapshot reporting available. - Fleet telemetry reader checkouts (`ReaderOperation::Maintenance`) opt into SQLx `close_on_drop` before their first post-acquire await, so an abandoned collection's connection is closed within SQLx's 5s `CLOSE_ON_DROP_TIMEOUT` instead of going through the unbounded return-to-pool ping while it holds a slot in the reader pool that serving reads share. Serving reads keep pooled reuse; the cost is one reader reconnect per fleet refresh. - On leader demotion, cached snapshots and availability are cleared, and a fresh collection is forced after the lock is reacquired. - Fleet-only mode passes no host map to the worker-snapshot storage emitter. It therefore omits community attribution instead of reporting every byte as `buzz_storage_unmapped_community_bytes`. - Lifetime stored-message and active-channel totals are no longer queried in fleet-only mode. Event throughput remains available through the existing stored-event counter. - No test-selector changes: main's `config::tests::` selector in `just test-unit` and the `run-tests.sh unit` fallback already runs the new replica-budget config test, and the new fleet schedule, availability, and storage tests run through the existing bin `tests::` and `storage_sweep::tests::` selectors. - Added seeded PostgreSQL contract tests and operator docs covering rollout, rollback, availability, the no-replica behavior, and leader election. No migrations, tables, indexes, or new persistent storage are added. ### What got simpler The relay-side storage half of the original branch (totals-only S3 listing and `buzz_storage_community_breakdown_available`) was dropped. Main moved S3 listing out of the relay in #7543 and #7845, so that code was dead on arrival. The rework adds no storage code path; it only gates attribution on the presence of a host map. ## Risk Medium. The default changes which telemetry is emitted: - In fleet-only mode, `buzz_communities_estimated` replaces the exact `buzz_communities_total`. - `buzz_total_messages` and `buzz_total_active_channels` are not refreshed in fleet-only mode (also called out in the rollout steps of `docs/usage-metrics.md`). - Deployments without a verified reader (including local compose and single-node self-hosts) skip database-derived fleet snapshots instead of querying the writer, and report both families unavailable. - Multi-pod dashboards must filter database and storage series to `buzz_usage_poller_is_leader == 1`. Setting `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` restores the legacy collector for temporary rollback. ## Verification At head `1879ee3aedf4227942120a3b521c8d35fc7839fe` (tree `78477e74`; merges main `da651ad6`; the PR's diff against main is byte-identical to `1613823f`'s, minus the Justfile and `run-tests.sh` selector lines that main's `config::tests::` selector now subsumes): - On Blox: `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::` 34/34, `config::tests::` 55/55, `storage_sweep::tests::` 10/10 and `store::usage::tests` pass, as do the four `fleet_` and lock PostgreSQL tests that use scratch databases, including `abandoned_fleet_collection_releases_its_reader_slot`. - Against a freshly migrated PostgreSQL database: `store::usage::postgres_tests::` 14/14, `storage_sweep::postgres_tests::` and `buzz-db --test observability_source` 14/14 pass. The new reader code passes main's crate-wide typed-acquisition and serving-table policy checks. - E2E at `1613823f` is reused for this head. It ran in an isolated relay against a real PostgreSQL primary and streaming standby. In 2/2 runs a stock read went dark mid-result; the deadline abandoned it at about 10s, the reader pool's active and size counts dropped to zero about 1s later, and a fresh activity read and the scheduled stock retry succeeded on new backends while the dark session stayed open. The `all`-mode and healthy-replica controls also passed. Main's merged changes are on the writer path and in startup step timing, and touch none of the reader checkout, the `close_on_drop` seam, the fleet poller or the deadlines. At head `1613823fa8ef1a423bc1a499c57bc40d692fcd4f` (tree `12bc6d1b`; test/docs-only on top of `57749418`), on a Blox workstation: - `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::` 34/34 pass. - New PostgreSQL test `abandoned_fleet_collection_releases_its_reader_slot`: a one-slot reader pool behind a loopback proxy that goes dark mid-session; the collection is cut at a deadline, then a later acquire must succeed in under 8s on a fresh connection. The dark call runs on a fresh `Db` with a cold Aurora capability cache, so the first post-acquire await is the capability probe. It passes in 3.4s. With `close_on_drop` removed, or moved below the probe await, it fails with `PoolTimedOut` after the 15s acquire timeout. The other three `fleet_` PostgreSQL tests that use scratch databases pass. - `each_fleet_family_is_abandoned_at_its_own_deadline` now has an outer timeout: replacing the stock deadline with a bare `.await` fails it (`Elapsed`) instead of hanging. At head `fd194ec79d6c6796d6764564de225ad56485cfc7` (tree `79e94727`, main merged at `932228936`), on a Blox workstation: - `cargo fmt --all -- --check` and `cargo clippy -p buzz-relay -p buzz-db --all-targets -- -D warnings` pass. - Relay bin `tests::` nextest selection: 27/27 pass (plus `store::usage::tests`), including `fleet_refresh_outcomes_drive_availability_through_production_mapping` and the new `stalled_fleet_collection_times_out_into_failed_refresh` (paused time: two never-completing collections are abandoned at exactly 10s + 20s; both families read unavailable with cached values kept, `reason="timeout"` skips, retry scheduled; the next tick runs and recovers; no route decision is recorded for an abandoned attempt) and `each_fleet_family_is_abandoned_at_its_own_deadline`. Compile-time asserts keep each deadline above its `statement_timeout`. - Falsifiability: replacing the stock deadline with a bare `.await` makes the new test fail (the tick stalls); swapping the two deadlines fails the per-family test; deleting `schedule.mark_stock_failure(now)` from the `Ok(None)` arm fails the mapping test. Trees restored and re-verified. - At `970414db`: CI PostgreSQL Tests 809/809 including the production-method matrix with `reader/maintenance`; Unit Tests ran `finish_usage_read_records_every_attempt_in_route_decisions`. - `just ci` / `just test` at `5b18914a` on Blox: not green for environmental reasons (`mobile-check` could not reach pub.dev; `_ensure-services` failed so service-backed suites did not run; `buzz-acp git_bootstrap` fails identically on `origin/main`). PostgreSQL and integration coverage comes from GitHub CI at this head. E2E at the previous head `0a01e5f22` (isolated relay, real Postgres primary plus streaming replica, worker storage snapshot, desktop frontend in relay mode): with no reader, both families report skipped/unavailable and route `skipped/disabled` (no writer queries); fleet-only storage emits totals without `buzz_community_storage_*` or unmapped bytes; `all` mode reports both availability gauges `1` with per-community storage series; with a fresh replica, both fleet paths route `replica/fresh` on `reader/maintenance` acquisitions; desktop send/receive works. The cached success → due failure → recovery transition was not reached live (hourly/daily cadence); it is covered by the seam test above. The original branch also recorded live verification at `aa3e7023`, before this rework: reduced-idle cache re-emission, two-relay leader takeover with demoted availability clearing, the no-reader skip with 60-second retry, `all` rollback restoring per-community series, and a desktop smoke test. Those runs predate the availability and storage changes and are not claimed for this head. ## Linear tracking Related work: BUZZ-167. --------- Signed-off-by: tornquist <[email protected]> Signed-off-by: Luke Tornquist <[email protected]> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent c96d707 commit 6d07a29

11 files changed

Lines changed: 1976 additions & 89 deletions

File tree

‎.env.example‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ PGUSER=buzz
2727
PGPASSWORD=buzz_dev
2828
PGDATABASE=buzz
2929

30+
# Fleet usage telemetry is replica-only and independent of serving-read
31+
# routing. This freshness budget defaults to 30000ms; 0 disables the database
32+
# telemetry families. When the reader is absent, stale, or unavailable, the
33+
# relay skips the collection and reports its availability as 0 instead of
34+
# falling back to the writer.
35+
# BUZZ_USAGE_METRICS_REPLICA_MAX_AGE_MS=30000
36+
# Per-community metric series are disabled by default to bound relay memory and
37+
# monitoring cardinality. `all` is the temporary rollback mode.
38+
# BUZZ_USAGE_METRICS_PER_COMMUNITY=off
39+
3040
# -----------------------------------------------------------------------------
3141
# Redis 7
3242
# -----------------------------------------------------------------------------

‎crates/buzz-db/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ pub use runtime::{
3636
};
3737

3838
/// Valid low-cardinality `(pool_role, operation)` pairs for pool-acquisition telemetry.
39-
pub const DB_POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 11] =
39+
pub const DB_POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 12] =
4040
runtime::observability::POOL_ACQUIRE_VALID_PAIRS;
4141

4242
/// Raw Prometheus series ceiling per relay pod for the operation-aware contract.

‎crates/buzz-db/src/runtime/mod.rs‎

Lines changed: 67 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,10 @@ pub struct Db {
219219
/// bounded-stale read semantics are a product decision, not an
220220
/// invariant, so the gate ships off.
221221
pub(crate) replica_read_max_age: Option<Duration>,
222+
/// Replica freshness budget used only by the leader's fleet telemetry
223+
/// queries. This is independent of serving-read rollout policy: telemetry
224+
/// may be stale or skipped, but it must never fall back to the writer.
225+
pub(crate) usage_metrics_replica_max_age: Option<Duration>,
222226
/// Whether the reader endpoint supports the Aurora PostgreSQL identity
223227
/// function ([`replica_fence::AURORA_IDENTITY_FN`]) — probed
224228
/// once per process on the first routed read (on a plain autocommit
@@ -598,6 +602,10 @@ pub struct DbConfig {
598602
/// than the staleness gate never routes anyway, so a larger budget
599603
/// would only misrepresent the config.
600604
pub replica_read_max_age_ms: u64,
605+
/// Replica freshness budget for usage telemetry, in milliseconds
606+
/// (`BUZZ_USAGE_METRICS_REPLICA_MAX_AGE_MS`). `0` disables fleet database
607+
/// telemetry. This does not enable replica routing for serving reads.
608+
pub usage_metrics_replica_max_age_ms: u64,
601609
/// Session `lock_timeout` in milliseconds for writer connections (env
602610
/// `BUZZ_DB_LOCK_TIMEOUT_MS`). `0` disables the timeout.
603611
pub lock_timeout_ms: u64,
@@ -628,6 +636,7 @@ impl Default for DbConfig {
628636
max_lifetime_secs: 1800,
629637
idle_timeout_secs: 600,
630638
replica_read_max_age_ms: 0,
639+
usage_metrics_replica_max_age_ms: 30_000,
631640
lock_timeout_ms: DEFAULT_LOCK_TIMEOUT_MS,
632641
idle_txn_timeout_ms: DEFAULT_IDLE_TXN_TIMEOUT_MS,
633642
statement_timeout_ms: 0,
@@ -690,13 +699,16 @@ impl Db {
690699
None => None,
691700
};
692701
let replica_read_max_age = read_budget_from_ms(config.replica_read_max_age_ms);
702+
let usage_metrics_replica_max_age =
703+
read_budget_from_ms(config.usage_metrics_replica_max_age_ms);
693704
Ok(Self {
694705
pool,
695706
max_connections: config.max_connections,
696707
read_pool,
697708
read_max_connections,
698709
fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()),
699710
replica_read_max_age,
711+
usage_metrics_replica_max_age,
700712
reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()),
701713
})
702714
}
@@ -931,6 +943,7 @@ impl Db {
931943
read_pool: None,
932944
fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()),
933945
replica_read_max_age: None,
946+
usage_metrics_replica_max_age: None,
934947
reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()),
935948
}
936949
}
@@ -950,6 +963,7 @@ impl Db {
950963
read_pool: Some(read_pool),
951964
fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()),
952965
replica_read_max_age: None,
966+
usage_metrics_replica_max_age: Some(replica_fence::FENCE_STALENESS),
953967
reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()),
954968
}
955969
}
@@ -1055,20 +1069,24 @@ impl Db {
10551069
// `read_pool` separately would spend a second budget whenever the
10561070
// capability is uncached — i.e. after a failed boot ping, which is
10571071
// precisely the reader-unavailable case the bound must hold for.
1058-
let conn = match observability::acquire_reader_with_legacy_metrics(read_pool, operation)
1059-
.await
1060-
{
1061-
Ok(conn) => conn,
1062-
Err(sqlx::Error::PoolTimedOut) => {
1063-
tracing::warn!("reader pool acquire timed out; routing to writer");
1064-
return Err("reader_acquire_timeout");
1065-
}
1066-
Err(e) => {
1067-
tracing::warn!(error = %e, "reader connection acquire failed; routing to writer");
1068-
return Err("reader_validation_error");
1069-
}
1070-
};
1072+
let conn =
1073+
match observability::acquire_reader_with_legacy_metrics(read_pool, operation).await {
1074+
Ok(conn) => conn,
1075+
Err(sqlx::Error::PoolTimedOut) => {
1076+
tracing::warn!("reader pool acquire timed out");
1077+
return Err("reader_acquire_timeout");
1078+
}
1079+
Err(e) => {
1080+
tracing::warn!(error = %e, "reader connection acquire failed");
1081+
return Err("reader_validation_error");
1082+
}
1083+
};
10711084
let mut conn = conn;
1085+
if operation.bounded_release() {
1086+
// Must precede the first post-acquire await: a caller deadline
1087+
// can drop this checkout at any of them.
1088+
conn.close_on_drop();
1089+
}
10721090
let aurora = self.reader_aurora_capability_on(&mut conn).await;
10731091
let mut tx = match sqlx::Transaction::begin(
10741092
conn,
@@ -1103,19 +1121,19 @@ impl Db {
11031121
// between samples entirely, so absence of elevated active is
11041122
// NOT evidence of a cold connect.
11051123
Err(sqlx::Error::PoolTimedOut) => {
1106-
tracing::warn!("reader pool acquire timed out; routing to writer");
1124+
tracing::warn!("reader pool acquire timed out");
11071125
return Err("reader_acquire_timeout");
11081126
}
11091127
Err(e) => {
1110-
tracing::warn!(error = %e, "reader transaction begin failed; routing to writer");
1128+
tracing::warn!(error = %e, "reader transaction begin failed");
11111129
return Err("reader_validation_error");
11121130
}
11131131
};
11141132
let obs = match replica_fence::observe_heartbeat(&mut tx, aurora).await {
11151133
Ok(Some(observation)) => observation,
11161134
Ok(None) => return Err("reader_validation_error"),
11171135
Err(e) => {
1118-
tracing::warn!(error = %e, "heartbeat observation failed; routing to writer");
1136+
tracing::warn!(error = %e, "heartbeat observation failed");
11191137
return Err("reader_validation_error");
11201138
}
11211139
};
@@ -1368,6 +1386,39 @@ impl Db {
13681386
Ok(result)
13691387
}
13701388

1389+
/// Route a usage-telemetry query only to a proved, sufficiently fresh
1390+
/// reader. Unlike serving reads, an unavailable reader is recorded as
1391+
/// skipped and never represented as or executed on the writer.
1392+
pub(crate) async fn route_usage_read(
1393+
&self,
1394+
path: &'static str,
1395+
) -> Option<(sqlx::Transaction<'static, sqlx::Postgres>, &'static str)> {
1396+
let skip = |reason| {
1397+
Self::record_route(path, "skipped", reason);
1398+
None
1399+
};
1400+
let Some(read_pool) = &self.read_pool else {
1401+
return skip("disabled");
1402+
};
1403+
let Some(budget) = self.usage_metrics_replica_max_age else {
1404+
return skip("disabled");
1405+
};
1406+
let Some(newest) = self.fence.newest() else {
1407+
return skip("uninitialized");
1408+
};
1409+
if newest.committed_at.elapsed() > budget {
1410+
return skip("stale");
1411+
}
1412+
match self
1413+
.proved_reader(read_pool, observability::ReaderOperation::Maintenance)
1414+
.await
1415+
{
1416+
Ok((tx, entry)) if entry.committed_at.elapsed() <= budget => Some((tx, "fresh")),
1417+
Ok((_tx, _entry)) => skip("stale"),
1418+
Err(reason) => skip(reason),
1419+
}
1420+
}
1421+
13711422
/// Shared route decision for one read: evaluate the predicate against a
13721423
/// proved reader session and record the decision. Fail closed to the
13731424
/// writer everywhere.

‎crates/buzz-db/src/runtime/observability.rs‎

Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ enum PoolOperation {
2828
ReaderSubscriptionHistory,
2929
WriterEventWrite,
3030
WriterMaintenance,
31+
ReaderMaintenance,
3132
}
3233

3334
/// Writer-pool operations. Reader-only combinations cannot be constructed.
@@ -76,27 +77,44 @@ pub(crate) enum ReaderOperation {
7677
Bootstrap,
7778
Authorization,
7879
SubscriptionHistory,
80+
/// Leader-only background reads such as fleet usage telemetry.
81+
Maintenance,
7982
}
8083

8184
impl ReaderOperation {
8285
#[cfg(test)]
83-
const ALL: [Self; 3] = [
86+
const ALL: [Self; 4] = [
8487
Self::Bootstrap,
8588
Self::Authorization,
8689
Self::SubscriptionHistory,
90+
Self::Maintenance,
8791
];
8892

8993
const fn pair(self) -> PoolOperation {
9094
match self {
9195
Self::Bootstrap => PoolOperation::ReaderBootstrap,
9296
Self::Authorization => PoolOperation::ReaderAuthorization,
9397
Self::SubscriptionHistory => PoolOperation::ReaderSubscriptionHistory,
98+
Self::Maintenance => PoolOperation::ReaderMaintenance,
9499
}
95100
}
101+
102+
/// Whether a dropped checkout must close the connection under SQLx's
103+
/// bounded `close_on_drop` instead of returning it to the pool.
104+
///
105+
/// Maintenance reads run under relay-side deadlines that drop them
106+
/// mid-query when a reader goes dark. SQLx's return-to-pool path pings
107+
/// that connection with no timeout while still holding a pool slot, and
108+
/// the reader pool is shared with serving reads, so each abandoned read
109+
/// would strand a serving slot until the kernel gives up on the socket.
110+
/// Serving reads keep pooled reuse.
111+
pub(crate) const fn bounded_release(self) -> bool {
112+
matches!(self, Self::Maintenance)
113+
}
96114
}
97115

98116
impl PoolOperation {
99-
pub(crate) const ALL: [Self; 11] = [
117+
pub(crate) const ALL: [Self; 12] = [
100118
Self::WriterBootstrap,
101119
Self::ReaderBootstrap,
102120
Self::WriterReadiness,
@@ -108,13 +126,15 @@ impl PoolOperation {
108126
Self::ReaderSubscriptionHistory,
109127
Self::WriterEventWrite,
110128
Self::WriterMaintenance,
129+
Self::ReaderMaintenance,
111130
];
112131

113132
pub(crate) const fn pool_role(self) -> &'static str {
114133
match self {
115-
Self::ReaderBootstrap | Self::ReaderAuthorization | Self::ReaderSubscriptionHistory => {
116-
DbPoolRole::Reader.as_str()
117-
}
134+
Self::ReaderBootstrap
135+
| Self::ReaderAuthorization
136+
| Self::ReaderSubscriptionHistory
137+
| Self::ReaderMaintenance => DbPoolRole::Reader.as_str(),
118138
_ => DbPoolRole::Writer.as_str(),
119139
}
120140
}
@@ -130,7 +150,7 @@ impl PoolOperation {
130150
"subscription_history"
131151
}
132152
Self::WriterEventWrite => "event_write",
133-
Self::WriterMaintenance => "maintenance",
153+
Self::WriterMaintenance | Self::ReaderMaintenance => "maintenance",
134154
}
135155
}
136156

@@ -139,7 +159,7 @@ impl PoolOperation {
139159
}
140160
}
141161

142-
pub(crate) const POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 11] = [
162+
pub(crate) const POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 12] = [
143163
(DbPoolRole::Writer.as_str(), "bootstrap"),
144164
(DbPoolRole::Reader.as_str(), "bootstrap"),
145165
(DbPoolRole::Writer.as_str(), "readiness"),
@@ -151,9 +171,10 @@ pub(crate) const POOL_ACQUIRE_VALID_PAIRS: [(&str, &str); 11] = [
151171
(DbPoolRole::Reader.as_str(), "subscription_history"),
152172
(DbPoolRole::Writer.as_str(), "event_write"),
153173
(DbPoolRole::Writer.as_str(), "maintenance"),
174+
(DbPoolRole::Reader.as_str(), "maintenance"),
154175
];
155176

156-
/// Eleven valid pairs × (12 histogram series + 1 start counter + 4 outcome counters + 1 gauge).
177+
/// Twelve valid pairs × (12 histogram series + 1 start counter + 4 outcome counters + 1 gauge).
157178
pub(crate) const POOL_ACQUIRE_RAW_SERIES_PER_POD: usize = POOL_ACQUIRE_VALID_PAIRS.len() * 18;
158179

159180
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
@@ -610,9 +631,10 @@ mod tests {
610631
PoolOperation::ReaderBootstrap,
611632
PoolOperation::ReaderAuthorization,
612633
PoolOperation::ReaderSubscriptionHistory,
634+
PoolOperation::ReaderMaintenance,
613635
]
614636
);
615-
assert_eq!(super::POOL_ACQUIRE_RAW_SERIES_PER_POD, 198);
637+
assert_eq!(super::POOL_ACQUIRE_RAW_SERIES_PER_POD, 216);
616638
assert_eq!(
617639
LockType::ALL.map(LockType::as_str),
618640
[
@@ -1494,6 +1516,7 @@ mod tests {
14941516
let _ = routed_db
14951517
.query_events_routed("pool_operation_matrix_reader", &query)
14961518
.await;
1519+
let _ = routed_db.usage_fleet_stock_snapshot().await;
14971520
routed_db.refresh_pool_waiter_metrics();
14981521

14991522
let snapshot = snapshotter.snapshot().into_vec();

0 commit comments

Comments
 (0)