Skip to content

Commit 4142d2b

Browse files
authored
Use worker snapshots for relay storage metrics (#7845)
## Summary Make the relay read completed storage-accounting snapshots from PostgreSQL by default. An environment with no snapshot emits no storage metrics. When a worker publishes its first result, the relay picks it up on the next successful leader usage tick, without a restart or configuration change. This changes how storage usage is measured and reported. **S3 remains the object store. PostgreSQL stores the completed counts, byte totals, community breakdown, and calculation metadata—not the objects themselves.** ### Why move relay storage accounting entirely to database reads? Today the relay supports two sources: its own S3 scan (`inline`, the default) and the worker's saved snapshot (`external`). That requires operators to deploy a worker and then coordinate a separate relay-mode change in each environment. The worker already owns the S3 listing and calculation. It runs in a separate process with its own object cap, memory limit, and deadline. Keeping a second scan inside the serving relay retains the resource pressure and failure modes that this isolation is meant to remove. PostgreSQL provides a durable handoff. The worker replaces one completed snapshot atomically after a successful calculation. The relay reads that row and exposes its values through the existing metrics endpoint. A relay restart or leadership change can load the last completed result without scanning the bucket again. There is deliberately no S3 fallback when a snapshot is absent or a read fails. Such a fallback would put the expensive scan back in the relay precisely when a worker is absent or unhealthy. It would also leave two calculation paths with different resource limits and freshness behavior. ### What happens if no worker exists? **A worker that has never published is a normal, successful inactive state.** The snapshot table must exist through the normal database schema, but it may contain zero rows. The reader returns success, emits no storage totals or failed-load gauge, and checks again on the next usage tick. It does not exit the relay, start a worker, or list S3. The relay checks for a completed database row; it does not inspect Kubernetes Jobs or CronJobs. Worker presence and snapshot presence are therefore different: | Situation | Storage-reader behavior | |---|---| | No worker and no snapshot row | Return success, emit no storage metrics, and query again next tick. | | Worker starts later and publishes its first row | Read and emit it on the next successful leader usage tick; no restart or mode change. | | Worker is removed or fails, but its last valid row remains | Continue reporting the last completed totals with their increasing age. | | Query succeeds but a previously present row has been removed | Clear the cache and stop refreshing storage series; existing exporter series expire through the idle timeout. | | Database read fails, times out, or returns an invalid snapshot | Log a warning, set `buzz_storage_snapshot_load_ok=0`, retain any last-good cached totals, and retry next tick. | | Snapshot table is missing | Treat this as a schema error, not normal worker absence; follow the failed-read behavior above. | | `BUZZ_STORAGE_METRICS=off` | Skip the snapshot query and all storage metric emission. | No row means “not measured,” not zero bytes. A valid completed snapshot of an empty bucket can report zero. If a read fails before any good snapshot has been cached, only failed-load health is emitted. These reads happen in the existing background metrics task, not the relay startup path. A storage-read failure does not request process exit or change readiness directly. A wider database outage can still affect the relay's existing database-dependent operations and readiness. ### Read, parse, and emit path ```text Worker: S3 listing -> BucketSnapshot -> JSON -> PostgreSQL Relay: PostgreSQL -> JSON -> BucketSnapshot -> existing metric gauges Collector: relay /metrics endpoint -> Datadog ``` 1. `main.rs::run_storage_sweep_tick` calls the reader from the existing leader-only usage task. The default interval remains 300 seconds. 2. `storage_sweep.rs::run_storage_metrics_tick` coordinates the read, cache update, failure health, and metric emission. 3. `refresh_persisted_snapshot` calls the existing `Db::load_storage_accounting_snapshot()` method. A five-second timeout covers pool acquisition and the query. It uses the relay's existing pool. 4. The existing query in `buzz-db/src/store/storage_accounting.rs` reads `snapshot`, `completed_at`, `duration_ms`, `max_objects`, and `code_sha` from the singleton row. 5. `serde_json::from_value(stored.snapshot)` decodes the JSON into the existing `buzz_media::BucketSnapshot` type. Rust infers that type from `cache_persisted_snapshot`'s argument. The generated `Deserialize` implementation handles all fields, including the UUID-keyed `per_community` map. 6. `emit_cached_storage_metrics` sets the existing physical/logical totals and community byte/object gauges. The existing Prometheus exporter exposes them for Datadog collection; this code does not send snapshot JSON to Datadog. The worker's `BucketSnapshot`/`CommunityStorage` structures, JSON field names, SQL publication method, and database schema are unchanged. The old external-mode JSON conversion moves out of `main.rs` into the reader helper. No new format or hand-written field parser is needed. Duration and object-cap metadata are validated before replacing the cache. The five-second relay read timeout is separate from the worker's 30-second startup acquisition budget introduced in #7770. This PR does not change the worker's pool or startup logic. ### Configuration and code changes | Configuration | Before | After | |---|---|---| | Unset, `inline`, or `on` | Relay-local S3 scan | Read completed database snapshots. `inline` logs one migration warning. | | `external` or `snapshot` | Read completed database snapshots | Continue reading snapshots. | | `off` | Disabled | Disabled. | | Unknown or empty value | Disabled with a configuration error | Same. | Existing `inline` settings intentionally become reader aliases. Merely changing the default would leave older deployments on their explicit `inline` setting and preserve the need for coordinated configuration PRs. Remove relay-local scan scheduling, in-flight scan state, scan configuration, and obsolete scan-attempt tests and health metrics. Preserve the physical/logical total and per-community metric names, leader-only emission, community scope filtering, and cleanup of old community labels. The shared relay-state field remains in place; its comment now describes a cached worker result. The Helm chart defaults to `relayMode: external` while leaving `storageAccounting.enabled: false`. Its rendering test verifies that the reader is configured even when no worker is created. The reader regression tests cover missing rows, later publication, updates, invalid data, timeout, removal, reactivation, and the explicit off switch. `docs/storage-accounting.md` describes this operating model. ### Freshness and rollout impact `buzz_storage_snapshot_age_seconds` uses the worker's original completion timestamp. Re-reading an old row never makes it fresh. `buzz_storage_snapshot_load_ok=1` means the row was read and decoded; it does not prove that the worker's latest attempt succeeded. Monitor snapshot age against the worker's schedule and allowed runtime, and monitor Job failures separately. The old relay scan-attempt health gauges are retired because the relay no longer runs those scans. Deploy this relay version through the usual release process. Once it reaches an environment, workers can be enabled independently. Already-released charts that still set `inline` work with the new binary; future chart releases default explicitly to `external`. An intentional `off` override still requires an operator to enable the reader. **Environments with no completed snapshot lose relay-calculated storage totals after this upgrade.** That is the intended tradeoff for removing scans from the serving process. Environments with a prior snapshot continue showing its last completed totals and age. This PR does not install workers in additional environments or deploy the new relay image. ### Related issue Related to #4601. Follows the worker startup fix in #7770. The separate fleet-usage collector work in #7176 overlaps this code and may need reconciliation when it merges. ### Testing Built and ran the relay locally against isolated PostgreSQL and Redis with the legacy `inline` setting, five-second usage ticks, and a 15-second metric idle timeout. The same process was exercised through these steps: 1. Start with an empty snapshot table: relay becomes ready and exposes no storage series. 2. Publish a snapshot containing 123 bytes: the community gauge becomes 123. 3. Replace it with 456 bytes: the gauge becomes 456. 4. Remove the row: storage series expire without fabricating a zero-byte measurement. 5. Publish 789 bytes: metrics reactivate without restarting the relay; readiness remains healthy. The local S3 endpoint recorded zero requests. The unrelated Git conformance startup probe was disabled for this isolated reader check; the test covers storage accounting, not every other use of S3 in the relay. No new relay image was deployed to staging or production during this test. The broad local integration run also hit an existing `p0_pool_acquisitions_use_typed_operation_pairs_without_other` failure in `buzz-db/tests/observability_source.rs`. The same source-only test fails on unmodified `main` at `0ef7a2222`; this PR changes neither the test nor its failing source. The full local `just ci` run stopped at mobile native-asset setup: the `objective_c` build hook received no SDK path from `xcrun`. Mobile tests did not run. Generated with Codex Signed-off-by: Ravneet Arora <[email protected]>
1 parent 3b2e50b commit 4142d2b

7 files changed

Lines changed: 408 additions & 863 deletions

File tree

‎crates/buzz-relay/src/main.rs‎

Lines changed: 13 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -1745,84 +1745,26 @@ async fn run_usage_metrics_tick(
17451745
Ok(())
17461746
}
17471747

1748-
/// Storage-sweep half of the leader-only tick: harvest/spawn (never awaits
1749-
/// the sweep itself) then re-emit whatever snapshot is cached. Split out of
1750-
/// [`run_usage_metrics_tick`] because it has its own always-on config
1751-
/// (independent of `EmissionScope`) and a hard kill switch — a disabled
1752-
/// sweep must never touch a single storage-family gauge, including the
1753-
/// health ones, so a relay without `s3:ListBucket` can turn the whole
1754-
/// feature off cleanly.
1748+
/// Read worker storage snapshots only on the existing leader metrics tick.
17551749
async fn run_storage_sweep_tick(
17561750
state: &AppState,
17571751
emission_scope: &EmissionScope,
17581752
host_map: &HashMap<Uuid, String>,
17591753
) {
1760-
static SWEEP_CONFIG: std::sync::OnceLock<storage_sweep::StorageSweepConfig> =
1754+
static MODE: std::sync::OnceLock<storage_sweep::StorageMetricsMode> =
17611755
std::sync::OnceLock::new();
1762-
// SWEEP_CONFIG is a function-local OnceLock by design: it is localized
1763-
// feature config consumed only by this code path, read once on the first
1764-
// leader tick, and stable for the process lifetime (env is immutable).
1765-
// Keeping it here avoids widening Config/AppState for a single consumer.
1766-
let config = *SWEEP_CONFIG.get_or_init(storage_sweep::StorageSweepConfig::from_env);
1767-
match config.mode {
1768-
storage_sweep::StorageMetricsMode::Disabled => return,
1769-
storage_sweep::StorageMetricsMode::Inline => {
1770-
let media_storage = Arc::clone(&state.media_storage);
1771-
let max_objects = config.max_objects;
1772-
storage_sweep::maybe_spawn_sweep(
1773-
&state.storage_sweep,
1774-
config.interval,
1775-
config.timeout,
1776-
async move {
1777-
buzz_media::fold_bucket_listing(max_objects, move |token| {
1778-
let media_storage = Arc::clone(&media_storage);
1779-
async move { media_storage.list_page(token, 1000).await }
1780-
})
1781-
.await
1782-
},
1783-
)
1784-
.await;
1785-
}
1786-
storage_sweep::StorageMetricsMode::Snapshot => {
1787-
match state.db.load_storage_accounting_snapshot().await {
1788-
Ok(Some(stored)) => match serde_json::from_value(stored.snapshot) {
1789-
Ok(snapshot) => {
1790-
let duration = std::time::Duration::from_millis(
1791-
u64::try_from(stored.duration_ms).unwrap_or_default(),
1792-
);
1793-
let max_objects = u64::try_from(stored.max_objects).unwrap_or_default();
1794-
storage_sweep::cache_persisted_snapshot(
1795-
&state.storage_sweep,
1796-
snapshot,
1797-
stored.completed_at,
1798-
duration,
1799-
max_objects,
1800-
)
1801-
.await;
1802-
}
1803-
Err(error) => {
1804-
warn!(error = %error, "stored storage snapshot is invalid");
1805-
storage_sweep::record_persisted_snapshot_load_failure(&state.storage_sweep)
1806-
.await;
1807-
}
1808-
},
1809-
Ok(None) => {
1810-
storage_sweep::record_persisted_snapshot_load_failure(&state.storage_sweep)
1811-
.await;
1812-
}
1813-
Err(error) => {
1814-
warn!(error = %error, "failed to load stored storage snapshot");
1815-
storage_sweep::record_persisted_snapshot_load_failure(&state.storage_sweep)
1816-
.await;
1817-
}
1818-
}
1819-
}
1756+
let mode = *MODE.get_or_init(storage_sweep::StorageMetricsMode::from_env);
1757+
if let Err(error) = storage_sweep::run_storage_metrics_tick(
1758+
&state.db,
1759+
&state.storage_sweep,
1760+
mode,
1761+
host_map,
1762+
|id| emission_scope.allows(id),
1763+
)
1764+
.await
1765+
{
1766+
warn!(error = %error, "failed to load stored storage snapshot; retrying next usage tick");
18201767
}
1821-
1822-
storage_sweep::emit_storage_metrics(&state.storage_sweep, config.mode, host_map, |id| {
1823-
emission_scope.allows(id)
1824-
})
1825-
.await;
18261768
}
18271769

18281770
/// Emit the database-derived usage snapshot from the stable leader only.

‎crates/buzz-relay/src/state.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -703,7 +703,7 @@ pub struct AppState {
703703
pub audit_tx: Option<mpsc::Sender<buzz_audit::NewAuditEntry>>,
704704
/// Media storage client (S3/MinIO).
705705
pub media_storage: Arc<MediaStorage>,
706-
/// Single-flight + cache state for the hourly S3 storage sweep. See
706+
/// Cached worker snapshot and storage metric emission bookkeeping. See
707707
/// `storage_sweep` module docs; shared with the usage-metrics tick via
708708
/// `Arc` the same way other cross-tick poller state lives on `AppState`.
709709
pub storage_sweep: Arc<tokio::sync::Mutex<crate::storage_sweep::StorageSweepState>>,

0 commit comments

Comments
 (0)