Skip to content

Commit 6b1e45d

Browse files
ravarora2codex
andauthored
Isolate S3 storage metrics from the relay (#7543)
## Why **This PR moves the S3 storage scan out of the serving relay and into a separate batch job.** The job saves a complete result in PostgreSQL. The relay reads that result, exports the existing storage totals, and reports the snapshot's freshness and database-read health. ```text Today: relay -> scan S3 -> calculate totals -> Datadog metrics After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics ``` The code in this PR is the new pipeline. It adds the worker, database snapshot, relay read mode, and Helm CronJob template. It does **not** enable the pipeline in production. The chart keeps the CronJob disabled, and the relay keeps its current inline mode by default. [squareup/builderbot-platform-core-infrastructure#244](squareup/builderbot-platform-core-infrastructure#244) will set the production limits, enable the job, and switch the relay to the saved snapshot after this PR merges. The relay currently lists the media bucket once an hour and calculates storage totals inside the serving process. The production bucket now exceeds the relay's one million object cap, so each sweep fails and the last good per-community values remain stale. The Datadog dashboard can no longer show current storage use by community. Raising that cap inside the serving relay would move a larger, memory-heavy scan into the process that owns live WebSocket connections. The scan retains state for unique blob hashes and community bindings, so its memory use grows with the bucket. A large scan could raise relay memory use, trigger restarts, and drop user connections. The goal is to restore current metrics without putting a production-scale bucket scan in the relay's failure domain. No media objects move or change. Only the process that calculates the totals and the handoff to the relay change. ## What - Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and saves one complete accounting snapshot in PostgreSQL. - Add `external` relay mode to read the saved snapshot. Keep the current `inline` mode as the default, and add `off` as a fail-closed option. - Add an opt-in Helm CronJob for the worker. The 10 million object cap exists only on this Job. - Keep the current fleet and per-community gauges, and add snapshot age, generation duration, configured cap, cap utilization, and database-load health gauges. - Add an additive migration for the latest complete snapshot and its run metadata. ## Overall plan 1. Merge this PR to add the worker, database handoff, relay modes, and disabled-by-default chart support. This step does not change production behavior. 2. Publish the relay image and Helm chart `0.1.9` from `main`. 3. Update [squareup/builderbot-platform-core-infrastructure#244](squareup/builderbot-platform-core-infrastructure#244) to pin those final artifacts and rerender its manifests. 4. Use the infrastructure PR to deploy the CronJob with a 10 million object cap, a one-hour deadline, and a 20 GiB memory limit. 5. Start one manual Job during a staffed window and verify its snapshot, duration, memory use, relay stability, and Datadog output. 6. Keep the daily schedule and `external` relay mode only after that gate passes, then add alerts for stale snapshots, snapshot-load failure, failed jobs, cap pressure, duration, and worker memory. The code and infrastructure remain separate so this PR can ship the mechanism without silently enabling a production-scale scan. The infrastructure PR owns the production settings and rollout gate. ## How ### Complete snapshot handoff The worker takes a deployment-wide PostgreSQL advisory lock before it lists S3. This prevents scheduled and manual workers from overlapping, even if Kubernetes starts both. The final database replacement runs through that same lock-owning session. If the session dies during the scan, publication fails instead of letting the stale worker overwrite its successor. A worker that no longer owns the lock cannot publish through another pooled connection. The worker reads S3 in pages of 1,000 objects and folds each page into physical totals, logical per-community usage, and anomaly counts. It stops before folding a page that would exceed `--max-objects` and emits structured progress every 100,000 objects. The worker replaces the singleton database row only after the full scan succeeds. A listing error, malformed page, cap breach, serialization error, lost lock session, or database error cannot publish a partial snapshot. The last completed snapshot stays available. Each stored result includes its completion time, duration, cap, and image or source revision. PostgreSQL is the handoff between the batch worker and the relay. The worker writes one complete deployment-wide snapshot, and the relay reads that snapshot on its normal leader metrics tick. No new network service or worker endpoint is required. ### Relay behavior `BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and its one million object default. `external` loads the newest completed worker snapshot on each leader metrics tick. `off` emits no storage-family metrics. An unknown value fails closed to `off`. External mode republishes the existing fleet, per-community, and anomaly metric names, so those current queries remain valid. Snapshot handoff health uses a separate family: `buzz_storage_snapshot_age_seconds`, `buzz_storage_snapshot_duration_seconds`, `buzz_storage_snapshot_max_objects`, `buzz_storage_snapshot_cap_utilization`, and `buzz_storage_snapshot_load_ok`. The `buzz_storage_sweep_*` gauges remain inline-only because they describe a scan attempted by the relay process. External mode does not synthesize `sweep_ok=1` from the last successful database row: an old row proves only that an earlier worker completed. Kubernetes CronJob/Job status is authoritative for the latest worker result, including non-zero exit, deadline expiry, and OOM termination. The relay keeps the last complete snapshot available when a worker run fails. Snapshot age shows that no replacement arrived on schedule, while `buzz_storage_snapshot_load_ok` distinguishes staleness from a missing row, database read failure, or invalid payload. If a community disappears, changes host, or leaves the configured emission scope, the relay zeroes the old labeled series instead of leaving stale values behind. Per-community bytes are logical referenced media usage. If two communities reference the same blob, each community receives the blob's bytes. Fleet physical totals still count each S3 object once. This PR preserves those existing metric semantics. ### Deployment boundary The CronJob is disabled by default. When enabled, it uses `concurrencyPolicy: Forbid`, no retry, a configurable active deadline, bounded history, explicit CPU and memory resources, a read-only root filesystem, and no Istio sidecar by default. The worker receives only `DATABASE_URL`, S3 settings, optional S3 credential references, and the code revision. It does not receive the relay private key, Git hook HMAC secret, Redis URL, or the full relay secret as environment variables. Operators may reuse the Buzz service account or select a dedicated one. ### Non-goals - This PR does not enable the worker or `external` mode in production. - This PR does not change media uploads, object ownership, retention, or deletion. - This PR does not change the names or meaning of the existing fleet, per-community, or anomaly storage gauges. - This PR does not add a lease service or allow multiple workers to publish concurrently. ## Risk The application and chart defaults do not enable the new worker or raise the serving relay's cap. The database change is an additive deployment-global singleton table. The relay's default remains `inline`, so merging this PR alone does not change the active metrics path. The main rollout risk is worker memory use because the fold retains state for distinct blob hashes and community bindings. The first production-sized run must confirm peak memory and duration before the schedule is trusted. The 10 million cap and one-hour deadline bound the first deployment, but they do not replace measurement. External mode also needs stale-snapshot, snapshot-load, and job-failure monitors because the relay intentionally keeps publishing the last complete result after a failed job or read error. This behavior avoids gaps and partial data, while snapshot age and load health tell operators when the result or PostgreSQL handoff needs attention. Job status remains necessary for immediate failures and termination reasons that a killed process cannot record itself. ## Testing No production S3 scan was run from this PR. Automated regressions verify that external mode keeps stale last-good totals visible without emitting inline attempt-health gauges, reports database-load failure while retaining the cache, and leaves inline telemetry unchanged. Mutation checks also prove those tests fail if external mode is routed through inline emission or a failed load is reported as healthy. After deployment, create one manual Job from the CronJob during a staffed window. Require a completed Job, exactly one new snapshot, fewer objects than the configured cap, acceptable duration and peak memory, no relay restart regression, restored Datadog points, and no OpenMetrics sample truncation. ## Next steps 1. Complete review and merge this PR. 2. Publish chart `0.1.9` and the final main-branch relay image. 3. Replace the provisional pins in [squareup/builderbot-platform-core-infrastructure#244](squareup/builderbot-platform-core-infrastructure#244), rerender, and require clean CI. 4. Run the staffed manual Job gate and record object count, peak memory, duration, snapshot revision, and relay stability. 5. Confirm the relay reads the new snapshot in `external` mode and the existing `block.buzz_relay.*` storage series update again. 6. Enable the daily 03:17 UTC schedule and add dashboard panels and monitors for snapshot age, snapshot-load health, Job failures, cap pressure, duration, and worker memory. If production must roll back to an older relay image, set `BUZZ_STORAGE_METRICS=off` first. Older images do not understand `external` and would resume the failing inline scan. ## Bigger picture This is the application half of the storage-metrics repair. The paired infrastructure PR owns the production resource limits, schedule, artifact pins, secret boundary, and monitored rollout. Keeping those changes separate lets this PR remain safe by default while still providing a complete path to current per-community S3 totals. Related issue: N/A. No matching issue or pull request exists in `block/buzz`. Generated with Codex --------- Signed-off-by: Ravneet Arora <[email protected]> Co-authored-by: Codex <[email protected]>
1 parent 4ab4f78 commit 6b1e45d

19 files changed

Lines changed: 1169 additions & 78 deletions

File tree

‎Justfile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -385,6 +385,12 @@ test-unit:
385385
# #[ignore]d, so --lib runs only the infra-free set. Without this gate a
386386
# stray file in migrations/ or a broken lint ships green.
387387
cargo nextest run -p buzz-db --lib
388+
# Storage accounting crosses three crates whose focused regression
389+
# suites are otherwise absent from the infra-free unit lane.
390+
cargo nextest run -p buzz-media --lib \
391+
-E 'test(=bucket_index::tests::bucket_snapshot_json_round_trip_preserves_community_keys)'
392+
cargo nextest run -p buzz-admin \
393+
-E 'test(=storage_snapshot_tests::failed_fold_never_invokes_snapshot_persistence)'
388394
# Multi-tenant conformance gate (buzz-conformance): the independent
389395
# replay checker + golden fixtures. No infra — pure in-process trace
390396
# replay — so it belongs in the unit job. Run all targets (lib + the
@@ -451,7 +457,7 @@ test-unit:
451457
# the ~30s sqlx acquire timeout, so they do not belong in the infra-free
452458
# unit job either.
453459
cargo nextest run -p buzz-relay --lib \
454-
-E '(test(/^api::admin::/) - test(=api::admin::tests::disabled_mode_allows_unauthenticated_requests_on_the_admin_host) - test(=api::admin::tests::nip98_mode_unrostered_signer_does_not_consume_a_replay_slot)) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/)'
460+
-E '(test(/^api::admin::/) - test(=api::admin::tests::disabled_mode_allows_unauthenticated_requests_on_the_admin_host) - test(=api::admin::tests::nip98_mode_unrostered_signer_does_not_consume_a_replay_slot)) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^storage_sweep::tests::/)'
455461
# ACP author-gate and queue tests protect the trust boundary between
456462
# relay events and agent prompts. They are infra-free; ignored lifecycle
457463
# tests remain excluded and run in their dedicated integration lanes.

‎crates/buzz-admin/src/main.rs‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,16 @@
2222
2323
mod deletions;
2424

25+
use std::future::Future;
26+
use std::sync::atomic::{AtomicU64, Ordering};
2527
use std::sync::Arc;
28+
use std::time::Instant;
2629

2730
use anyhow::Result;
2831
use buzz_core::kind::KIND_NIP43_MEMBERSHIP_LIST;
2932
use buzz_core::tenant::{relay_url_authority, TenantContext};
3033
use buzz_db::{Db, DbConfig};
34+
use buzz_media::{BucketSnapshot, MediaConfig, MediaStorage, S3AddressingStyle, SweepError};
3135
use buzz_pubsub::{EventTopic, PubSubManager};
3236
use clap::{Parser, Subcommand};
3337
use nostr::{EventBuilder, Keys, Kind, Tag};
@@ -78,6 +82,12 @@ enum Command {
7882
GenerateKey,
7983
/// Run pending database migrations.
8084
Migrate,
85+
/// Compute one complete S3 storage snapshot and persist it for relay readers.
86+
StorageSnapshot {
87+
/// Abort before folding a page that would exceed this object count.
88+
#[arg(long, default_value_t = 10_000_000)]
89+
max_objects: u64,
90+
},
8191
/// Inspect deployment-wide Buzz product feedback.
8292
ProductFeedback {
8393
#[command(subcommand)]
@@ -154,6 +164,7 @@ async fn run(cli: Cli) -> Result<i32> {
154164
println!("Database migrations complete.");
155165
Ok(0)
156166
}
167+
Command::StorageSnapshot { max_objects } => cmd_storage_snapshot(max_objects).await,
157168
Command::AddMember { pubkey, role } => cmd_add_member(pubkey, role).await,
158169
Command::RemoveMember { pubkey, role } => cmd_remove_member(pubkey, role).await,
159170
Command::ListMembers => cmd_list_members().await,
@@ -168,6 +179,140 @@ async fn run(cli: Cli) -> Result<i32> {
168179
}
169180
}
170181

182+
async fn cmd_storage_snapshot(max_objects: u64) -> Result<i32> {
183+
let max_objects_db = i64::try_from(max_objects)
184+
.map_err(|_| anyhow::anyhow!("--max-objects must be at most {}", i64::MAX))?;
185+
if max_objects == 0 {
186+
return Err(anyhow::anyhow!("--max-objects must be greater than zero"));
187+
}
188+
189+
let db = connect_db().await?;
190+
let mut leader = db.try_lock_storage_accounting().await?.ok_or_else(|| {
191+
anyhow::anyhow!("another storage-snapshot worker already holds the lease")
192+
})?;
193+
let storage = Arc::new(MediaStorage::new(&storage_config_from_env()?)?);
194+
let code_sha =
195+
std::env::var("BUZZ_STORAGE_SNAPSHOT_CODE_SHA").unwrap_or_else(|_| "unknown".to_string());
196+
if code_sha.is_empty() || code_sha.len() > 128 {
197+
return Err(anyhow::anyhow!(
198+
"BUZZ_STORAGE_SNAPSHOT_CODE_SHA must contain 1 to 128 bytes"
199+
));
200+
}
201+
202+
println!(
203+
"{}",
204+
serde_json::json!({
205+
"event": "storage_snapshot_started",
206+
"max_objects": max_objects,
207+
"code_sha": code_sha,
208+
})
209+
);
210+
let run_started = Instant::now();
211+
let listed_objects = Arc::new(AtomicU64::new(0));
212+
let fold = buzz_media::fold_bucket_listing(max_objects, move |token| {
213+
let storage = Arc::clone(&storage);
214+
let listed_objects = Arc::clone(&listed_objects);
215+
async move {
216+
let page = storage.list_page(token, 1000).await?;
217+
let page_objects = u64::try_from(page.objects.len()).unwrap_or(u64::MAX);
218+
let before = listed_objects.fetch_add(page_objects, Ordering::Relaxed);
219+
let after = before.saturating_add(page_objects);
220+
if before / 100_000 != after / 100_000 {
221+
println!(
222+
"{}",
223+
serde_json::json!({
224+
"event": "storage_snapshot_progress",
225+
"listed_objects": after,
226+
"max_objects": max_objects,
227+
})
228+
);
229+
}
230+
Ok(page)
231+
}
232+
});
233+
let snapshot_code_sha = code_sha.clone();
234+
let persisted = persist_completed_fold(fold, move |encoded, duration_ms| async move {
235+
leader
236+
.save_snapshot(&encoded, duration_ms, max_objects_db, &snapshot_code_sha)
237+
.await?;
238+
Ok(())
239+
})
240+
.await;
241+
let (snapshot, duration_ms) = match persisted {
242+
Ok(completed) => completed,
243+
Err(error) => {
244+
println!(
245+
"{}",
246+
serde_json::json!({
247+
"event": "storage_snapshot_failed",
248+
"duration_ms": i64::try_from(run_started.elapsed().as_millis()).unwrap_or(i64::MAX),
249+
"max_objects": max_objects,
250+
"code_sha": code_sha,
251+
"error": error.to_string(),
252+
})
253+
);
254+
return Err(error);
255+
}
256+
};
257+
println!(
258+
"{}",
259+
serde_json::json!({
260+
"event": "storage_snapshot_completed",
261+
"duration_ms": duration_ms,
262+
"listed_objects": snapshot.physical_objects,
263+
"listed_bytes": snapshot.physical_bytes,
264+
"logical_objects": snapshot.logical_objects,
265+
"logical_bytes": snapshot.logical_bytes,
266+
"max_objects": max_objects,
267+
"code_sha": code_sha,
268+
})
269+
);
270+
Ok(0)
271+
}
272+
273+
async fn persist_completed_fold<FoldFuture, Persist, PersistFuture>(
274+
fold: FoldFuture,
275+
persist: Persist,
276+
) -> Result<(BucketSnapshot, i64)>
277+
where
278+
FoldFuture: Future<Output = std::result::Result<BucketSnapshot, SweepError>>,
279+
Persist: FnOnce(serde_json::Value, i64) -> PersistFuture,
280+
PersistFuture: Future<Output = Result<()>>,
281+
{
282+
let started = Instant::now();
283+
let snapshot = fold.await?;
284+
let duration_ms = i64::try_from(started.elapsed().as_millis()).unwrap_or(i64::MAX);
285+
let encoded = serde_json::to_value(&snapshot)?;
286+
persist(encoded, duration_ms).await?;
287+
Ok((snapshot, duration_ms))
288+
}
289+
290+
fn storage_config_from_env() -> Result<MediaConfig> {
291+
let required = |name: &str| {
292+
std::env::var(name).map_err(|_| anyhow::anyhow!("{name} must be set for storage-snapshot"))
293+
};
294+
let addressing_style = std::env::var("BUZZ_S3_ADDRESSING_STYLE")
295+
.unwrap_or_else(|_| "path".to_string())
296+
.parse::<S3AddressingStyle>()
297+
.map_err(anyhow::Error::msg)?;
298+
Ok(MediaConfig {
299+
s3_endpoint: std::env::var("BUZZ_S3_ENDPOINT").unwrap_or_default(),
300+
s3_access_key: std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_default(),
301+
s3_secret_key: std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_default(),
302+
s3_bucket: required("BUZZ_S3_BUCKET")?,
303+
s3_region: std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
304+
s3_addressing_style: addressing_style,
305+
max_image_bytes: 1,
306+
max_gif_bytes: 1,
307+
max_video_bytes: 1,
308+
max_file_bytes: 1,
309+
public_base_url: "http://storage-snapshot.invalid/media".to_string(),
310+
upload_records_enabled: false,
311+
upload_ip_header: None,
312+
upload_port_header: None,
313+
})
314+
}
315+
171316
async fn cmd_add_member(pubkey_arg: String, role: String) -> Result<i32> {
172317
if let Err(msg) = validate_role(&role) {
173318
eprintln!("error: {msg}");
@@ -631,3 +776,27 @@ async fn reconcile_channels(
631776
);
632777
Ok(())
633778
}
779+
780+
#[cfg(test)]
781+
mod storage_snapshot_tests {
782+
use std::sync::atomic::{AtomicUsize, Ordering};
783+
784+
use super::*;
785+
786+
#[tokio::test]
787+
async fn failed_fold_never_invokes_snapshot_persistence() {
788+
let persist_calls = Arc::new(AtomicUsize::new(0));
789+
let observed_calls = Arc::clone(&persist_calls);
790+
let result = persist_completed_fold(
791+
async { Err::<BucketSnapshot, _>(SweepError::MalformedPage) },
792+
move |_, _| async move {
793+
observed_calls.fetch_add(1, Ordering::SeqCst);
794+
Ok(())
795+
},
796+
)
797+
.await;
798+
799+
assert!(result.is_err());
800+
assert_eq!(persist_calls.load(Ordering::SeqCst), 0);
801+
}
802+
}

‎crates/buzz-db/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ pub use store::{
5050
admin_moderation, allowlist, api_token, archived_identities, channel, channel_members,
5151
community, deletion, dm, event, feed, git_repo, moderation, partition, product_feedback, push,
5252
reaction, relay_admin_actions, relay_invite, relay_members, relay_operators, reminder,
53-
replaceable, thread, usage, user, workflow,
53+
replaceable, storage_accounting, thread, usage, user, workflow,
5454
};
5555

5656
pub use allowlist::AllowlistEntry;

‎crates/buzz-db/src/runtime/migration.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -489,6 +489,7 @@ mod postgres_tests {
489489
"relay_admin_actions",
490490
"relay_admin_outbox",
491491
"relay_operator_audit",
492+
"storage_accounting_snapshots",
492493
] {
493494
if normalized[insert_pos..].contains(&format!("'{value}'")) {
494495
globals.insert(value.to_owned());
@@ -702,7 +703,7 @@ mod postgres_tests {
702703
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
703704
migrations.sort_by_key(|migration| migration.version);
704705

705-
assert_eq!(migrations.len(), 45);
706+
assert_eq!(migrations.len(), 46);
706707
assert_eq!(migrations[0].version, 1);
707708
assert_eq!(&*migrations[0].description, "initial schema");
708709
assert!(migrations[0]
@@ -1282,6 +1283,11 @@ mod postgres_tests {
12821283
// The restored exclusion function must NOT list any NIP-FI relation.
12831284
assert!(!ledger_removal.contains("'authorization_operation_receipts'"));
12841285
assert!(!ledger_removal.contains("'identity_bindings'"));
1286+
assert_eq!(migrations[45].version, 46);
1287+
assert!(migrations[45]
1288+
.sql
1289+
.as_str()
1290+
.contains("CREATE TABLE storage_accounting_snapshots"));
12851291
// schema.sql exclusion list must match the restored (pre-0041) body.
12861292
assert!(
12871293
desired_schema.contains("'rate_limit_violations'\n ]::TEXT[])"),

‎crates/buzz-db/src/store/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ pub mod relay_operators;
4646
pub mod reminder;
4747
/// Replaceable-event persistence and coordinate locking.
4848
pub mod replaceable;
49+
/// Durable completed snapshots from the isolated media-storage worker.
50+
pub mod storage_accounting;
4951
/// Thread metadata persistence.
5052
pub mod thread;
5153
/// Per-community usage rollup queries for Prometheus gauges.

0 commit comments

Comments
 (0)