Skip to content

Commit 14a752f

Browse files
Aresoncoder 0codexclaude
authored
fix(db): audit partition catalog before creation (#6515)
## Why The partition manager checks partition existence by name only and treats Postgres 42P17 "would overlap partition" errors as success. On databases where a catch-all partition (`events_p_future` / `delivery_log_p_future`) covers future months, the monthly creation path silently does nothing while real coverage erodes toward a write outage. There is currently no visibility into partition catalog health and no signal before writes start failing. ## What - **Read-only catalog audit** in `buzz-db` (`partition.rs`): classifies the live partition catalog for `events` and `delivery_log` via `pg_inherits` and bound parsing — coverage bounds, catch-all detection, anomalous children, and trigger parity checked recursively down to routable leaf partitions. - **Creation gated on the audit verdict**: the 42P17-to-success swallow is removed; monthly CREATE runs only for months the audit proves uncovered, a canonical-name mismatch is an error, and per-month outcome capture replaces first-error abort. New `BUZZ_PARTITION_MANAGER_CREATE_ENABLED` kill switch (default `true`). - **Prometheus metrics** per table: `buzz_partition_serving_safe`, `buzz_partition_uncovered_months` (audit window is now through +3 months), `buzz_partition_catch_all_covered_months`, `buzz_partition_anomalous_children`, `buzz_partition_trigger_parity_missing`/`_extra`, `buzz_partition_audit_duration_seconds`, `buzz_partition_audit_runs_total{outcome}`, `buzz_partition_audit_failures_total`, `buzz_partition_create_attempts_total`, and `buzz_partition_audit_last_success_timestamp_seconds` for staleness alerting. - **Cached diagnostics**: `/_status.partition_catalog` reports partition serving-safety, audit age, and freshness. `/_readiness` stays process-local, consistent with #7341; missing, unsafe, or stale audits do not remove pods from service. - **Periodic audit task** in the relay, plus config surface for the audit interval and the monthly-create kill switch. Catch-alls are detected from catalog bounds rather than configured names. - Doc correction: the mesh heartbeat comment no longer claims readiness gating (it never was gated; behavior unchanged). ## Behavior changes & risk - No DETACH/DROP/ATTACH and no schema migrations. The only DDL is the pre-existing monthly CREATE path, now strictly narrower (audit-proven-uncovered months only) and behind a kill switch. - Partition audit health does not gate readiness. Until the first completed audit, read-only retries back off from 5 seconds to a 60-second cap. Any completed audit, including unsafe coverage, restores the normal interval. - Partition-overlap errors now surface as errors instead of being swallowed as success. - Rollback: revert the commit, or set `BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` to disable the creation path while keeping the audit and metrics. ## Validation - New Postgres-backed regression tests for catalog classification and trigger parity (disabled child trigger, `ENABLE ALWAYS` parent, disabled nested leaf clone in a multi-level layout, healthy-catalog zero-parity checks). Each new regression was verified to fail against the previous logic before the fix. - Local affected-target validation passed against a disposable current-schema database, with the known mesh forwarded-echo and tracing callsite failures excluded. This is not a claim that the full suite passes. Follow-up work (separate PRs): bounded catch-all advance (empty-only golden path; non-empty reattach only behind an explicit operator flag, default off), fresh-install convergence, and proactive coverage-runway alerting. --- ## Update — 2026-08-21 - Fixed nested-leaf trigger parity auditing so child-only row triggers are checked on every routable descendant leaf and reported as `{leaf}:{trigger}` under the owning root child. - Added a Postgres regression test for a child-only trigger on a nested leaf; mutation-reverting to immediate-child-only extra-trigger scanning makes it fail. Generated with Codex Signed-off-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz> Signed-off-by: Ian Oberst <[email protected]> Signed-off-by: Ian Oberst <[email protected]> Co-authored-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz> Co-authored-by: Codex <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 4ef2360 commit 14a752f

13 files changed

Lines changed: 4010 additions & 151 deletions

File tree

‎.env.example‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,10 @@ REDIS_URL=redis://localhost:6379
3737
# Max connections in each of the relay's Postgres pools — writer and, when
3838
# READ_DATABASE_URL is set, reader (default 50).
3939
# BUZZ_DB_POOL_SIZE=50
40+
# Read-only partition catalog audit interval, clamped to 60s..24h (default 900s).
41+
# BUZZ_PARTITION_AUDIT_INTERVAL_SECS=900
42+
# Emergency kill switch for monthly partition CREATE; audits remain enabled.
43+
# BUZZ_PARTITION_MANAGER_CREATE_ENABLED=true
4044

4145
# Writer-session Postgres timeouts for buzz-db-backed pools and the relay audit
4246
# pool, all in milliseconds; 0 disables. The separately deployed push gateway

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/buzz-admin/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ buzz-media = { workspace = true }
2424
nostr = { workspace = true }
2525
tokio = { workspace = true }
2626
serde_json = { workspace = true }
27+
serde = { workspace = true }
2728
anyhow = { workspace = true }
2829
hex = { workspace = true }
2930
deadpool-redis = { workspace = true }

‎crates/buzz-admin/src/main.rs‎

Lines changed: 114 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,13 @@ use std::time::Instant;
3131
use anyhow::Result;
3232
use buzz_core::kind::KIND_NIP43_MEMBERSHIP_LIST;
3333
use buzz_core::tenant::{relay_url_authority, TenantContext};
34-
use buzz_db::{Db, DbConfig};
34+
use buzz_db::{partition::PartitionAuditReport, Db, DbConfig};
3535
use buzz_media::{BucketSnapshot, MediaConfig, MediaStorage, S3AddressingStyle, SweepError};
3636
use buzz_pubsub::{EventTopic, PubSubManager};
3737
use clap::{Parser, Subcommand};
3838
use nostr::{EventBuilder, Keys, Kind, Tag};
39+
use serde::Serialize;
40+
use sqlx::Row;
3941
use tracing::warn;
4042

4143
#[derive(Parser)]
@@ -89,6 +91,12 @@ enum Command {
8991
#[arg(long, default_value_t = 10_000_000)]
9092
max_objects: u64,
9193
},
94+
/// Run the partition catalog audit using a read-only database session.
95+
PartitionAudit {
96+
/// Future months to include in the coverage check.
97+
#[arg(long, default_value_t = 3)]
98+
months_ahead: u32,
99+
},
92100
/// Inspect deployment-wide Buzz product feedback.
93101
ProductFeedback {
94102
#[command(subcommand)]
@@ -166,6 +174,7 @@ async fn run(cli: Cli) -> Result<i32> {
166174
Ok(0)
167175
}
168176
Command::StorageSnapshot { max_objects } => cmd_storage_snapshot(max_objects).await,
177+
Command::PartitionAudit { months_ahead } => cmd_partition_audit(months_ahead).await,
169178
Command::AddMember { pubkey, role } => cmd_add_member(pubkey, role).await,
170179
Command::RemoveMember { pubkey, role } => cmd_remove_member(pubkey, role).await,
171180
Command::ListMembers => cmd_list_members().await,
@@ -314,6 +323,94 @@ fn storage_config_from_env() -> Result<MediaConfig> {
314323
})
315324
}
316325

326+
#[derive(Serialize)]
327+
struct PartitionAuditIdentity {
328+
database: String,
329+
user: String,
330+
schema: String,
331+
default_transaction_read_only: bool,
332+
transaction_read_only: bool,
333+
}
334+
335+
#[derive(Serialize)]
336+
struct PartitionAuditOutput {
337+
schema_version: u32,
338+
mode: &'static str,
339+
source_sha: &'static str,
340+
build_id: &'static str,
341+
build_url: &'static str,
342+
outcome: &'static str,
343+
months_ahead: u32,
344+
identity: PartitionAuditIdentity,
345+
report: PartitionAuditReport,
346+
}
347+
348+
async fn cmd_partition_audit(months_ahead: u32) -> Result<i32> {
349+
let db_url = std::env::var("DATABASE_URL")
350+
.map_err(|_| anyhow::anyhow!("DATABASE_URL is required for partition-audit"))?;
351+
let config = DbConfig {
352+
database_url: db_url,
353+
max_connections: 1,
354+
min_connections: 1,
355+
statement_timeout_ms: 30_000,
356+
default_transaction_read_only: true,
357+
..DbConfig::default()
358+
};
359+
let pool = Db::connect_writer_pool(&config).await?;
360+
let row = sqlx::query(
361+
"SELECT current_database() AS database, current_user AS user, \
362+
current_schema() AS schema, \
363+
current_setting('default_transaction_read_only') = 'on' \
364+
AS default_transaction_read_only, \
365+
current_setting('transaction_read_only') = 'on' AS transaction_read_only",
366+
)
367+
.fetch_one(&pool)
368+
.await?;
369+
let identity = PartitionAuditIdentity {
370+
database: row.try_get("database")?,
371+
user: row.try_get("user")?,
372+
schema: row.try_get("schema")?,
373+
default_transaction_read_only: row.try_get("default_transaction_read_only")?,
374+
transaction_read_only: row.try_get("transaction_read_only")?,
375+
};
376+
if !identity.default_transaction_read_only || !identity.transaction_read_only {
377+
anyhow::bail!("partition-audit connection is not read-only");
378+
}
379+
380+
let report = Db::from_pool(pool)
381+
.audit_partitions_report(months_ahead)
382+
.await;
383+
let outcome = if !report.errors.is_empty() {
384+
"error"
385+
} else if !report.serving_safe() {
386+
"unsafe"
387+
} else if report.tables.iter().any(|table| table.degraded()) {
388+
"degraded"
389+
} else {
390+
"ok"
391+
};
392+
let code = match outcome {
393+
"error" => 5,
394+
"unsafe" => 2,
395+
_ => 0,
396+
};
397+
println!(
398+
"{}",
399+
serde_json::to_string_pretty(&PartitionAuditOutput {
400+
schema_version: 1,
401+
mode: "read_only",
402+
source_sha: option_env!("BUZZ_SOURCE_SHA").unwrap_or("unknown"),
403+
build_id: option_env!("BUZZ_BUILD_ID").unwrap_or("local"),
404+
build_url: option_env!("BUZZ_BUILD_URL").unwrap_or("unknown"),
405+
outcome,
406+
months_ahead,
407+
identity,
408+
report,
409+
})?
410+
);
411+
Ok(code)
412+
}
413+
317414
async fn cmd_add_member(pubkey_arg: String, role: String) -> Result<i32> {
318415
if let Err(msg) = validate_role(&role) {
319416
eprintln!("error: {msg}");
@@ -581,8 +678,7 @@ async fn connect_db() -> Result<Db> {
581678
}
582679

583680
fn db_config_from_env() -> DbConfig {
584-
let db_url = std::env::var("DATABASE_URL")
585-
.unwrap_or_else(|_| "postgres://buzz:buzz_dev@localhost:5432/buzz".to_string());
681+
let db_url = std::env::var("DATABASE_URL").unwrap_or_else(|_| DbConfig::default().database_url);
586682
DbConfig {
587683
database_url: db_url,
588684
..DbConfig::default()
@@ -801,3 +897,18 @@ mod storage_snapshot_tests {
801897
assert_eq!(persist_calls.load(Ordering::SeqCst), 0);
802898
}
803899
}
900+
901+
#[cfg(test)]
902+
mod tests {
903+
use super::*;
904+
905+
#[test]
906+
fn parses_partition_audit_month_horizon() {
907+
let cli = Cli::try_parse_from(["buzz-admin", "partition-audit", "--months-ahead", "6"])
908+
.expect("parse partition-audit command");
909+
assert!(matches!(
910+
cli.command,
911+
Command::PartitionAudit { months_ahead: 6 }
912+
));
913+
}
914+
}

‎crates/buzz-db/src/runtime/mod.rs‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -544,6 +544,9 @@ pub struct DbConfig {
544544
/// (env `BUZZ_DB_STATEMENT_TIMEOUT_MS`). `0` disables it and is the
545545
/// default because migrations and backfills may legitimately run long.
546546
pub statement_timeout_ms: u64,
547+
/// Whether every new writer-pool connection defaults all transactions to
548+
/// read-only. Intended for operator audit commands, never the relay pool.
549+
pub default_transaction_read_only: bool,
547550
}
548551

549552
impl Default for DbConfig {
@@ -564,6 +567,7 @@ impl Default for DbConfig {
564567
lock_timeout_ms: DEFAULT_LOCK_TIMEOUT_MS,
565568
idle_txn_timeout_ms: DEFAULT_IDLE_TXN_TIMEOUT_MS,
566569
statement_timeout_ms: 0,
570+
default_transaction_read_only: false,
567571
}
568572
}
569573
}
@@ -649,6 +653,7 @@ impl Db {
649653
let lock_timeout_ms = config.lock_timeout_ms;
650654
let idle_txn_timeout_ms = config.idle_txn_timeout_ms;
651655
let statement_timeout_ms = config.statement_timeout_ms;
656+
let default_transaction_read_only = config.default_transaction_read_only;
652657
let options = PgPoolOptions::new()
653658
.max_connections(config.max_connections)
654659
.min_connections(config.min_connections)
@@ -693,11 +698,17 @@ impl Db {
693698
if let Err(error) = sqlx::query(
694699
"SELECT set_config('lock_timeout', $1, false), \
695700
set_config('idle_in_transaction_session_timeout', $2, false), \
696-
set_config('statement_timeout', $3, false)",
701+
set_config('statement_timeout', $3, false), \
702+
set_config('default_transaction_read_only', $4, false)",
697703
)
698704
.bind(lock_timeout_ms.to_string())
699705
.bind(idle_txn_timeout_ms.to_string())
700706
.bind(statement_timeout_ms.to_string())
707+
.bind(if default_transaction_read_only {
708+
"on"
709+
} else {
710+
"off"
711+
})
701712
.execute(&mut *conn)
702713
.await
703714
{

‎crates/buzz-db/src/store/artifact_postgres_tests.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ impl Fixture {
2424
crate::migration::run_migrations(&pool).await.unwrap();
2525
}
2626
let db = Db::from_pool(pool);
27-
db.ensure_future_partitions(1).await.unwrap();
27+
db.ensure_future_partitions(1, true).await.unwrap();
2828
let community = CommunityId::from_uuid(Uuid::new_v4());
2929
sqlx::query("INSERT INTO communities(id,host) VALUES($1,$2)")
3030
.bind(community.as_uuid())

0 commit comments

Comments
 (0)