Skip to content

Commit d0fd96f

Browse files
TheSentinel454codexElrond
authored
BUZZ-175: Route serving event writes through the tenant lock (#7828)
## Summary Route every serving event write through one community-admitted transaction entry point. `Db::begin_event_write_transaction(community)` is now the only `Db` constructor for event-write transactions: it takes the shared community admission lock before returning, so callers that use it cannot take domain or row locks ahead of tenant admission, and a quiescing community rejects the write at entry. Database triggers and commit-time fences remain the authoritative backstop. This migrates normal, replaceable, roster, relay membership, reaction, reminder, admin-delete, push-owned, artifact-accept, huddle-join (48101), workflow-deletion, and command-executor event writes. Coupled mention writes stay in the same transaction; pool-level `insert_mentions` now opens its own admitted transaction. **Review fixes (since `950d5f6`):** - The unguarded event-write constructor is gone. Removed with it: the deprecated `begin_transaction` alias, the test-only replica-floor constructor, and `begin_community_write_transaction` (now identical to the single constructor). Artifact accept, the huddle join, and both workflow-deletion paths now admit before any domain or row lock; `persist_command_event` no longer hand-rolls `guard_transaction`; read-only `query_artifacts` no longer opens an event-write transaction. - `insert_channel_head_checked` is now `insert_canvas_head_checked`: it rejects non-canvas kinds before opening a transaction and always takes the coordinate lock. - Source policy now also scans `buzz-relay`: any function that hands a transaction to an event-write helper must have opened it through the admitted constructor. The scan drops each `#[cfg(test)]` item instead of stopping at the first one: it tracks brace depth and ends the item on a line ending in `;` or `}` (ignoring a trailing `//` comment, even after a string that contains `//`), with a column-0 `}` as fallback. Fixtures pin a unit struct, a multi-line static, a `const { … }` block, a struct with fields, a module, and one-line items with trailing comments, including after `"https://…"` and `"ws://…"` strings; a raw opener after each must still be flagged. Known limit: braces inside string or char literals or comments are still counted, and only `//` trailing comments are recognized, so a test-only item with unbalanced braces in a literal or comment, or with a trailing `/* */`, would hide what follows; none exists today. The scan checks that admission is present; the per-path PostgreSQL tests pin that it comes before domain locks. - `just test-unit` (and the non-nextest fallback in `scripts/run-tests.sh`) now runs `buzz-db --test observability_source`. Before this, no CI job ran these source-policy tests: the unit lane ran `buzz-db --lib` only and the PostgreSQL lane runs only ignored tests. - Renamed or removed since `950d5f6`: `begin_transaction`, `begin_community_write_transaction`, and the replica-floor test constructor are gone; `insert_channel_head_checked` → `insert_canvas_head_checked`; test helper `quiesce_community_for_tests` now delegates to `main`'s `set_deletion_state`, which reads the real `deletion_fence_generation` instead of hard-coding `'0'`; `runtime::insert_mentions` is pinned to the admitted constructor by `pool_level_insert_mentions_opens_the_tenant_local_chokepoint`. **What got simpler:** four transaction constructors collapse to one, and the hand-rolled admission guard is deleted. The compiler rejects callers of the removed constructors, but a transaction opened from `Db::pool()` can still reach the public `*_in_transaction` helpers; only the source-policy scan and the database fences catch that until BUZZ-254. **Behavior note:** #7932 made `insert_event` skip the transaction for kinds that are not listener mentions. Every serving event write now has to run in a transaction that holds the community admission lock, so `insert_event` always opens the admitted transaction again. `main`'s `soft_delete_event_and_update_thread` / `_in_tx` split and its removal-marker guard are kept; the outer function now opens the admitted transaction. ### Related issue [BUZZ-175](https://linear.app/squareup/issue/BUZZ-175). Caller-migration stage following #7706 (merged); rebased onto `main` at `5173fad66`. Follow-up: [BUZZ-254](https://linear.app/squareup/issue/BUZZ-254) makes event-write helpers require a type-level admitted transaction. **Known debt, out of scope:** `create_channel`, `update_channel`, and `insert_thread_metadata` write channel and thread rows (not events) outside the admitted constructor. ### Testing At `efe4a94623dd69973780a3c39f16dd972cf7cb76` on a cloud workstation: fmt; `buzz-db` clippy (all targets, `-D warnings`); `observability_source` 14/14; `just --dry-run test-unit` includes the new step. Reverting just the end-of-item check to an earlier rule makes a trailing-comment fixture fail. At `eaf190ae59d1595741c3d19dc4c23ce492b5f4d4` (the commit after it only changes this test file, `Justfile`, and `scripts/run-tests.sh`): full GitHub CI green; on the workstation, clippy for `buzz-db` and `buzz-relay`, ordinary `buzz-db` 154/154, `just test-unit` green, and the PostgreSQL lane 805 of 806 (exception below). An end-to-end run against a local relay and the built desktop app covered artifact accept, huddle join, and workflow deletion for both a healthy and a quiescing community, plus p-tag reaction rollback and replay of the same signed event. New tests: one fenced-community test per migrated path (artifact accept, huddle join, both workflow deletions) puts the community into `quiescing`, holds the lock the path takes next, and asserts rejection at entry with nothing stored; a DB-free test pins the non-canvas rejection. Exceptions: - `runtime::postgres_tests::writer_pool_rejects_non_read_committed_database_default` hangs on that workstation's local Postgres 16 at both this head and an earlier `main` base, `f0eb5575f` (2/2 runs each). The test, `Db::new`, and the pool setup are unchanged by this PR, and the test passes in GitHub CI. Live traces put the hang in the client, not the database: the 1 s acquire timeout stops firing after sqlx's `after_connect` rejection path under tokio's `current_thread` runtime. No database lock waits were observed. - Two `observability_source` assertions were already failing on `main`, unnoticed because no CI job ran that test binary: a stale `fetch_optional` pin after #8005, and lease-seam attribution that now lives in the `bounded_serving_lease_sql` helper. Both are corrected here, and the binary now runs in `just test-unit`. Generated with Codex --------- Signed-off-by: tornquist <[email protected]> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <[email protected]> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent 9322289 commit d0fd96f

25 files changed

Lines changed: 2536 additions & 283 deletions

‎Justfile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,11 @@ test-unit:
397397
# #[ignore]d, so --lib runs only the infra-free set. Without this gate a
398398
# stray file in migrations/ or a broken lint ships green.
399399
cargo nextest run -p buzz-db --lib
400+
# buzz-db source-policy tests: infra-free scans of the crate sources
401+
# that enforce the event-write admission chokepoint, metric/provenance
402+
# contracts, and their fixtures. They live in an integration-test
403+
# binary, so `--lib` above does not run them.
404+
cargo nextest run -p buzz-db --test observability_source
400405
# Storage accounting crosses three crates whose focused regression
401406
# suites are otherwise absent from the infra-free unit lane.
402407
cargo nextest run -p buzz-media --lib \

‎crates/buzz-db/src/lib.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,9 @@ pub const DB_CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionO
5858
/// Raw Prometheus series ceiling per pod for connection-step telemetry.
5959
pub const DB_CONNECTION_RAW_SERIES_PER_POD: usize = runtime::CONNECTION_RAW_SERIES_PER_POD;
6060
pub(crate) use runtime::{
61-
insert_mentions_in_transaction, observability, route_proof, ReadSessionInner, RouteDecision,
62-
RoutePredicate,
61+
begin_community_event_write_transaction,
62+
begin_community_event_write_transaction_with_legacy_metrics, insert_mentions_in_transaction,
63+
observability, route_proof, ReadSessionInner, RouteDecision, RoutePredicate,
6364
};
6465
pub use store::{
6566
admin_moderation, allowlist, api_token, archived_identities, artifact, channel,

‎crates/buzz-db/src/runtime/mod.rs‎

Lines changed: 87 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -21,19 +21,26 @@ use buzz_core::{CommunityId, StoredEvent};
2121

2222
/// Extract p-tag mentions from an event and insert into the `event_mentions` table.
2323
///
24-
/// This pool-owning wrapper propagates failures to its caller. Replacement writes
25-
/// use the transaction-bound helper below so event storage and mention indexing
26-
/// commit or roll back together. Duplicate inserts are silently skipped with
27-
/// `INSERT ... ON CONFLICT DO NOTHING`.
24+
/// This pool-owning wrapper indexes mentions in its own admitted transaction,
25+
/// after the event has already committed. Writes that need event storage and
26+
/// mention indexing to commit or roll back together should open
27+
/// `begin_community_event_write_transaction` and call
28+
/// `insert_mentions_in_transaction` instead.
29+
///
30+
/// Duplicate inserts are silently skipped with `INSERT ... ON CONFLICT DO
31+
/// NOTHING`.
2832
pub async fn insert_mentions(
2933
pool: &PgPool,
3034
community_id: CommunityId,
3135
event: &nostr::Event,
3236
channel_id: Option<Uuid>,
3337
) -> Result<()> {
34-
let connection =
35-
observability::acquire_writer(pool, observability::WriterOperation::EventWrite).await?;
36-
let mut tx = sqlx::Transaction::begin(connection, None).await?;
38+
let mut tx = begin_community_event_write_transaction(
39+
pool,
40+
community_id,
41+
observability::WriterOperation::EventWrite,
42+
)
43+
.await?;
3744
insert_mentions_in_transaction(&mut tx, community_id, event, channel_id).await?;
3845
tx.commit().await?;
3946
Ok(())
@@ -121,6 +128,63 @@ pub(crate) async fn insert_mentions_in_transaction(
121128
Ok(())
122129
}
123130

131+
/// Start a tenant-local event-write transaction and take the shared community
132+
/// deletion lock before any serving mutation.
133+
async fn begin_community_event_write_transaction_with_metric_population(
134+
pool: &PgPool,
135+
community: CommunityId,
136+
operation: observability::WriterOperation,
137+
metric_population: CommunityEventWriteMetricPopulation,
138+
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
139+
let connection = match metric_population {
140+
CommunityEventWriteMetricPopulation::TypedOnly => {
141+
observability::acquire_writer(pool, operation).await?
142+
}
143+
CommunityEventWriteMetricPopulation::LegacyCompatibility => {
144+
observability::acquire_writer_with_legacy_metrics(pool, operation).await?
145+
}
146+
};
147+
let mut tx = sqlx::Transaction::begin(connection, None).await?;
148+
deletion::DeletionStore::new(pool.clone())
149+
.guard_transaction(&mut tx, community)
150+
.await?;
151+
Ok(tx)
152+
}
153+
154+
#[derive(Clone, Copy)]
155+
enum CommunityEventWriteMetricPopulation {
156+
TypedOnly,
157+
LegacyCompatibility,
158+
}
159+
160+
pub(crate) async fn begin_community_event_write_transaction(
161+
pool: &PgPool,
162+
community: CommunityId,
163+
operation: observability::WriterOperation,
164+
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
165+
begin_community_event_write_transaction_with_metric_population(
166+
pool,
167+
community,
168+
operation,
169+
CommunityEventWriteMetricPopulation::TypedOnly,
170+
)
171+
.await
172+
}
173+
174+
pub(crate) async fn begin_community_event_write_transaction_with_legacy_metrics(
175+
pool: &PgPool,
176+
community: CommunityId,
177+
operation: observability::WriterOperation,
178+
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
179+
begin_community_event_write_transaction_with_metric_population(
180+
pool,
181+
community,
182+
operation,
183+
CommunityEventWriteMetricPopulation::LegacyCompatibility,
184+
)
185+
.await
186+
}
187+
124188
/// Database handle. Clone is cheap (Arc-backed pool).
125189
#[derive(Clone, Debug)]
126190
pub struct Db {
@@ -1220,60 +1284,29 @@ impl Db {
12201284
})
12211285
}
12221286

1223-
/// Begin a database transaction for atomic multi-statement operations.
1287+
/// Begin an event-write transaction admitted for `community`.
1288+
///
1289+
/// This is the only `Db` constructor for event-write transactions. It takes
1290+
/// the shared community admission lock before returning, so callers that use
1291+
/// it cannot take domain or row locks ahead of tenant admission, and a
1292+
/// quiescing community rejects the write at entry rather than at its first
1293+
/// fenced statement. The compiler does not enforce this: a transaction
1294+
/// opened from [`Db::pool`] can still reach the public `*_in_transaction`
1295+
/// helpers, and only the source-policy tests and the commit-time database
1296+
/// fences, which remain the authoritative backstop, catch it.
12241297
///
12251298
/// Returns a `'static` transaction because `PgPool` is `Arc`-backed internally.
12261299
/// The transaction holds an owned pool handle, not a borrow.
12271300
pub async fn begin_event_write_transaction(
12281301
&self,
1302+
community: CommunityId,
12291303
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
1230-
let connection = observability::acquire_writer_with_legacy_metrics(
1304+
begin_community_event_write_transaction_with_legacy_metrics(
12311305
&self.pool,
1306+
community,
12321307
observability::WriterOperation::EventWrite,
12331308
)
1234-
.await?;
1235-
sqlx::Transaction::begin(connection, None)
1236-
.await
1237-
.map_err(Into::into)
1238-
}
1239-
1240-
/// Begin an event-write transaction and guard its community against
1241-
/// concurrent deletion.
1242-
pub async fn begin_community_write_transaction(
1243-
&self,
1244-
community: CommunityId,
1245-
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
1246-
let mut tx = self.begin_event_write_transaction().await?;
1247-
self.deletion_store()
1248-
.guard_transaction(&mut tx, community)
1249-
.await?;
1250-
Ok(tx)
1251-
}
1252-
1253-
/// Begin an event-write transaction that takes the shared replica-floor
1254-
/// advisory lock.
1255-
///
1256-
/// This is a lock-ordering foundation only. Floor correctness remains
1257-
/// authoritative at commit time via the existing trigger/GUC contract.
1258-
pub async fn begin_replica_floor_locked_event_write_transaction(
1259-
&self,
1260-
) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
1261-
let mut tx = self.begin_event_write_transaction().await?;
1262-
sqlx::query("SELECT pg_advisory_xact_lock_shared($1)")
1263-
.bind(replica_fence::REPLICA_FLOOR_LOCK_KEY)
1264-
.execute(&mut *tx)
1265-
.await?;
1266-
Ok(tx)
1267-
}
1268-
1269-
/// Begin an event-write transaction through the pre-operation API name.
1270-
///
1271-
/// New callers should use [`Self::begin_event_write_transaction`] so the
1272-
/// semantic intent is explicit. This alias preserves the crate's public
1273-
/// API while emitting the same operation-aware and compatibility metrics.
1274-
#[deprecated(note = "use Db::begin_event_write_transaction")]
1275-
pub async fn begin_transaction(&self) -> Result<sqlx::Transaction<'static, sqlx::Postgres>> {
1276-
self.begin_event_write_transaction().await
1309+
.await
12771310
}
12781311

12791312
/// Insert an event while holding and validating an admitted serving-write
@@ -1305,6 +1338,8 @@ impl Db {
13051338
self.deletion_store()
13061339
.guard_transaction_with_serving_lease(&mut tx, lease)
13071340
.await?;
1341+
event::acquire_canvas_event_write_lock_if_needed(&mut tx, community_id, event, channel_id)
1342+
.await?;
13081343
let result = event::insert_event_with_thread_metadata_tx(
13091344
&mut tx,
13101345
community_id,

‎crates/buzz-db/src/runtime/observability.rs‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1479,14 +1479,9 @@ mod tests {
14791479
let _ = writer_db
14801480
.query_events_routed("pool_operation_matrix_writer", &query)
14811481
.await;
1482-
let write_tx = writer_db
1483-
.begin_event_write_transaction()
1484-
.await
1485-
.expect("event-write semantic entry point begins a real transaction");
1486-
write_tx
1487-
.rollback()
1488-
.await
1489-
.expect("rollback operation-label fixture");
1482+
// The scope is not a live community, so admission rejects it after the
1483+
// writer acquisition has been attributed to `event_write`.
1484+
let _ = writer_db.begin_event_write_transaction(test_scope).await;
14901485
let _ = writer_db
14911486
.is_community_active_for_maintenance(test_scope)
14921487
.await;

‎crates/buzz-db/src/runtime/tests.rs‎

Lines changed: 85 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ async fn setup_db() -> Db {
7878
}
7979

8080
#[tokio::test]
81-
async fn begin_transaction_compatibility_alias_is_preserved() {
81+
async fn event_write_transaction_preserves_legacy_acquisition_metrics() {
8282
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
8383

8484
let pool = sqlx::postgres::PgPoolOptions::new()
@@ -90,8 +90,9 @@ async fn begin_transaction_compatibility_alias_is_preserved() {
9090
let snapshotter = recorder.snapshotter();
9191
let _guard = metrics::set_default_local_recorder(&recorder);
9292

93-
#[allow(deprecated)]
94-
let result = db.begin_transaction().await;
93+
let result = db
94+
.begin_event_write_transaction(CommunityId::from_uuid(Uuid::new_v4()))
95+
.await;
9596
assert!(matches!(
9697
result,
9798
Err(DbError::Sqlx(sqlx::Error::PoolClosed))
@@ -169,6 +170,69 @@ fn nip43_reconciliation_compatibility_alias_is_preserved() {
169170
let _ = call;
170171
}
171172

173+
#[tokio::test]
174+
async fn community_write_transaction_compatibility_metrics_are_limited_to_legacy_entrypoints() {
175+
use metrics_util::debugging::DebuggingRecorder;
176+
177+
let pool = sqlx::postgres::PgPoolOptions::new()
178+
.connect_lazy(&crate::test_support::database_url())
179+
.expect("construct lazy compatibility pool");
180+
pool.close().await;
181+
let db = Db::from_pool(pool);
182+
let recorder = DebuggingRecorder::new();
183+
let snapshotter = recorder.snapshotter();
184+
let _guard = metrics::set_default_local_recorder(&recorder);
185+
let community = CommunityId::from_uuid(Uuid::new_v4());
186+
187+
let direct = begin_community_event_write_transaction(
188+
&db.pool,
189+
community,
190+
observability::WriterOperation::EventWrite,
191+
)
192+
.await;
193+
assert!(matches!(
194+
direct,
195+
Err(DbError::Sqlx(sqlx::Error::PoolClosed))
196+
));
197+
assert_eq!(
198+
legacy_acquisition_count(&snapshotter.snapshot().into_vec()),
199+
0,
200+
"typed-only tenant-local chokepoint must not emit legacy compatibility metrics"
201+
);
202+
203+
let legacy = db.begin_event_write_transaction(community).await;
204+
assert!(matches!(
205+
legacy,
206+
Err(DbError::Sqlx(sqlx::Error::PoolClosed))
207+
));
208+
assert_eq!(
209+
legacy_acquisition_count(&snapshotter.snapshot().into_vec()),
210+
1,
211+
"Db::begin_event_write_transaction must preserve the legacy compatibility population"
212+
);
213+
}
214+
215+
fn legacy_acquisition_count(
216+
snapshot: &[(
217+
metrics_util::CompositeKey,
218+
Option<metrics::Unit>,
219+
Option<metrics::SharedString>,
220+
metrics_util::debugging::DebugValue,
221+
)],
222+
) -> u64 {
223+
snapshot
224+
.iter()
225+
.filter_map(|(key, _, _, value)| {
226+
(key.key().name() == "buzz_db_pool_acquisitions_total")
227+
.then_some(value)
228+
.map(|value| match value {
229+
metrics_util::debugging::DebugValue::Counter(value) => *value,
230+
_ => panic!("legacy acquisitions must be a counter"),
231+
})
232+
})
233+
.sum()
234+
}
235+
172236
#[tokio::test]
173237
#[ignore = "requires Postgres"]
174238
async fn readiness_check_distinguishes_pool_exhaustion_from_success() {
@@ -3118,6 +3182,22 @@ async fn armed_pool_rejects_old_channel_inserts_through_public_api() {
31183182
db.pool.close().await;
31193183
}
31203184

3185+
/// A writer transaction holding the shared replica-floor advisory lock, the
3186+
/// shape a floor-compliant writer takes before the exclusive probe can run.
3187+
async fn begin_replica_floor_locked_writer(db: &Db) -> sqlx::Transaction<'static, sqlx::Postgres> {
3188+
let mut tx = db
3189+
.pool
3190+
.begin()
3191+
.await
3192+
.expect("begin floor-guarded writer tx");
3193+
sqlx::query("SELECT pg_advisory_xact_lock_shared($1)")
3194+
.bind(crate::replica_fence::REPLICA_FLOOR_LOCK_KEY)
3195+
.execute(&mut *tx)
3196+
.await
3197+
.expect("take shared replica-floor lock");
3198+
tx
3199+
}
3200+
31213201
#[tokio::test]
31223202
#[ignore = "requires Postgres"]
31233203
async fn replica_floor_writer_transaction_holds_shared_lock() {
@@ -3137,10 +3217,7 @@ async fn replica_floor_writer_transaction_holds_shared_lock() {
31373217
.await
31383218
.expect("connect armed Db");
31393219

3140-
let writer = db
3141-
.begin_replica_floor_locked_event_write_transaction()
3142-
.await
3143-
.expect("open compliant floor-guarded writer tx");
3220+
let writer = begin_replica_floor_locked_writer(&db).await;
31443221

31453222
let mut shared_contender = db.pool.begin().await.expect("begin shared contender");
31463223
let shared_taken: bool = sqlx::query_scalar("SELECT pg_try_advisory_xact_lock_shared($1)")
@@ -3201,10 +3278,7 @@ async fn replica_floor_probe_waits_for_shared_writer_and_records_after_release()
32013278
.await
32023279
.expect("read token before probe");
32033280

3204-
let writer = db
3205-
.begin_replica_floor_locked_event_write_transaction()
3206-
.await
3207-
.expect("open compliant floor-guarded writer tx");
3281+
let writer = begin_replica_floor_locked_writer(&db).await;
32083282

32093283
let probe_pool = db.pool.clone();
32103284
let probe_fence = std::sync::Arc::clone(db.fence());

‎crates/buzz-db/src/store/artifact.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ impl Db {
9898
authorized_source: Option<Uuid>,
9999
relay_keys: &Keys,
100100
) -> Result<ArtifactOutcome> {
101-
let mut tx = self.begin_event_write_transaction().await?;
101+
let mut tx = self.begin_event_write_transaction(community).await?;
102102
sqlx::query("SET LOCAL statement_timeout='5s'")
103103
.execute(&mut *tx)
104104
.await?;

0 commit comments

Comments
 (0)