Skip to content

Commit 6878520

Browse files
TheSentinel454Elrondclaude
authored
buzz-db: check events created_at floor in app writers (phase 1) (#8325)
## What This is phase 1 of 2 for retiring the `events_created_at_floor` deferred constraint trigger and its `events_created_at_floor_guard()` function (migration 0021). The invariant now holds in the app writers. **The trigger, function and `buzz.created_at_floor` GUC arming all stay** as the backstop while pods run mixed versions. A later PR drops them. One helper, `replica_fence::guard_event_created_at_floor(&mut AdmittedTx, created_at, channel_id)`, runs on the admitted write connection. If `created_at < clock_timestamp() - CREATED_AT_FLOOR_SECS`, it rejects the write with the typed, non-retryable `DbError::CreatedAtFloorRejected`. It uses the same 960 s constant as the GUC and the fence wall. Channel-less rows skip it and cost no query. **Why checking at insert time is enough:** the fence wall is `min(oldest_xact_start, S) - floor - 5s`. A writer transaction that is open during the probe's activity scan is bounded by its own `xact_start`. A transaction that starts after the scan samples a clock later than `S`. So any `clock_timestamp()` sample taken inside the writing transaction is as strong as the trigger's commit-time sample. This needs no commit hook, no candidate-row tracking and no ordering against the TTL refresh, and `AdmittedTx::commit` is unchanged. | Site | Check placement | |---|---| | `store/event.rs` `insert_event_on` | after the `ON CONFLICT DO NOTHING` INSERT, only when a row was inserted, before follow-ups | | `store/event.rs` `insert_event_with_thread_metadata_tx` | after the INSERT, only when a row was inserted, before follow-ups and thread metadata | | `store/replaceable.rs` parameterized replacement | after the `ON CONFLICT DO NOTHING` INSERT, only when a row was inserted, inside the supersede savepoint (a rejection rolls back to it and restores the old head) | | `store/replaceable.rs` addressable replacement | after the `ON CONFLICT DO NOTHING` INSERT, only when a row was inserted, before follow-ups (the caller's rollback restores the old head) | | `store/channel_members.rs` `LockedMemberSnapshot::replace_member_event` | before the old head is superseded | **Writer inventory:** there are 7 production `INSERT INTO events` functions. The other two are structurally `channel_id = NULL`: `push.rs` `accept_lease_event` (kind 30350) and `relay_members.rs` `publish_nip43_membership_locked` (kind 13534). Both write a literal `NULL` for `channel_id` in their INSERT SQL, and the policy exempts them only while that NULL sits at the `channel_id` position. No production UPDATE writes `events.created_at` or `events.channel_id`, and the events `ON CONFLICT` clauses are `DO NOTHING`. **Client-visible outcome is unchanged.** The trigger's SQLSTATE 23514 surfaced as a generic DB error, which the relay maps to `IngestError::Internal` (WS `OK false` / HTTP 500). The new variant follows the same path. Relay mappings are untouched. **`scripts/seed-admin-dashboard.sh`** writes a channel event with `created_at = 1790000000` through raw psql, and it accepted any `PGHOST`. It now refuses non-local hosts (`localhost`, `127.0.0.1`, `::1` and Unix socket paths are still allowed), a non-empty `PGHOSTADDR` or `PGSERVICE`, a multi-host `PGHOST`, and any `PGDATABASE` that is not a plain name (psql treats URI and `key=value` `-d` values as connection strings that override `-h`). Both psql paths pass `-X`, so a psqlrc can't `\connect` elsewhere after the guard. ## Tests There are eleven new PG tests in `store::event_floor_postgres_tests`. Each runs in its own per-test database with `ALTER TABLE events DISABLE TRIGGER events_created_at_floor`, and a `pg_partition_tree` check asserts that every partition clone is disabled, so the trigger can't hide a missing app check. - `insert_event_floor_rejects_old_and_accepts_fresh`, `thread_metadata_event_floor_rejects_old_and_accepts_fresh`, `parameterized_event_floor_rejects_old_and_accepts_fresh`, `addressable_event_floor_rejects_old_and_accepts_fresh`, `member_snapshot_event_floor_rejects_old_and_accepts_fresh`: each rejects a too-old event with the typed error, confirms no row was written, then accepts a fresh event. - `insert_event_floor_accepts_expired_duplicate`, `thread_metadata_event_floor_accepts_expired_duplicate`: replaying an already-stored event whose `created_at` is now below the floor returns `Duplicate` (inserted = false), leaves one row and runs no thread follow-ups. - `parameterized_replay_of_soft_deleted_aged_event_is_a_duplicate`, `addressable_replay_of_soft_deleted_aged_event_is_a_duplicate`: replaying a replacement whose stored row is soft-deleted and has aged below the floor returns a duplicate, not a floor rejection, as the trigger did (it never fired on `ON CONFLICT`). - `cluster_global_event_floor_held_transaction_cannot_write_below_probe_wall`: runs a real heartbeat probe while a writer transaction is open. It asserts that the wall is bounded by that writer's `xact_start`, that a below-wall event is rejected, and that a near-floor event sampled after the probe lands above the wall. - `event_floor_samples_statement_clock_after_transaction_start`: waits on the database until a boundary timestamp that was valid at BEGIN expires (normally ≤1.1 s), so `now()` would pass where `clock_timestamp()` fails. Source policy (`observability_source.rs`): every channel-bearing production events INSERT in any crate's `src` must call the helper, no production UPDATE may assign `created_at` or `channel_id` (including `ONLY`, schema-qualified, aliased and row-constructor forms; quoted identifiers are normalized), the helper must use `clock_timestamp()` and the shared constant, and the seed script must refuse a remote host. Negative fixtures cover each rule. Each check was confirmed to fail when its fix is removed. All of these mutants failed on assertions: - removing the helper from all five writers, which failed all seven PG tests - switching `clock_timestamp()` to `now()`, which failed both the clock PG test and the clock policy - removing one call - adding a `created_at` UPDATE - disabling the policy self-test - removing the seed guard - (review round 2) restoring the pre-INSERT check placement, which failed both duplicate-replay tests; the round-1 UPDATE matcher, quoted-identifier normalization, buzz-db-only scan, NIP-43 bind exemption, process-wide canvas base and each seed override were each restored and failed their tests - (review round 3) restoring the round-2 seed script, which failed the extended seed test on its URI and key/value `PGDATABASE` cases; restoring the script without `-X`, which failed the control's `-X` assertion **Fixture changes:** about 40 existing buzz-db PG tests seeded historical rows (fixed old timestamps) through the serving writers. The trigger never fired on them because the test pool is unarmed. They now seed through a `#[cfg(test)]` raw-SQL helper, `test_support::insert_historical_event` (and `insert_historical_event_with_thread_metadata` for thread fixtures), which keeps admission, follow-ups and mentions. The canvas CAS fixtures keep their logical seconds and ties but each test passes its own fresh base. The existing trigger backstop tests still exercise raw armed INSERT/UPDATE plus COMMIT returning 23514. The new PG tests are selected by the `postgres-ci` filter `(test(/postgres_tests::/) or binary(/^postgres_/)) and not test(/(^|::)external_infra[^:]*::/)`. The `cluster_global_` test also joins the existing serial group. ## Behavior notes - Prepared transactions (2PC) are covered only by the retained trigger, which runs at PREPARE. The insert-time app guard runs earlier, and no app writer issues `PREPARE TRANSACTION`. The comment in `replica_fence.rs` now says so, so the trigger drop has to keep this case covered. - One extra round trip for each newly inserted channel event. Duplicate replays add none. - Ordinary and thread event writes check the floor only for newly inserted rows, before follow-ups, mentions or thread metadata. Replaying an already-stored id stays `Duplicate` even when its timestamp has aged below the floor, as with the trigger. (Internal writers such as the admin outbox worker do replay relay-signed events with their original `created_at`.) A too-old new row returns the non-retryable `CreatedAtFloorRejected`, and the owning transaction rolls back; every current caller propagates the error and drops the transaction. The transaction-composition seam documents that contract. - Replacement writers check after their INSERT, only when it inserted a row, so a replay of any stored id (live or soft-deleted) stays a duplicate. The locked roster writer checks before the old head is superseded. - The trigger is stricter for a row that ages below the floor between the app check and COMMIT. The proof above doesn't need that strictness, and phase 2 gives it up. ## What got simpler The NIP-43 insert drops a bind for a literal `NULL`, the floor and follow-up policies share one workspace walk, and the canvas fixtures lose their process-wide `OnceLock`. No commit-time machinery was added: no candidate tracking, commit hook, cleanup protocol or new lock. Test-only historical writes now go through one helper instead of being scattered through the serving APIs. Nothing could be deleted in this phase because mixed-version pods still rely on the trigger. **Phase 2** drops the trigger and function, the GUC arming, `verify_floor_guard_catalog` / `verify_floor_guard_behavior` and the schema reconciliation entries. Its deploy gate: the whole fleet on this binary, no rollback expected, and the replica breaker held closed through the drop. **Identified debt (pre-existing, outside this diff):** three `buzz-relay` `handlers::group_state_postgres_tests::*` tests fail with `ensure community: Sqlx(PoolTimedOut)` on the Blox host. They fail the same way on a clean checkout of the base (`state.rs` test pool, 100 ms acquire timeout). `writer_pool_rejects_non_read_committed_database_default` and `writer_pool_metrics_stop_after_session_timeout_setup_failure` hang on that host at base too. ## Verification Head `b046df76d` (after the placement fix and main's newer test fixtures): on Blox, fmt, clippy and buzz-db unit (241/241) pass, and a filtered buzz-db PG lane over the floor tests and every converted fixture passes 88/88. CI is running at this head. Earlier full gate, at tree `67c4fe5e1` + this diff, before the placement fix: | Gate | Result | |---|---| | `cargo fmt --all -- --check` | pass | | `cargo clippy --workspace --all-targets -- -D warnings` | pass | | `cargo nextest run -p buzz-db --lib` | 167/167 | | `cargo nextest run -p buzz-db --test observability_source` | 22/22 | | `scripts/test-postgres-test-discovery.sh` | pass | | `scripts/postgres-test-run.sh -p buzz-db` (excluding the two host hangs) | 517/517 | | `scripts/postgres-test-run.sh -p` buzz-admin / audit / deletion / push-gateway / search / workflow | all rc 0 (4, 7, 11, 11, 20, 2) | | `scripts/postgres-test-run.sh -p buzz-relay` | 402 pass, 3 fail (the pre-existing `PoolTimedOut` failures above) | Local gates used a reverted `rustls` lock bump, because the build mirror lacks 0.23.46. The committed `Cargo.lock` is unchanged, and CI uses the real lock. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Signed-off-by: Luke Tornquist <[email protected]> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 473d592 commit 6878520

20 files changed

Lines changed: 1187 additions & 176 deletions

‎crates/buzz-db/src/error.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,13 @@ pub enum DbError {
114114
#[error("invalid timestamp: {0}")]
115115
InvalidTimestamp(i64),
116116

117+
/// A channel event is too old for replica-fence completeness. Retrying the
118+
/// same signed event cannot make its timestamp newer. Like the mixed-version
119+
/// trigger's SQLx error, this remains a database error at the relay boundary
120+
/// (WS OK false / HTTP 500), rather than a different ingest rejection class.
121+
#[error("database error: events.created_at is below the replica-fence floor")]
122+
CreatedAtFloorRejected,
123+
117124
/// A roster mutation would remove the last effective relay Operator,
118125
/// leaving no one able to administer the deployment through the API.
119126
/// The transaction is rolled back and the caller must add a replacement

‎crates/buzz-db/src/runtime/replica_fence.rs‎

Lines changed: 47 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@
33
//! A read replica may serve a cursor page only when every row the page could
44
//! contain is provably present on the replica. The proof rests on two parts:
55
//!
6-
//! 1. **Commit-time floor** (migration 0021): a deferred constraint trigger
7-
//! aborts, at COMMIT, any transaction inserting a channel-bearing `events`
8-
//! row with `created_at` more than `floor` seconds before commit time
9-
//! (`clock_timestamp()`, evaluated inside commit processing). Enforcement
10-
//! is armed per session via the `buzz.created_at_floor` GUC, which the
11-
//! relay's writer pool sets on every connection.
6+
//! 1. **Insert-time floor**: supported channel-event writers check `created_at`
7+
//! against database `clock_timestamp()` on their admitted transaction for
8+
//! each new row. Duplicate INSERTs need no floor check. This is the primary
9+
//! guard. Migration 0021's deferred trigger and the writer-pool
10+
//! `buzz.created_at_floor` GUC remain the mixed-version backstop until phase 2;
11+
//! they also reject rows that age below the floor before COMMIT.
1212
//! 2. **Ordered heartbeat handshake** (this module): on one pinned writer
1313
//! connection, one transaction first takes the exclusive replica-floor
1414
//! advisory lock (draining compliant shared-lock channel writers), then
@@ -25,7 +25,7 @@
2525
//! commit, so the replica session has replayed it;
2626
//! (b) open at the activity scan — represented by `xact_start`, so it is
2727
//! bounded by the `oldest_xact_start` term;
28-
//! (c) started after the activity scan — its deferred floor guard runs
28+
//! (c) started after the activity scan — its insert-time app guard runs
2929
//! after `S`, so it cannot commit a row with
3030
//! `created_at < S - floor`.
3131
//! There is no fourth bucket. Each committed token `M` therefore proves a
@@ -57,9 +57,9 @@
5757
//! keeps failing, the newest entry ages past [`FENCE_STALENESS`] and the
5858
//! fence closes.
5959
//!
60-
//! Operational bypasses (sessions without the GUC, `session_replication_role
61-
//! = replica` restores) are outside the proof by design and require holding
62-
//! the fence closed for their duration; see `migrations/0021`.
60+
//! Raw SQL without the armed GUC and `session_replication_role = replica`
61+
//! restores bypass the supported app writers. These operational bypasses are
62+
//! outside the proof and require holding the fence closed; see `migrations/0021`.
6363
6464
use std::collections::VecDeque;
6565
use std::sync::{Arc, Mutex};
@@ -73,14 +73,46 @@ use uuid::Uuid;
7373

7474
use buzz_datastore_tracing::datastore_span;
7575

76-
/// Seconds of `created_at` history the commit-time floor guard tolerates.
76+
/// Seconds of `created_at` history the app and mixed-version trigger tolerate.
7777
///
7878
/// Must exceed the relay's ingest envelope (±900 s) by enough slack that a
7979
/// legitimately accepted event still commits within the floor even under
8080
/// slow validation/lock waits. The writer pool arms the guard with this value
8181
/// and the fence subtracts it; the two uses must never diverge.
8282
pub const CREATED_AT_FLOOR_SECS: i64 = 960;
8383

84+
/// Enforce the channel-event floor on the admitted INSERT connection.
85+
///
86+
/// One SELECT keeps the five existing INSERT shapes intact. Every writer
87+
/// checks after its INSERT, only when it inserted a row, before follow-ups, so
88+
/// replaying a stored id (even a soft-deleted one) stays a duplicate.
89+
/// Replacements roll back the supersede with it (savepoint or caller).
90+
/// The caller must propagate failure and roll back the write transaction. Its
91+
/// database clock is sampled after BEGIN: transactions open at the probe scan
92+
/// are bounded by xact_start; transactions starting later sample after S. No
93+
/// commit hook, candidate tracking or ordering against TTL refresh is needed.
94+
/// NULL-channel rows never enter covered channel windows and need no query.
95+
pub(crate) async fn guard_event_created_at_floor(
96+
tx: &mut crate::AdmittedTx,
97+
created_at: DateTime<Utc>,
98+
channel_id: Option<Uuid>,
99+
) -> crate::Result<()> {
100+
if channel_id.is_none() {
101+
return Ok(());
102+
}
103+
let below_floor: bool = sqlx::query_scalar(
104+
"SELECT $1::timestamptz < clock_timestamp() - make_interval(secs => $2::double precision)",
105+
)
106+
.bind(created_at)
107+
.bind(CREATED_AT_FLOOR_SECS as f64)
108+
.fetch_one(tx.conn())
109+
.await?;
110+
if below_floor {
111+
return Err(crate::DbError::CreatedAtFloorRejected);
112+
}
113+
Ok(())
114+
}
115+
84116
/// Deployment-global advisory-lock key ordering compliant channel-event write
85117
/// transactions against the writer probe handshake.
86118
///
@@ -835,7 +867,10 @@ async fn sample_writer(
835867
// prepared they have left `pg_stat_activity` but can still commit
836868
// after the token. Their deferred floor guard already ran at PREPARE,
837869
// so `pg_prepared_xacts.prepared` bounds their rows exactly like
838-
// `xact_start`; fold it into the same minimum.
870+
// `xact_start`; fold it into the same minimum. Only the retained
871+
// deferred trigger provides that PREPARE-time check: the insert-time
872+
// app guard runs earlier, and no app writer issues PREPARE TRANSACTION.
873+
// Dropping the trigger must keep this case covered.
839874
let row = sqlx::query(
840875
r#"
841876
SELECT

‎crates/buzz-db/src/runtime/tests.rs‎

Lines changed: 41 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -913,27 +913,27 @@ fn signed_event_at(keys: &nostr::Keys, content: &str, secs: u64) -> nostr::Event
913913
.expect("sign event")
914914
}
915915

916+
// Routing tests intentionally model stored history and divergent replicas.
917+
// Raw fixture SQL keeps them independent of serving-time floor admission.
916918
async fn insert_top_level(pool: &PgPool, community: Uuid, channel: Uuid, ev: &nostr::Event) {
917-
let ts = chrono::DateTime::from_timestamp(ev.created_at.as_secs() as i64, 0).expect("valid ts");
918-
event::insert_event_with_thread_metadata(
919+
crate::test_support::insert_historical_event(
919920
pool,
920921
CommunityId::from_uuid(community),
921922
ev,
922-
Some(channel),
923-
Some(event::ThreadMetadataParams {
924-
event_id: ev.id.as_bytes(),
925-
event_created_at: ts,
926-
channel_id: channel,
927-
parent_event_id: None,
928-
parent_event_created_at: None,
929-
root_event_id: None,
930-
root_event_created_at: None,
931-
depth: 0,
932-
broadcast: true,
933-
}),
923+
channel,
924+
)
925+
.await;
926+
sqlx::query(
927+
"INSERT INTO thread_metadata (community_id,event_id,event_created_at,channel_id,depth,broadcast) \
928+
VALUES ($1,$2,to_timestamp($3),$4,0,true)",
934929
)
930+
.bind(community)
931+
.bind(ev.id.as_bytes().as_slice())
932+
.bind(ev.created_at.as_secs() as f64)
933+
.bind(channel)
934+
.execute(pool)
935935
.await
936-
.expect("insert top-level event");
936+
.expect("seed top-level metadata");
937937
}
938938

939939
async fn insert_thread_reply(
@@ -943,29 +943,37 @@ async fn insert_thread_reply(
943943
root: &nostr::Event,
944944
reply: &nostr::Event,
945945
) {
946-
let reply_ts =
947-
chrono::DateTime::from_timestamp(reply.created_at.as_secs() as i64, 0).expect("valid ts");
948-
let root_ts =
949-
chrono::DateTime::from_timestamp(root.created_at.as_secs() as i64, 0).expect("valid ts");
950-
event::insert_event_with_thread_metadata(
946+
crate::test_support::insert_historical_event(
951947
pool,
952948
CommunityId::from_uuid(community),
953949
reply,
954-
Some(channel),
955-
Some(event::ThreadMetadataParams {
956-
event_id: reply.id.as_bytes(),
957-
event_created_at: reply_ts,
958-
channel_id: channel,
959-
parent_event_id: Some(root.id.as_bytes()),
960-
parent_event_created_at: Some(root_ts),
961-
root_event_id: Some(root.id.as_bytes()),
962-
root_event_created_at: Some(root_ts),
963-
depth: 1,
964-
broadcast: false,
965-
}),
950+
channel,
951+
)
952+
.await;
953+
sqlx::query(
954+
"INSERT INTO thread_metadata \
955+
(community_id,event_id,event_created_at,channel_id,parent_event_id,parent_event_created_at, \
956+
root_event_id,root_event_created_at,depth,broadcast) \
957+
VALUES ($1,$2,to_timestamp($3),$4,$5,to_timestamp($6),$5,to_timestamp($6),1,false)",
966958
)
959+
.bind(community)
960+
.bind(reply.id.as_bytes().as_slice())
961+
.bind(reply.created_at.as_secs() as f64)
962+
.bind(channel)
963+
.bind(root.id.as_bytes().as_slice())
964+
.bind(root.created_at.as_secs() as f64)
965+
.execute(pool)
966+
.await
967+
.expect("seed reply metadata");
968+
sqlx::query(
969+
"UPDATE thread_metadata SET reply_count=reply_count+1,descendant_count=descendant_count+1, \
970+
last_reply_at=now() WHERE community_id=$1 AND event_id=$2",
971+
)
972+
.bind(community)
973+
.bind(root.id.as_bytes().as_slice())
974+
.execute(pool)
967975
.await
968-
.expect("insert reply");
976+
.expect("seed root counters");
969977
}
970978

971979
/// Composite thread cursor: 8-byte BE seconds + raw event id.

‎crates/buzz-db/src/runtime/tests/thread_window_postgres_tests.rs‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,7 @@ async fn thread_window_upper_fence_terminal_snapshot_and_fallback() {
8787
.sign_with_keys(&keys)
8888
.unwrap();
8989
for pool in [&writer, &replica] {
90-
event::insert_event(pool, community, &edit, Some(channel))
91-
.await
92-
.unwrap();
90+
crate::test_support::insert_historical_event(pool, community, &edit, channel).await;
9391
event::insert_event(pool, community, &deletion, None)
9492
.await
9593
.unwrap();

‎crates/buzz-db/src/store/channel_members.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -503,6 +503,12 @@ impl LockedMemberSnapshot {
503503
}
504504
let tags = serde_json::to_value(&event.tags)?;
505505
validate_channel_roster_snapshot(&mut self.tx, channel_id, &tags).await?;
506+
crate::replica_fence::guard_event_created_at_floor(
507+
&mut self.tx,
508+
created_at,
509+
Some(channel_id),
510+
)
511+
.await?;
506512
sqlx::query(superseded_with_mentions_sql!(
507513
"UPDATE events SET deleted_at = NOW() WHERE community_id = $1 AND kind = $2 AND pubkey = $3 AND channel_id = $4 AND deleted_at IS NULL"
508514
))

0 commit comments

Comments
 (0)