Skip to content

Commit 63496cc

Browse files
tlongwell-blocknpub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67dTyler Longwell
authored
feat(replica): portable heartbeat-token fence with snapshot-local reader routing (#3268)
## Summary Expands read-replica usage on the relay per the Rev 2 design (thread 39d1e174 in #buzz-read-only-replica-usage): replaces the Aurora-incompatible WAL-LSN read-side fence observation with a portable **heartbeat token**, makes the freshness proof **snapshot-local to the serving reader session**, and adds a default-off bounded-staleness gate for head fetches. ### Probe (writer side, unchanged ordering) The ordered writer scan is kept verbatim — `S = clock_timestamp()` → masked-visibility `pg_stat_activity` oldest-xact scan — and now ends by committing a heartbeat token **last** on the same pinned connection (single-row `UPDATE replica_heartbeat ... RETURNING token, epoch`, migration `0026`). The single-row update serializes all pods' probes, so tokens are globally commit-ordered; the three-bucket completeness argument carries over unchanged. Ring retains `(token, committed_at, fence_wall)`; epoch change resets the ring; a same-epoch token regression (restore adversary) clears the ring **and rotates the epoch on the writer** so pre-rewind readers fail the epoch check. Cadence 1s. ### Routing (snapshot-local proof) Every routed read opens `BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY` on a reader session and observes the heartbeat as the transaction's **first statement** — the proof's snapshot is exactly the snapshot the page, participants batch, and bridge aux closure read from (`ReadSession` carries the open transaction; drop = rollback). Fail-closed everywhere: begin failure, missing heartbeat row, epoch mismatch, token below the ring, over-budget entry → writer. - **Predicate B (cursor pages, default-on):** same completeness math as the existing fence — cursor timestamp must be ≤ the proved wall; thread candidate-terminal and above-wall pages re-run on the writer. - **Predicate A (head fetches, default-OFF):** gated by `BUZZ_REPLICA_HEAD_MAX_AGE_SECS` (0 = off, clamped to fence staleness). Bounded-stale head semantics are an explicit product decision — **do not enable anywhere without Tyler's backdated-event-semantics acceptance.** ### Observability `buzz_db_route_decision{path, decision, reason}` across all five paths, `buzz_db_replica_heartbeat_age_seconds` gauge, and per-decision debug logs carrying the proved token plus backend identity: `addr:port pid=N`, prefixed with `aurora_db_instance_identifier()` when the endpoint supports it (probed once per process on an autocommit checkout; SQLSTATE 42883 caches a definitive false; identity is evidence, never a routing gate). ## Review & verification - **Wren:** full review 9/9/9 at `5f81b10e5`; identity delta re-review approved at exact head `fedb46368` (90/90 unit, clippy `-D warnings`, fmt independently reproduced). - **Max:** local E2E **PASS** at `5f81b10e5` — isolated PG17 writer + two streaming standbys behind HAProxy; paused-reader legs split exactly as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR snapshot hardening observed in runtime logs. Evidence: `WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md`. - **My gates at head (same shell):** buzz-db 90 unit + 143 Postgres-gated green (scratch DBs); buzz-relay 767/768 — lone red is the pre-existing `mesh_demo` flake, red on base; clippy `-D warnings` + fmt clean. - Key regression tests: `routed_request_holds_one_snapshot_across_page_and_aux` (mutation-verified both directions), same-epoch rotation, capability-probe negative, head-gate truth table, divergent-fixture routing suite. ## Post-merge plan Merge publishes the immutable `sha-*` main image → bb-public PR pins it → ArgoCD sync → Max runs the production cursor-routing canary on Aurora (positive identity branch proven live). Head gate stays off. Client `since`-widening ships separately. --- ## Update — full read routing (Rev 6, commit `9fa3c9c0b`) Extends routing to the remaining read seams per `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` Rev 6 (thread cf5deba4 in #buzz-read-only-replica-usage). ### New routed seams — ALL deploy-default dark The new seams (`query_events_routed`, `query_events_routed_bounded`, `count_events_routed`, `get_events_by_ids_routed`, `query_feed_{mentions,needs_action,activity}_routed`) are **Bounded-only** and gated on `BUZZ_REPLICA_READ_MAX_AGE_MS`: unset ⇒ every new seam records `writer/disabled` and merging is a no-op. COUNT and feed/by-ids never take the covered arm (deletion visibility: covered bounds insert-completeness only). **Note:** the pre-existing cursor paths (channel windows, thread pages) are *not* gated by this env var — they route at B=0 today and that status quo is unchanged. ### Reader pool (D4/D5) - Lazy pool (`connect_lazy`, `min_connections(0)`): reader-down at boot can't crash the relay; a warn-only boot ping is the only boot-time visibility, and it primes the Aurora identity capability cache. Priming is an optimization only — the routed path itself spends a single acquire budget regardless (see the single-checkout fix below), because a boot ping that *fails* is correlated with exactly the reader-unavailable case the budget bound exists for. - `READER_ACQUIRE_TIMEOUT` = 150ms; a miss fails closed to the writer with reason **`reader_acquire_timeout`** — named for the mechanism, not a diagnosis. The budget includes cold connects and sqlx's `size` counts in-flight dials, so this metric alone does not distinguish contention from slow connection establishment (see `proved_reader` doc-comment for the runbook guidance; the pool gauges are 10s samples and are for capacity planning). - `BUZZ_DB_READ_POOL_SIZE` sizes the reader independently (invalid/0 inherits writer sizing); `read_pool_stats().max` reports the reader's own ceiling. ### Community isolation (formal-model question) Isolation is structural — an explicit `community_id = $n` predicate compiled into every query builder; no RLS, no session-GUC tenant state (zero `CREATE POLICY` across migrations). The `_on` variants reuse the exact same builders with only the executor swapped. Proven by a seven-seam two-community divergent-fixture test (`routed_reads_are_confined_to_the_requested_community`), mutation-tested by Dawn: all four single-predicate stubs killed; the mention-join feeds are defended in depth (three independent predicates) so only complete removal leaks there. ### Accepted limitation D6: client-side staleness on bounded reads (up to `_MS`) is accepted product behavior when the gate is enabled; gate stays off at merge. ## Update — single acquire budget per routed read (commit `dd26caa9f`) Max found (and Dawn independently reproduced, 302–330ms measured) that the boot-unavailable cold path spent **two** stacked `READER_ACQUIRE_TIMEOUT` budgets: the Aurora capability probe did its own `pool.acquire()` before `begin_with` acquired again. Fixed by acquiring **once** per routed read — the capability probe runs on the held connection (`reader_aurora_capability_on`) and the read-only `REPEATABLE READ` transaction begins on that same connection (`Transaction::begin` accepts a `PoolConnection` at `'static` via sqlx's `MaybePoolConnection`). Reason codes unchanged; capability still never negatively cached. Measured routed fallback: ~150ms (one budget). Ships with a PG-gated regression fixture (`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`, authored by Dawn): size-1 reader saturated, capability cache asserted cold, routes through `count_events_routed`, asserts writer answer + one-budget elapsed + `writer/reader_acquire_timeout` label (mutation-tested: `pool_busy` and `reader_validation_error` mutants both killed). It fails at 330ms on the previous commit and passes on this one. ### Verification at `dd26caa9f` (pinned toolchain rustc 1.95.0 via repo `bin/`; ambient rustc 1.89 fails sqlx resolution — always prepend `bin/`) - buzz-db: 94 unit + **150/151** PG-gated serial (`--test-threads=1`; suite is not parallel-safe against one Postgres). The 1 red is `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` — a **pre-existing** same-key-same-database advisory-lock collision with any live relay pointed at the shared `buzz` DB (the relay re-arms `USAGE_METRICS_LOCK_KEY` on a 300s tick). It passes on a private scratch DB with the relay still running — verified this run. Remedy is a scratch `TEST_DATABASE_URL`, **not** terminating lock holders: inspect `pg_locks`/`pg_stat_activity` and resolve the owning process first; an idle advisory holder may be a live dev relay. Pre-existing, not a #3268 blocker — filed as #3619. - buzz-relay: 773/773; clippy `-D warnings` + fmt clean. (`mesh_demo` echo test is a known pre-existing Redis-dependent flake — Dawn confirmed it fails identically on the base commit, untouched by this PR.) - Dawn: independent re-verification at `dd26caa9f` — byte-identical diff application confirmed, suite reproduced, and fixture proven to still discriminate (fails at 320ms with only the production fix reverted). Gate clear. Max: independent rig pass at `dd26caa9f` — 94/94 unit, **15/15 routing/fallback matrix** (cursor behavior, default-dark seams, seven-seam confinement, one-snapshot, dead-reader/no-URL fallback, hard-delete fail-closed, reader max), 773/773 relay, private-DB lock control. Gate clear; recommends merge + staged bb-block rollout (no-reader-URL no-op → cursor-only → observe `buzz_db_route_decision` → conservative nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`; rollback is config-only). ## Known gap — CI does not run the PG-backed fixtures (#3622) Dawn found post-sign-off (confirmed by Max and Eva at `dd26caa9f`): **every Postgres-backed fixture in this PR is `#[ignore]`d and no CI job selects it.** `Unit Tests` runs `cargo nextest run -p buzz-db --lib` (`Justfile:279-285`, ignored tests excluded); the only `--run-ignored ignored-only` invocations (`ci.yml:690`, `:702`) filter to `relay_invite::tests`. So the one-budget regression, the seven-seam isolation fixture, and the fence/floor-guard/fallback tests have zero automated execution — the verification record above is exhaustive but manual, at this exact SHA. Not introduced by this PR (the `#[ignore]` + narrow-filter pattern predates it; 34 ignored tests total) and not a merge blocker: all new seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set. But it changes the rollout gate — **do not set a nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS` in bb-block until CI enforces these fixtures.** Filed as #3622 (explicit CI selection against the existing backend-integration Postgres archive; ordering note: #3619 must land first or be excluded, since widening the filter would select the colliding usage-metrics lock test). ## Live redteam results (Max, real Helm + PG17 streaming replication at `dd26caa9f`) — #3643, #3644 Max deployed this exact SHA via the repo chart against a physical writer/standby pair and attacked it. **The PR's fenced routing passed every fault**: healthy baseline routes proven on the real standby; paused WAL replay moved head/cursor reads to `writer/stale`; B=0 moved routed reads to `writer/disabled`; reader-URL removal rolled back cleanly. The redteam also found a **pre-existing** production risk this PR does not create and does not fix: NIP-50 search builds its own eager, unfenced pool straight from `READ_DATABASE_URL` (`main.rs:389-402`, introduced by #2084, present in the deployed bb-block `sha-dd222a5` and bb-public `sha-22be8bb` images, and both production ESOs already template `READ_DATABASE_URL`). Measured: ~30s search stall on reader loss, acknowledged-but-unsearchable writes under replica lag (unaffected by `BUZZ_REPLICA_READ_MAX_AGE_MS`), fatal eager connect blocking pod startup when the reader is down, and `replica=true` FTS even at B=0. Filed as **#3643** (fix: pin FTS to the writer, or fence it like the routed seams) with **#3644** for the writer-pointing-reader-URL telemetry gap. Aurora's cluster-ro writer-fallback DNS softens the outage modes in prod but not the lag mode. **Revised ladder consequence:** step 2 ("add `READ_DATABASE_URL`, budget unset") is NOT a no-op with today's binary — it hands FTS an unfenced replica pool. Rollout order is now: merge (still a true no-op — bb-block/bb-public already run the unfenced-FTS code and this PR only adds dark seams) → fix #3643 → CI enforcement #3622 → then reader URL + budget per the original ladder, re-running Max's three deployment tests (reader-down latency, reader-down cold boot, paused-replay read-your-own-write search). ## Activation gate (consolidated) — merge is clear; ALL of the below precede any nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS` Six independent verification passes at `dd26caa9f` (Dawn ×2 incl. byte-level + defang check, Max matrix + live Helm redteam, Wren full CRUD regression + greenfield 0026 + 151/151 PG, Eva). Every induced fault was either handled correctly by this PR's machinery or traced to pre-existing main code. Remaining work gates **activation, not merge**: 1. **#3643** — unfenced NIP-50 FTS pool (pre-existing #2084; live in prod today; also explains the blackhole search hang Wren observed — the routed path was measured bounded at 151-152ms against a silent endpoint, sqlx `inner.rs:252-255`). 2. **#3651** — reader `statement_timeout`: statements after acquire are unbounded; mid-transaction blackhole reproduced hanging ≥15s (Dawn). Small fix via `.after_connect` on the reader pool. 3. **#3622** — CI enforcement of the PG fixtures (with #3619 ordering). 4. **Recovery-conflict cancellation live test** — standby cancels a routed read mid-snapshot → complete writer page, never partial (#3651's fix also bounds the cancellation-never-arrives shape). 5. **DDL replication lag test** — migration on writer + paused replay + budget on → fallback, not client-visible SQL errors. 6. **Reader-tx soak** — 30-60min mixed load; no idle-in-tx accumulation on the standby. Then Max's three deployment tests (reader-down latency, reader-down cold boot, paused-replay read-your-own-write search) re-run on the fixed binary before the first nonzero budget on bb-block. --------- Signed-off-by: Tyler Longwell <[email protected]> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <[email protected]>
1 parent 788b3c0 commit 63496cc

15 files changed

Lines changed: 3375 additions & 252 deletions

File tree

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/buzz-db/Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ tracing = { workspace = true }
2121
thiserror = { workspace = true }
2222
nostr = { workspace = true }
2323
rand = { workspace = true }
24+
metrics = { workspace = true }
2425

2526
[dev-dependencies]
2627
tokio = { workspace = true }
28+
metrics-util = { workspace = true }

‎crates/buzz-db/src/event.rs‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,17 @@ pub async fn insert_event(
316316
/// Uses `QueryBuilder` for dynamic filter composition — avoids string concatenation
317317
/// while keeping all user values in bind parameters.
318318
pub async fn query_events(pool: &PgPool, q: &EventQuery) -> Result<Vec<StoredEvent>> {
319+
let mut conn = pool.acquire().await?;
320+
query_events_on(&mut conn, q).await
321+
}
322+
323+
/// [`query_events`] on a specific session — the replica-routing path runs
324+
/// follow-up (aux) queries on the exact reader connection whose heartbeat
325+
/// observation proved coverage for the page they annotate.
326+
pub(crate) async fn query_events_on(
327+
conn: &mut sqlx::PgConnection,
328+
q: &EventQuery,
329+
) -> Result<Vec<StoredEvent>> {
319330
// Composite cursor requires both halves.
320331
if q.before_id.is_some() && q.until.is_none() {
321332
return Err(DbError::InvalidData(
@@ -538,7 +549,7 @@ pub async fn query_events(pool: &PgPool, q: &EventQuery) -> Result<Vec<StoredEve
538549
qb.push_bind(limit_val);
539550
qb.push(" OFFSET ").push_bind(offset_val);
540551

541-
let rows = qb.build().fetch_all(pool).await?;
552+
let rows = qb.build().fetch_all(&mut *conn).await?;
542553

543554
let mut out = Vec::with_capacity(rows.len());
544555
for row in rows {
@@ -596,6 +607,14 @@ pub(crate) fn row_to_stored_event(row: sqlx::postgres::PgRow) -> Result<Option<S
596607
///
597608
/// Uses the same filter logic as `query_events` but returns only the count.
598609
pub async fn count_events(pool: &PgPool, q: &EventQuery) -> Result<i64> {
610+
let mut conn = pool.acquire().await?;
611+
count_events_on(&mut conn, q).await
612+
}
613+
614+
/// [`count_events`] on a specific session — the replica-routing path runs
615+
/// the count on the exact reader connection whose heartbeat observation
616+
/// proved its predicate.
617+
pub(crate) async fn count_events_on(conn: &mut sqlx::PgConnection, q: &EventQuery) -> Result<i64> {
599618
// Empty list means "match nothing" — return 0 immediately.
600619
if q.kinds.as_deref().is_some_and(|k| k.is_empty()) {
601620
return Ok(0);
@@ -730,7 +749,7 @@ pub async fn count_events(pool: &PgPool, q: &EventQuery) -> Result<i64> {
730749
}
731750
}
732751

733-
let row = qb.build().fetch_one(pool).await?;
752+
let row = qb.build().fetch_one(&mut *conn).await?;
734753
let cnt: i64 = row.try_get("cnt")?;
735754

736755
Ok(cnt)
@@ -990,6 +1009,21 @@ pub async fn get_events_by_ids(
9901009
pool: &PgPool,
9911010
community_id: CommunityId,
9921011
ids: &[&[u8]],
1012+
) -> Result<Vec<StoredEvent>> {
1013+
if ids.is_empty() {
1014+
return Ok(vec![]);
1015+
}
1016+
let mut conn = pool.acquire().await?;
1017+
get_events_by_ids_on(&mut conn, community_id, ids).await
1018+
}
1019+
1020+
/// [`get_events_by_ids`] on a specific session — the replica-routing path
1021+
/// runs the query on the exact reader connection whose heartbeat
1022+
/// observation proved its predicate.
1023+
pub(crate) async fn get_events_by_ids_on(
1024+
conn: &mut sqlx::PgConnection,
1025+
community_id: CommunityId,
1026+
ids: &[&[u8]],
9931027
) -> Result<Vec<StoredEvent>> {
9941028
if ids.is_empty() {
9951029
return Ok(vec![]);
@@ -1008,7 +1042,7 @@ pub async fn get_events_by_ids(
10081042
}
10091043
qb.push(")");
10101044

1011-
let rows = qb.build().fetch_all(pool).await?;
1045+
let rows = qb.build().fetch_all(&mut *conn).await?;
10121046

10131047
let mut out = Vec::with_capacity(rows.len());
10141048
for row in rows {

‎crates/buzz-db/src/feed.rs‎

Lines changed: 59 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,29 @@ pub async fn query_mentions(
132132
accessible_channel_ids: &[Uuid],
133133
since: Option<DateTime<Utc>>,
134134
limit: i64,
135+
) -> Result<Vec<StoredEvent>> {
136+
let mut conn = pool.acquire().await?;
137+
query_mentions_on(
138+
&mut conn,
139+
community,
140+
pubkey_bytes,
141+
accessible_channel_ids,
142+
since,
143+
limit,
144+
)
145+
.await
146+
}
147+
148+
/// [`query_mentions`] on a specific session — the replica-routing path runs
149+
/// the query on the exact reader connection whose heartbeat observation
150+
/// proved its predicate.
151+
pub(crate) async fn query_mentions_on(
152+
conn: &mut sqlx::PgConnection,
153+
community: CommunityId,
154+
pubkey_bytes: &[u8],
155+
accessible_channel_ids: &[Uuid],
156+
since: Option<DateTime<Utc>>,
157+
limit: i64,
135158
) -> Result<Vec<StoredEvent>> {
136159
let mut qb = build_mentions_query(
137160
community,
@@ -140,7 +163,7 @@ pub async fn query_mentions(
140163
since,
141164
limit,
142165
);
143-
let rows = qb.build().fetch_all(pool).await?;
166+
let rows = qb.build().fetch_all(&mut *conn).await?;
144167
collect_stored_events(rows)
145168
}
146169

@@ -193,6 +216,27 @@ pub async fn query_needs_action(
193216
accessible_channel_ids: &[Uuid],
194217
since: Option<DateTime<Utc>>,
195218
limit: i64,
219+
) -> Result<Vec<StoredEvent>> {
220+
let mut conn = pool.acquire().await?;
221+
query_needs_action_on(
222+
&mut conn,
223+
community,
224+
pubkey_bytes,
225+
accessible_channel_ids,
226+
since,
227+
limit,
228+
)
229+
.await
230+
}
231+
232+
/// [`query_needs_action`] on a specific session — see [`query_mentions_on`].
233+
pub(crate) async fn query_needs_action_on(
234+
conn: &mut sqlx::PgConnection,
235+
community: CommunityId,
236+
pubkey_bytes: &[u8],
237+
accessible_channel_ids: &[Uuid],
238+
since: Option<DateTime<Utc>>,
239+
limit: i64,
196240
) -> Result<Vec<StoredEvent>> {
197241
let mut qb = build_needs_action_query(
198242
community,
@@ -201,7 +245,7 @@ pub async fn query_needs_action(
201245
since,
202246
limit,
203247
);
204-
let rows = qb.build().fetch_all(pool).await?;
248+
let rows = qb.build().fetch_all(&mut *conn).await?;
205249
collect_stored_events(rows)
206250
}
207251

@@ -241,9 +285,21 @@ pub async fn query_activity(
241285
accessible_channel_ids: &[Uuid],
242286
since: Option<DateTime<Utc>>,
243287
limit: i64,
288+
) -> Result<Vec<StoredEvent>> {
289+
let mut conn = pool.acquire().await?;
290+
query_activity_on(&mut conn, community, accessible_channel_ids, since, limit).await
291+
}
292+
293+
/// [`query_activity`] on a specific session — see [`query_mentions_on`].
294+
pub(crate) async fn query_activity_on(
295+
conn: &mut sqlx::PgConnection,
296+
community: CommunityId,
297+
accessible_channel_ids: &[Uuid],
298+
since: Option<DateTime<Utc>>,
299+
limit: i64,
244300
) -> Result<Vec<StoredEvent>> {
245301
let mut qb = build_activity_query(community, accessible_channel_ids, since, limit);
246-
let rows = qb.build().fetch_all(pool).await?;
302+
let rows = qb.build().fetch_all(&mut *conn).await?;
247303
collect_stored_events(rows)
248304
}
249305

0 commit comments

Comments
 (0)