Skip to content

Commit 76cd429

Browse files
TheSentinel454Elrondclaude
authored
fix(relay): close a REQ whose history read failed instead of sending EOSE (#8692)
## Summary **Merge order:** this lands and deploys **before** [#8688](#8688). That PR's buzz-acp resume cursor counts on a failed history read never ending in EOSE. Until #8688 ships, buzz-acp re-sends a REQ as soon as it gets an error CLOSED. The relay's REQ rate limit bounds that loop, and buzz-acp already parks on a rate-limited CLOSED. #8688 replaces this with a per-subscription backoff. When a historical query fails for a reason other than a statement timeout, the relay logs the error and still sends `EOSE`. EOSE means "you have every stored event" (`ARCHITECTURE.md` §5, NIP-01). A client that resumes from its last-seen point therefore treats the failed read as complete, advances past it, and never replays that history. This is RPR #1: a failure must not become an authoritative empty result. The history task now returns the CLOSED reason to send: - **Statement timeout:** `["CLOSED", sub_id, "error: query timed out"]`, unchanged. - **Any other DB error:** `["CLOSED", sub_id, "error: database error"]`. - **Cleanup:** CLOSED goes through the existing `close_if_owner` path, so the subscription, its sub-registry filters and its topic refcount are released, the same as on a timeout. - **Docs:** `ARCHITECTURE.md` §5 now states this contract. **Clients don't hot-loop on it.** Checked at `main` 7c165c3: - **Desktop:** `relayClosedPolicy.ts` / `relayClosedRecovery.ts` classify `error: database error` as retryable. Live subscriptions back off from 1s to 30s and keep their `since`. History-mode requests reject. - **Mobile:** `relay_closed_policy.dart` / `relay_session.dart` do the same, with their own capped backoff. - **buzz-acp:** today it re-REQs at once, bounded by the relay's REQ rate limit. [#8688](#8688) adds a per-subscription backoff that also restores the loss record. ### What got simpler - **One constant for the CLOSED reason.** `"error: database error"` was a string literal repeated across `req.rs` (5×) and `count.rs` (4×). It is now one constant, `DATABASE_ERROR_CLOSED`. - **One mapping function.** `history_read_closed_reason` maps a DB error to its CLOSED reason in one place. ### Identified debt (not changed here) - **NIP-50 search path.** After a non-timeout full-text-search or hydrate error, the search path still sends `EOSE` with partial results. It needs the same treatment, but it has a separate result-assembly path, so it's left out of this PR. ### Related issue Adds the regression wpfleger96 asked for in [this review on #8653](#8653 (review)): a history error followed by a reconnect still replays the history. ### Testing New Postgres test `api::bridge::postgres_tests::failed_history_read_closes_and_reconnect_replays_the_history`: 1. Lock the replica's `events` table past its 300ms `statement_timeout` and send a history REQ. It must get exactly `["CLOSED","t","error: query timed out"]`. 2. Break the `events` table on both the writer and replica pools and send a REQ. It must get exactly `["CLOSED","h","error: database error"]`. 3. After each CLOSED, the subscription map, the sub-registry filters and the topic refcount must all be cleared. 4. Restore the table and send the same REQ on a new connection. It must get the stored event, then `EOSE`. Run on Blox at 95a8255: - `cargo fmt --check` - `cargo clippy -p buzz-relay --all-targets -D warnings` - Postgres lane via `scripts/postgres-test-run.sh -p buzz-relay` (499/499) - The new test and `routed_read_cancel_maps_to_timeout_contract` 5× - **Mutation checks:** each of these makes the new test fail at its frame assertion. - Restoring `EOSE` for a non-timeout error. - Mapping every failure to `error: database error`. - **`just test` integration half:** run by hand, because `_ensure-services` can't start its Redis container on that workstation while a native redis-server holds the port. - Workspace `cargo test --test '*'`: every binary passes except two `buzz-acp` `git_bootstrap` tests. - Those two fail the same way on that workstation without this change, so they come from the environment. - `buzz-db` and `buzz-auth` integration tests pass. End-to-end against a locally built relay at e5621f9, with buzz-acp from #8688: - **While `events` was missing:** every channel REQ got `CLOSED error: database error`, and no channel EOSE was sent. - **After the table was restored:** the next retry replayed the missed mention, followed by EOSE. The later commit only changes tests and docs. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <[email protected]> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent be1ac39 commit 76cd429

4 files changed

Lines changed: 142 additions & 21 deletions

File tree

‎ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -378,7 +378,7 @@ This prevents a race where a non-member receives live fan-out events from a priv
378378

379379
### Historical Query (EOSE)
380380

381-
After registering, the REQ handler queries Postgres for stored events matching the filters (up to 500 per filter, hard cap). These are sent as `["EVENT", sub_id, event]` frames before `["EOSE", sub_id]`. New events arriving after EOSE are delivered via the fan-out path.
381+
After registering, the REQ handler queries Postgres for stored events matching the filters (up to 500 per filter, hard cap). These are sent as `["EVENT", sub_id, event]` frames before `["EOSE", sub_id]`. New events arriving after EOSE are delivered via the fan-out path. For an ordinary (non-search) REQ, EOSE marks a successful bounded history query, so a failed history read never ends in EOSE: it retires the subscription with `["CLOSED", sub_id, "error: query timed out"]` for a statement timeout or `"error: database error"` for anything else.
382382

383383
**Client consumption invariant.** A client rebuilding channel state must
384384
open its live subscription before (or overlapping) the finite history

‎crates/buzz-relay/src/api/bridge.rs‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9448,6 +9448,112 @@ pub(crate) mod postgres_tests {
94489448
fx.drop().await;
94499449
}
94509450

9451+
/// A failed history read ends the REQ with CLOSED, never EOSE: EOSE would
9452+
/// tell a resuming client it holds every stored event, so its next REQ would
9453+
/// skip the history that failed. A statement timeout keeps its terminal
9454+
/// `query timed out` reason; any other DB error gets the retryable
9455+
/// `database error`. Either CLOSED retires the subscription, and once the DB
9456+
/// recovers a reconnect with the same REQ replays the missed event.
9457+
/// Mutations: send EOSE on the non-timeout error again, or map every error
9458+
/// to `database error` → RED.
9459+
#[tokio::test]
9460+
#[ignore = "requires Postgres"]
9461+
async fn failed_history_read_closes_and_reconnect_replays_the_history() {
9462+
use crate::handlers::req::handle_req;
9463+
let fx = cancel_fixture().await;
9464+
let frames = |rx: &mut tokio::sync::mpsc::Receiver<axum::extract::ws::Message>| {
9465+
std::iter::from_fn(|| rx.try_recv().ok())
9466+
.map(|msg| match msg {
9467+
axum::extract::ws::Message::Text(t) => t.to_string(),
9468+
other => panic!("expected text frame, got {other:?}"),
9469+
})
9470+
.collect::<Vec<_>>()
9471+
};
9472+
let req = || -> nostr::Filter {
9473+
serde_json::from_value(serde_json::json!({"kinds": [9], "#h": [&fx.channel]}))
9474+
.expect("history filter")
9475+
};
9476+
let channel: uuid::Uuid = fx.channel.parse().expect("channel uuid");
9477+
let topic = buzz_pubsub::EventTopic::Channel(channel);
9478+
let tenant = TenantContext::resolved(fx.community, fx.host.clone());
9479+
let fails_and_retires = |sub: &'static str, reason: &'static str| {
9480+
let state = fx.state.clone();
9481+
let (conn, mut rx) = fx.ws_conn();
9482+
let tenant = tenant.clone();
9483+
async move {
9484+
handle_req(
9485+
sub.into(),
9486+
vec![req()],
9487+
vec![None],
9488+
conn.clone(),
9489+
state.clone(),
9490+
)
9491+
.await;
9492+
assert_eq!(
9493+
frames(&mut rx),
9494+
vec![format!(r#"["CLOSED","{sub}","{reason}"]"#)],
9495+
"a failed history read must CLOSE with {reason:?}, not EOSE"
9496+
);
9497+
assert!(conn.subscriptions.lock().await.is_empty(), "conn map");
9498+
assert!(
9499+
state.sub_registry.get_filters(conn.conn_id, sub).is_none(),
9500+
"CLOSED must retire the fan-out registration"
9501+
);
9502+
assert_eq!(state.pubsub.topic_refcount(&tenant, topic).await, 0);
9503+
}
9504+
};
9505+
9506+
// A replica lock outlasting its 300ms `statement_timeout`: the read is
9507+
// cancelled with 57014, which keeps the terminal timeout reason.
9508+
let mut locker = fx.replica.begin().await.expect("begin locker");
9509+
sqlx::query("LOCK TABLE events IN ACCESS EXCLUSIVE MODE")
9510+
.execute(&mut *locker)
9511+
.await
9512+
.expect("lock replica events");
9513+
fails_and_retires("t", crate::handlers::req::QUERY_TIMED_OUT_CLOSED).await;
9514+
locker.rollback().await.expect("unlock");
9515+
9516+
// `events` gone on both pools: the history read fails with 42P01 on the
9517+
// replica and again on the writer re-run. Not a statement timeout.
9518+
let rename = |from: &'static str, to: &'static str| {
9519+
let pools = [fx.writer.clone(), fx.replica.clone()];
9520+
async move {
9521+
for pool in pools {
9522+
sqlx::query(sqlx::AssertSqlSafe(format!(
9523+
"ALTER TABLE {from} RENAME TO {to}"
9524+
)))
9525+
.execute(&pool)
9526+
.await
9527+
.expect("rename events");
9528+
}
9529+
}
9530+
};
9531+
rename("events", "events_gone").await;
9532+
fails_and_retires("h", crate::handlers::req::DATABASE_ERROR_CLOSED).await;
9533+
9534+
// The DB recovers and the client reconnects with the same REQ.
9535+
rename("events_gone", "events").await;
9536+
let (conn, mut rx) = fx.ws_conn();
9537+
handle_req(
9538+
"h".into(),
9539+
vec![req()],
9540+
vec![None],
9541+
conn.clone(),
9542+
fx.state.clone(),
9543+
)
9544+
.await;
9545+
let replay = frames(&mut rx);
9546+
assert_eq!(replay.len(), 2, "event then EOSE: {replay:?}");
9547+
assert!(
9548+
replay[0].starts_with(r#"["EVENT","h","#) && replay[0].contains(&fx.root),
9549+
"the reconnect must replay the missed event: {replay:?}"
9550+
);
9551+
assert_eq!(replay[1], r#"["EOSE","h"]"#);
9552+
9553+
drop(conn);
9554+
fx.drop().await;
9555+
}
9556+
94519557
/// HTTP agents reach `materialize_nip_oa_owner` on every request. Once the
94529558
/// owner link is confirmed in `observer_owner_cache` the call writes
94539559
/// nothing: the link is set once and never cleared.

‎crates/buzz-relay/src/handlers/count.rs‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ use crate::connection::{AuthState, ConnectionState};
1010
use crate::handlers::req::{
1111
event_visible_to_reader, filter_can_match_result_gated_kinds,
1212
filter_can_match_shared_gated_kinds, result_gated_count_safe_for_pushdown,
13+
DATABASE_ERROR_CLOSED,
1314
};
1415
use crate::protocol::RelayMessage;
1516
use crate::state::AppState;
@@ -88,7 +89,7 @@ pub async fn handle_count(
8889
Ok(ids) => ids,
8990
Err(e) => {
9091
warn!(sub_id = %sub_id, "Failed to get accessible channels: {e}");
91-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
92+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
9293
return;
9394
}
9495
};
@@ -116,7 +117,7 @@ pub async fn handle_count(
116117
Ok(ids) => ids,
117118
Err(e) => {
118119
warn!(sub_id = %sub_id, "Failed to get readable named-only channels: {e}");
119-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
120+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
120121
return;
121122
}
122123
};
@@ -178,7 +179,7 @@ pub async fn handle_count(
178179
.await
179180
{
180181
warn!(sub_id = %sub_id, "Channel membership confirmation failed: {e}");
181-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
182+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
182183
return;
183184
}
184185
if let Err(e) = super::system_channels::keep_repaired_named_only_channels_named(
@@ -191,7 +192,7 @@ pub async fn handle_count(
191192
.await
192193
{
193194
warn!(sub_id = %sub_id, "Failed to classify repaired channels: {e}");
194-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
195+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
195196
return;
196197
}
197198
let filter_scope = super::system_channels::filter_channel_scope(

‎crates/buzz-relay/src/handlers/req.rs‎

Lines changed: 30 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ pub async fn handle_req(
122122
Ok(ids) => ids,
123123
Err(e) => {
124124
warn!(conn_id = %conn_id, "Failed to get accessible channels: {e}");
125-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
125+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
126126
return;
127127
}
128128
}
@@ -146,7 +146,7 @@ pub async fn handle_req(
146146
Ok(ids) => ids,
147147
Err(e) => {
148148
warn!(conn_id = %conn_id, "Failed to get readable named-only channels: {e}");
149-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
149+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
150150
return;
151151
}
152152
};
@@ -185,7 +185,7 @@ pub async fn handle_req(
185185
Ok(confirmed) => confirmed,
186186
Err(e) => {
187187
warn!(conn_id = %conn_id, "Channel membership confirmation failed: {e}");
188-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
188+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
189189
return;
190190
}
191191
};
@@ -204,7 +204,7 @@ pub async fn handle_req(
204204
.await
205205
{
206206
warn!(conn_id = %conn_id, "Failed to classify repaired channels: {e}");
207-
conn.send(RelayMessage::closed(&sub_id, "error: database error"));
207+
conn.send(RelayMessage::closed(&sub_id, DATABASE_ERROR_CLOSED));
208208
return;
209209
}
210210
}
@@ -401,8 +401,9 @@ pub async fn handle_req(
401401
// Registration above is one uncancellable unit; the history read below is
402402
// read-only delivery and races gate cancellation, so expiry drops it and
403403
// releases the permit without sending another EVENT or EOSE. It returns
404-
// whether a statement timeout ended it: retirement then runs after the
405-
// race, because a dropped retirement would leak its unreleased topics.
404+
// the CLOSED reason when a failed read ended it: retirement then runs
405+
// after the race, because a dropped retirement would leak its unreleased
406+
// topics.
406407
let history = async {
407408
#[cfg(test)]
408409
crate::nip_fi_test_hooks::before_req_history(conn.tenant.community()).await;
@@ -489,11 +490,9 @@ pub async fn handle_req(
489490
Ok(evs) => evs,
490491
Err(e) => {
491492
warn!(conn_id = %conn_id, sub_id = %sub_id, "Historical query failed: {e}");
492-
if e.is_statement_cancelled() {
493-
return true;
494-
}
495-
conn.send(RelayMessage::eose(&sub_id));
496-
return false;
493+
// CLOSED, never EOSE: EOSE tells the client it has every
494+
// stored event, so a resuming client would skip this history.
495+
return Some(history_read_closed_reason(&e));
497496
}
498497
};
499498

@@ -570,7 +569,7 @@ pub async fn handle_req(
570569

571570
let msg = RelayMessage::event(&sub_id, &stored.event);
572571
if !conn.send(msg) {
573-
return false;
572+
return None;
574573
}
575574
total_sent += 1;
576575
if total_sent.is_multiple_of(100) {
@@ -587,13 +586,15 @@ pub async fn handle_req(
587586
count = total_sent,
588587
"EOSE sent after historical delivery"
589588
);
590-
false
589+
None
591590
};
592591
let outcome = unless_cancelled(&conn, history).await;
593592
drop(_req_permit);
594593
match outcome {
595-
Some(false) => {}
596-
Some(true) => close_timed_out_subscription(&sub_id, owner, &conn, &state).await,
594+
Some(None) => {}
595+
Some(Some(reason)) => {
596+
super::close::close_if_owner(&sub_id, owner, Some(reason), &conn, &state).await;
597+
}
597598
None => {
598599
super::close::close_if_owner(&sub_id, owner, None, &conn, &state).await;
599600
}
@@ -1439,7 +1440,7 @@ async fn handle_huddle_liveness_req(
14391440
Ok(links) => links,
14401441
Err(error) => {
14411442
warn!("Huddle liveness linkage batch failed: {error}");
1442-
conn.send(RelayMessage::closed(sub_id, "error: database error"));
1443+
conn.send(RelayMessage::closed(sub_id, DATABASE_ERROR_CLOSED));
14431444
return;
14441445
}
14451446
};
@@ -1600,6 +1601,19 @@ async fn close_timed_out_subscription(
16001601
/// Clients match it to skip retrying the same expensive query.
16011602
pub(crate) const QUERY_TIMED_OUT_CLOSED: &str = "error: query timed out";
16021603

1604+
/// CLOSED reason for any other failed DB read. Generic on purpose: the raw
1605+
/// error stays in the relay log, and clients treat it as retryable.
1606+
pub(crate) const DATABASE_ERROR_CLOSED: &str = "error: database error";
1607+
1608+
/// CLOSED reason for a failed REQ history read.
1609+
fn history_read_closed_reason(e: &buzz_db::DbError) -> &'static str {
1610+
if e.is_statement_cancelled() {
1611+
QUERY_TIMED_OUT_CLOSED
1612+
} else {
1613+
DATABASE_ERROR_CLOSED
1614+
}
1615+
}
1616+
16031617
/// CLOSED reason for a failed one-shot DB read (COUNT): a statement cancel gets
16041618
/// the stable timeout reason; anything else keeps the raw error.
16051619
pub(crate) fn db_read_closed_reason(e: &buzz_db::DbError) -> String {

0 commit comments

Comments
 (0)