Skip to content

Commit 95a8255

Browse files
TheSentinel454Elrondclaude
committed
test(relay): pin the timeout reason on a failed history REQ
Address review: the regression only drove an ordinary DB error, so a mapper that labelled every failure `database error` still passed. Lock the replica's `events` past its statement timeout first and assert the REQ ends in exactly `CLOSED error: query timed out` with the subscription retired, then run the existing 42P01 arm through the same helper. Clients treat the two reasons differently (timeout terminal, database error retryable), so the label is contract. Also qualify the ARCHITECTURE.md EOSE sentence: history is capped at 500 per filter, so EOSE marks a successful bounded query, and the claim is scoped to ordinary (non-search) REQs. Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Luke Tornquist <[email protected]>
1 parent e5621f9 commit 95a8255

2 files changed

Lines changed: 46 additions & 34 deletions

File tree

‎ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -378,7 +378,7 @@ This prevents a race where a non-member receives live fan-out events from a priv
378378

379379
### Historical Query (EOSE)
380380

381-
After registering, the REQ handler queries Postgres for stored events matching the filters (up to 500 per filter, hard cap). These are sent as `["EVENT", sub_id, event]` frames before `["EOSE", sub_id]`. New events arriving after EOSE are delivered via the fan-out path. EOSE means every stored match was sent, so a failed history read never ends in EOSE: it retires the subscription with `["CLOSED", sub_id, "error: query timed out"]` for a statement timeout or `"error: database error"` for anything else.
381+
After registering, the REQ handler queries Postgres for stored events matching the filters (up to 500 per filter, hard cap). These are sent as `["EVENT", sub_id, event]` frames before `["EOSE", sub_id]`. New events arriving after EOSE are delivered via the fan-out path. For an ordinary (non-search) REQ, EOSE marks a successful bounded history query, so a failed history read never ends in EOSE: it retires the subscription with `["CLOSED", sub_id, "error: query timed out"]` for a statement timeout or `"error: database error"` for anything else.
382382

383383
**Client consumption invariant.** A client rebuilding channel state must
384384
open its live subscription before (or overlapping) the finite history

‎crates/buzz-relay/src/api/bridge.rs‎

Lines changed: 45 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -9448,12 +9448,14 @@ pub(crate) mod postgres_tests {
94489448
fx.drop().await;
94499449
}
94509450

9451-
/// A history read that fails with an ordinary DB error ends the REQ with a
9452-
/// retryable CLOSED, never EOSE: EOSE would tell a resuming client it holds
9453-
/// every stored event, so its next REQ would skip the history that failed.
9454-
/// The CLOSED also retires the subscription, and once the DB recovers a
9455-
/// reconnect with the same REQ replays the missed event.
9456-
/// Mutation: send EOSE on the non-timeout error again → RED.
9451+
/// A failed history read ends the REQ with CLOSED, never EOSE: EOSE would
9452+
/// tell a resuming client it holds every stored event, so its next REQ would
9453+
/// skip the history that failed. A statement timeout keeps its terminal
9454+
/// `query timed out` reason; any other DB error gets the retryable
9455+
/// `database error`. Either CLOSED retires the subscription, and once the DB
9456+
/// recovers a reconnect with the same REQ replays the missed event.
9457+
/// Mutations: send EOSE on the non-timeout error again, or map every error
9458+
/// to `database error` → RED.
94579459
#[tokio::test]
94589460
#[ignore = "requires Postgres"]
94599461
async fn failed_history_read_closes_and_reconnect_replays_the_history() {
@@ -9474,6 +9476,42 @@ pub(crate) mod postgres_tests {
94749476
let channel: uuid::Uuid = fx.channel.parse().expect("channel uuid");
94759477
let topic = buzz_pubsub::EventTopic::Channel(channel);
94769478
let tenant = TenantContext::resolved(fx.community, fx.host.clone());
9479+
let fails_and_retires = |sub: &'static str, reason: &'static str| {
9480+
let state = fx.state.clone();
9481+
let (conn, mut rx) = fx.ws_conn();
9482+
let tenant = tenant.clone();
9483+
async move {
9484+
handle_req(
9485+
sub.into(),
9486+
vec![req()],
9487+
vec![None],
9488+
conn.clone(),
9489+
state.clone(),
9490+
)
9491+
.await;
9492+
assert_eq!(
9493+
frames(&mut rx),
9494+
vec![format!(r#"["CLOSED","{sub}","{reason}"]"#)],
9495+
"a failed history read must CLOSE with {reason:?}, not EOSE"
9496+
);
9497+
assert!(conn.subscriptions.lock().await.is_empty(), "conn map");
9498+
assert!(
9499+
state.sub_registry.get_filters(conn.conn_id, sub).is_none(),
9500+
"CLOSED must retire the fan-out registration"
9501+
);
9502+
assert_eq!(state.pubsub.topic_refcount(&tenant, topic).await, 0);
9503+
}
9504+
};
9505+
9506+
// A replica lock outlasting its 300ms `statement_timeout`: the read is
9507+
// cancelled with 57014, which keeps the terminal timeout reason.
9508+
let mut locker = fx.replica.begin().await.expect("begin locker");
9509+
sqlx::query("LOCK TABLE events IN ACCESS EXCLUSIVE MODE")
9510+
.execute(&mut *locker)
9511+
.await
9512+
.expect("lock replica events");
9513+
fails_and_retires("t", crate::handlers::req::QUERY_TIMED_OUT_CLOSED).await;
9514+
locker.rollback().await.expect("unlock");
94779515

94789516
// `events` gone on both pools: the history read fails with 42P01 on the
94799517
// replica and again on the writer re-run. Not a statement timeout.
@@ -9491,33 +9529,7 @@ pub(crate) mod postgres_tests {
94919529
}
94929530
};
94939531
rename("events", "events_gone").await;
9494-
let (conn, mut rx) = fx.ws_conn();
9495-
handle_req(
9496-
"h".into(),
9497-
vec![req()],
9498-
vec![None],
9499-
conn.clone(),
9500-
fx.state.clone(),
9501-
)
9502-
.await;
9503-
assert_eq!(
9504-
frames(&mut rx),
9505-
vec![format!(
9506-
r#"["CLOSED","h","{}"]"#,
9507-
crate::handlers::req::DATABASE_ERROR_CLOSED
9508-
)],
9509-
"a failed history read must CLOSE, not EOSE"
9510-
);
9511-
assert!(conn.subscriptions.lock().await.is_empty(), "conn map");
9512-
assert!(
9513-
fx.state
9514-
.sub_registry
9515-
.get_filters(conn.conn_id, "h")
9516-
.is_none(),
9517-
"CLOSED must retire the fan-out registration"
9518-
);
9519-
assert_eq!(fx.state.pubsub.topic_refcount(&tenant, topic).await, 0);
9520-
drop(conn);
9532+
fails_and_retires("h", crate::handlers::req::DATABASE_ERROR_CLOSED).await;
95219533

95229534
// The DB recovers and the client reconnects with the same REQ.
95239535
rename("events_gone", "events").await;

0 commit comments

Comments
 (0)