Skip to content

Commit 19b41e9

Browse files
wesbillmanPinkyBrain
authored
fix(desktop): stop rate-limited reconnect backfill from tearing down the authenticated socket (block#4990)
## Problem Users on v0.5.5 report "Can't reach the relay" toggling with brief "connected" flashes (field reports; also the macOS confirmation in block#4908). block#4737 closed the stuck-reconnect gaps; this is the opposite failure: the client redials fine, but then kills its own healthy socket. Mechanism (all on `main`): 1. AUTH succeeds → session emits `connected` (`relayClientSession.ts:583`), then awaits `replayLiveSubscriptions()`. 2. Paged channel backfill issues history REQs (`relayReconnectReplay.ts`, page limit 500). 3. A `CLOSED rate-limited:` on a **history** REQ arms the rate-limit gate but still rejects the history promise (`relayClosedRecovery.ts:38-51`). 4. The rejection escapes `replayLiveSubscriptions()` → `resetConnection()` tears down the authenticated socket. 5. Reconnect → AUTH OK → replay rate-limited again → loop. Each iteration re-spends the rate-limit budget, so the loop is self-sustaining. ## Fix Contain backfill failures inside the replay. Each subscription's paged backfill now retries behind the rate-limit gate up to `PAGE_REPLAY_MAX_ATTEMPTS` (3), then degrades to live-only **for this connection**. Socket health no longer depends on backfill success. Nothing is lost: the replay cursor (`lastSeenCreatedAt`) only advances on delivered events, so the next reconnect replays the same missed window. ## Red/green proof - Commit 1 (Pinky): e2e injecting `CLOSED rate-limited:` into the mid-replay history REQ — **red on main** (expected 1 reconnect dial, observed 2; connected-flash then teardown). - Commit 2 (this fix): same test **green unchanged** — one dial, state stays `connected` through the rate-limit hint plus the next backoff window. Why existing coverage missed it: the prior rate-limit e2e pre-armed the gate *before* replay (replay politely waits), and the CLOSED-injection test targeted a *live* subscription (which has its own retry path). Nobody injected back-pressure from the history REQ itself. ## Verification - `pnpm test`: 4374/4374 pass. - `playwright test tests/e2e/relay-reconnect.spec.ts`: 14/14 pass, including the new spec. - `tsc --noEmit` clean; Biome clean on touched files (pre-existing warnings on main in `personaCatalogRelay.test.mjs` / `terminal.css` untouched). ## Not addressed here (follow-ups from the same field reports) - AUTH terminal latch is too aggressive for relay-internal `error:` rejections (3 strikes during a relay bad window → stuck until click/relaunch; block#4908). - Server-side: `relay.drainJitterMs` (block#4542) defaults to 0 — enabling it on the hosted relay removes the deploy thundering herd that triggers these rate-limit storms. --------- Signed-off-by: Pinky <44b8e82baa6e0e254e0208d68f335c283c94e7b78dd1fa10d5a49d3f13dd0435@buzz.block.builderlab.xyz> Signed-off-by: Brain <21994759fc7a6fa6b965551d35cfd7897d262f2495467f2d78694ddcfa6a5c7e@buzz.block.builderlab.xyz> Co-authored-by: Pinky <44b8e82baa6e0e254e0208d68f335c283c94e7b78dd1fa10d5a49d3f13dd0435@buzz.block.builderlab.xyz> Co-authored-by: Brain <21994759fc7a6fa6b965551d35cfd7897d262f2495467f2d78694ddcfa6a5c7e@buzz.block.builderlab.xyz>
1 parent 96ae141 commit 19b41e9

5 files changed

Lines changed: 431 additions & 13 deletions

File tree

‎desktop/src/shared/api/relayClientShared.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,17 @@ type LiveSubscription = {
6060
onEvent: (event: RelayEvent) => void;
6161
resolveReady?: () => void;
6262
lastSeenCreatedAt?: number;
63+
/**
64+
* Lower bound of a reconnect backfill window that has not yet completed.
65+
*
66+
* Events on the restored live REQ advance `lastSeenCreatedAt` regardless of
67+
* backfill success, so after an exhausted backfill the cursor alone would
68+
* make the next reconnect skip the unresolved older window — silent message
69+
* loss. This floor is pinned when paging starts and cleared only when a
70+
* backfill pass completes; the next replay starts from
71+
* `min(pendingReplaySince, cursor window)`.
72+
*/
73+
pendingReplaySince?: number;
6374
closedRetryAttempt?: number;
6475
closedRetryTimeout?: number;
6576
};

‎desktop/src/shared/api/relayReconnectReplay.test.mjs‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,13 @@ import test from "node:test";
33

44
import {
55
buildReconnectReplayFilter,
6+
PAGE_REPLAY_MAX_ATTEMPTS,
67
replayLiveSubscriptions,
78
REPLAY_BATCH_SIZE,
89
shouldPageReconnectReplay,
910
} from "./relayReconnectReplay.ts";
1011
import { buildChannelFilter } from "./relayChannelFilters.ts";
12+
import { prepareSubscriptionEvent } from "./relayClosedRecovery.ts";
1113

1214
// ── Fake-timer + Date.now setup for gate tests ────────────────────────────────
1315

@@ -600,6 +602,254 @@ test("batch-1 arms gate mid-replay: batch-2 is withheld until gate expires", asy
600602
);
601603
});
602604

605+
// ── Backfill failure containment ─────────────────────────────────────────────
606+
607+
test("history backfill rejection never escapes replayLiveSubscriptions", async () => {
608+
resetGate(0);
609+
const filter = buildChannelFilter("channel-1", 50);
610+
const subscriptions = new Map([
611+
[
612+
"live-1",
613+
{
614+
mode: "live",
615+
filter,
616+
onEvent: () => {},
617+
lastSeenCreatedAt: 1000,
618+
},
619+
],
620+
]);
621+
622+
let historyCalls = 0;
623+
// Must resolve — a rejection here is the socket-killing flap regression.
624+
await replayLiveSubscriptions({
625+
subscriptions,
626+
now: 2000,
627+
sendRaw: async () => {},
628+
requestHistory: async () => {
629+
historyCalls++;
630+
throw new Error("rate-limited: quota exceeded; retry in 4s");
631+
},
632+
});
633+
634+
assert.equal(
635+
historyCalls,
636+
PAGE_REPLAY_MAX_ATTEMPTS,
637+
"backfill must retry a bounded number of times, then degrade",
638+
);
639+
});
640+
641+
test("backfill retry waits out the armed gate, then succeeds", async () => {
642+
resetGate(0);
643+
const delivered = [];
644+
const filter = buildChannelFilter("channel-1", 50);
645+
const subscriptions = new Map([
646+
[
647+
"live-1",
648+
{
649+
mode: "live",
650+
filter,
651+
onEvent: (event) => delivered.push(event),
652+
lastSeenCreatedAt: 1000,
653+
},
654+
],
655+
]);
656+
657+
const attemptAtMs = [];
658+
let armGate;
659+
const gateArmed = new Promise((resolve) => {
660+
armGate = resolve;
661+
});
662+
const replayPromise = replayLiveSubscriptions({
663+
subscriptions,
664+
now: 2000,
665+
sendRaw: async () => {},
666+
requestHistory: async () => {
667+
attemptAtMs.push(fakeNow);
668+
if (attemptAtMs.length === 1) {
669+
// Mirror relayClosedRecovery: the CLOSED handler arms the gate
670+
// before rejecting the history promise.
671+
activateRateLimit(4);
672+
armGate();
673+
throw new Error("rate-limited: quota exceeded; retry in 4s");
674+
}
675+
return [event("recovered", 1500)];
676+
},
677+
});
678+
679+
// Wait until the gate is actually armed, then expire it. The retry loop is
680+
// (or will be) suspended in waitForRateLimit; expiring the gate releases it.
681+
await gateArmed;
682+
tickTo(4_001);
683+
await replayPromise;
684+
685+
assert.equal(attemptAtMs.length, 2, "one failure, one retry");
686+
assert.ok(
687+
attemptAtMs[1] >= 4_001,
688+
"retry must not fire before the rate-limit gate expires",
689+
);
690+
assert.deepEqual(
691+
delivered.map((e) => e.id),
692+
["recovered"],
693+
"the retried backfill must deliver its events",
694+
);
695+
});
696+
697+
test("backfill retry aborts when the subscription was replaced", async () => {
698+
resetGate(0);
699+
const filter = buildChannelFilter("channel-1", 50);
700+
const subscription = {
701+
mode: "live",
702+
filter,
703+
onEvent: () => {},
704+
lastSeenCreatedAt: 1000,
705+
};
706+
const subscriptions = new Map([["live-1", subscription]]);
707+
708+
let historyCalls = 0;
709+
await replayLiveSubscriptions({
710+
subscriptions,
711+
now: 2000,
712+
sendRaw: async () => {},
713+
requestHistory: async () => {
714+
historyCalls++;
715+
// Simulate the subscription being torn down while the REQ is in flight.
716+
subscriptions.delete("live-1");
717+
throw new Error("rate-limited: quota exceeded; retry in 4s");
718+
},
719+
});
720+
721+
assert.equal(
722+
historyCalls,
723+
1,
724+
"no retry may target a subscription that no longer exists",
725+
);
726+
});
727+
728+
test("exhausted backfill pins the floor: next replay still requests the original window after live events advance the cursor", async () => {
729+
// The blocking review scenario on PR #4990: cursor=1000, all backfill
730+
// attempts fail, a live event at 2100 then advances lastSeenCreatedAt via
731+
// prepareSubscriptionEvent. Without the pinned floor, the next reconnect
732+
// would start near 2095 and silently skip 1001..1999.
733+
resetGate(0);
734+
const filter = buildChannelFilter("channel-1", 50);
735+
const subscription = {
736+
mode: "live",
737+
filter,
738+
onEvent: () => {},
739+
lastSeenCreatedAt: 1000,
740+
};
741+
const subscriptions = new Map([["live-1", subscription]]);
742+
743+
// Reconnect 1: every backfill attempt is rate-limited.
744+
await replayLiveSubscriptions({
745+
subscriptions,
746+
now: 2000,
747+
sendRaw: async () => {},
748+
requestHistory: async () => {
749+
throw new Error("rate-limited: quota exceeded; retry in 4s");
750+
},
751+
});
752+
assert.equal(
753+
subscription.pendingReplaySince,
754+
995,
755+
"exhausted backfill must pin the unresolved window's lower bound",
756+
);
757+
758+
// A live event arrives through the normal cursor path.
759+
prepareSubscriptionEvent(subscription, event("live-newer", 2100));
760+
assert.equal(subscription.lastSeenCreatedAt, 2100);
761+
762+
// Reconnect 2: backfill now succeeds. It must request the ORIGINAL window.
763+
const historyFilters = [];
764+
await replayLiveSubscriptions({
765+
subscriptions,
766+
now: 2200,
767+
sendRaw: async () => {},
768+
requestHistory: async (filter) => {
769+
historyFilters.push(filter);
770+
return [];
771+
},
772+
});
773+
774+
assert.equal(historyFilters.length, 1);
775+
assert.equal(
776+
historyFilters[0].since,
777+
995,
778+
"replay must start from the pinned floor, not the advanced cursor",
779+
);
780+
assert.equal(
781+
subscription.pendingReplaySince,
782+
undefined,
783+
"a completed backfill must clear the pinned floor",
784+
);
785+
786+
// Reconnect 3: with the floor cleared, replay returns to the cursor.
787+
const laterFilters = [];
788+
await replayLiveSubscriptions({
789+
subscriptions,
790+
now: 2300,
791+
sendRaw: async () => {},
792+
requestHistory: async (filter) => {
793+
laterFilters.push(filter);
794+
return [];
795+
},
796+
});
797+
assert.equal(
798+
laterFilters[0].since,
799+
2095,
800+
"after recovery the cursor governs again",
801+
);
802+
});
803+
804+
test("in-flight stale abort keeps the pinned floor for the superseding connection", async () => {
805+
// Race from re-review of b70a6716d/c493d378b: production supersession bumps
806+
// the connection GENERATION while the same subscription key and object
807+
// survive in the map. The identity guard alone stays true, so only the
808+
// combined guard (outer isActive && identity) aborts the stale pass. That
809+
// abort must NOT count as completion — the pinned floor belongs to the
810+
// superseding connection's replay.
811+
resetGate(0);
812+
const filter = buildChannelFilter("channel-1", 50);
813+
const subscription = {
814+
mode: "live",
815+
filter,
816+
onEvent: () => {},
817+
lastSeenCreatedAt: 1000,
818+
};
819+
const subscriptions = new Map([["live-1", subscription]]);
820+
821+
let generationActive = true;
822+
let historyCalls = 0;
823+
await replayLiveSubscriptions({
824+
subscriptions,
825+
now: 2000,
826+
sendRaw: async () => {},
827+
isActive: () => generationActive,
828+
requestHistory: async () => {
829+
historyCalls++;
830+
// Connection A is superseded while the REQ is in flight: the generation
831+
// advances, but the subscription keeps its key AND object identity —
832+
// exactly what production supersession does.
833+
generationActive = false;
834+
// A full page would otherwise continue paging — the post-await
835+
// combined guard must abort instead.
836+
return eventRange("full", 1001, 500);
837+
},
838+
});
839+
840+
assert.equal(historyCalls, 1, "stale generation must stop paging");
841+
assert.equal(
842+
subscriptions.get("live-1"),
843+
subscription,
844+
"precondition: key and object survive supersession untouched",
845+
);
846+
assert.equal(
847+
subscription.pendingReplaySince,
848+
995,
849+
"a stale-generation abort must not clear the floor the new connection needs",
850+
);
851+
});
852+
603853
// ── Teardown ──────────────────────────────────────────────────────────────────
604854

605855
test("teardown — restore Date.now", () => {

0 commit comments

Comments
 (0)